Data cleansing operations, such as handling missing values, are crucial in data preprocessing. PySpark provides several functions and methods to handle missing values in a DataFrame.
Here are some common techniques for handling missing values in PySpark:
Dropping Rows with Missing Values:
You can remove rows that contain missing values using the dropna() method. By default, it drops rows if any column has a missing value.
cleaned_df = df.dropna()
This drops all rows in the DataFrame df that have at least one missing value.
Dropping Rows with Specific Column Missing Values:
If you want to drop rows with missing values in specific columns, you can pass the column names to the subset parameter of the dropna() method.
cleaned_df = df.dropna(subset=["column1", "column2"])
This drops rows in the DataFrame df where either “column1” or “column2” has a missing value.
Filling Missing Values with a Constant:
You can fill missing values with a constant using the fillna() method.
filled_df = df.fillna({"column1": "N/A", "column2": 0})
This fills missing values in “column1” with “N/A” and missing values in “column2” with 0.
Filling Missing Values with Mean, Median, or Mode:
PySpark provides the fill() method to fill missing values with summary statistics like mean, median, or mode.
filled_df = df.fillna({"column1": df.select("column1").agg({"column1": "mean"}).first()[0]})
This fills missing values in “column1” with the mean value of “column1” calculated using the agg() method.
These are just a few examples of handling missing values in PySpark. Depending on your data and requirements, you can choose the most suitable approach. PySpark’s rich set of functions and methods provide flexibility in cleansing and preprocessing your data.