In PySpark, the withColumnRenamed() function is used to rename a column in a Dataframe. It allows you to change the name of a column to a new name while keeping the rest of the Dataframe intact.
The syntax of the withColumnRenamed() function:
df.withColumnRenamed(existing, new)
Usage of withColumnRenamed() in PySpark:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
# Create a DataFrame
data = [("Alice", 25), ("Bob", 30), ("Charlie", 35)]
df = spark.createDataFrame(data, ["Name", "Age"])
# Rename the "Age" column to "AgeGroup"
df_new = df.withColumnRenamed("Age", "AgeGroup")
df_new.show()
Output:
+-------+--------+ | Name|AgeGroup| +-------+--------+ | Alice| 25| | Bob| 30| |Charlie| 35| +-------+--------+
In the example above, we have a DataFrame with columns “Name” and “Age”. We use the withColumnRenamed() function to rename the “Age” column to “AgeGroup”.
The resulting DataFrame, df_new, retains the original data but with the renamed column.
It’s important to note that withColumnRenamed() does not modify the original Dataframe; it returns a new Dataframe with the renamed column. You can assign the result to a new variable, as shown in the example, or overwrite the original Dataframe if desired.
If you don’t want to modify existing column you can always add a new column by using withColumn() Function.
The withColumnRenamed() function is useful when you want to give a more descriptive or meaningful name to a column in your Dataframe. It allows you to update column names without altering the underlying data or structure of the Dataframe.