In PySpark, the collect() function is used to retrieve all the data from a Dataframe and return it as a local collection or list in the driver program. It brings the entire Dataframe into memory on the driver node.
The collect() function is typically used when you want to retrieve the entire Dataframe and perform local operations on it, such as printing the data, converting it to a Python list, or processing it using local Python libraries.
Usage of collect() in PySpark:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
# Create a DataFrame
data = [("Alice", 25), ("Bob", 30), ("Charlie", 35)]
df = spark.createDataFrame(data, ["Name", "Age"])
# Retrieve all the data using collect()
collected_data = df.collect()
# Print the collected data
for row in collected_data:
print(row)
# Convert the collected data to a Python list
data_list = [row for row in collected_data]
print(data_list)
Output:
Row(Name='Alice', Age=25) Row(Name='Bob', Age=30) Row(Name='Charlie', Age=35) [Row(Name='Alice', Age=25), Row(Name='Bob', Age=30), Row(Name='Charlie', Age=35)]
In the given example, we worked with a Dataframe called “df” that consisted of two columns: “Name” and “Age.” We applied the collect() function to the Dataframe, which retrieved all the data and returned it as a list of Row objects. We now iterate over this collected data and perform various operations on each row, such as printing or converting it to a Python list.
The collect() function produced a list where each element represented a row in the Dataframe, accessible through dot notation (e.g., row.Name, row.Age).
It’s important to consider that the collect() function brings the entire Dataframe into the driver program, consuming significant memory resource. Hence, it’s advisable to use collect() only when the data size can be comfortably accommodated in the driver node’s memory.
It’s important to note that the collect() function brings the entire Dataframe into the driver program, so if the Dataframe is large, it may consume a significant amount of memory. Therefore, it’s recommended to use collect() only when the data can comfortably fit in memory on the driver node
For a deeper understanding, explore memory management and persistence levels.
when dealing with large datasets, it’s often more efficient to perform distributed operations on the Dataframe using Spark transformations and actions instead of collecting the entire Dataframe to the driver.