map() Transformation: The map() transformation applies a specified function to each element of the RDD and returns a new RDD consisting of the transformed elements.
# Creating an RDD rdd = spark.sparkContext.parallelize([1, 2, 3, 4, 5]) # Applying map transformation to square each element squared_rdd = rdd.map(lambda x: x ** 2)
In the above example, the map() transformation squares each element of the RDD rdd, creating a new RDD squared_rdd with the squared values.
filter() Transformation: The filter() transformation applies a predicate function to each element of the RDD and returns a new RDD consisting of the elements that satisfy the condition.
# Creating an RDD rdd = spark.sparkContext.parallelize([1, 2, 3, 4, 5]) # Applying filter transformation to get even numbers even_rdd = rdd.filter(lambda x: x % 2 == 0)
In this example, the filter() transformation selects the even numbers from the RDD rdd, creating a new RDD even_rdd with the filtered elements.
reduceByKey() Transformation: The reduceByKey() transformation in PySpark is used to aggregate values of the same key in a pair RDD. It applies a specified binary function to the values of each key in parallel and returns a new RDD with the results.
Here’s an example of using reduceByKey() in PySpark:
# Creating a pair RDD with key-value pairs pair_rdd = spark.sparkContext.parallelize([(1, 2), (1, 4), (2, 3), (2, 5), (3, 1)]) # Applying reduceByKey transformation to calculate the sum of values for each key sum_by_key_rdd = pair_rdd.reduceByKey(lambda x, y: x + y)
In the above example, the reduceByKey() transformation is applied to the pair RDD pair_rdd. The lambda function lambda x, y: x + y is used to define the reduction operation, which in this case is the sum of values. The reduceByKey() transformation combines the values for each key in parallel and returns a new RDD sum_by_key_rdd with the key-value pairs representing the sum of values for each key.
The result of the reduceByKey() transformation in this example will be an RDD with the following key-value pairs:
[(1, 6), (2, 8), (3, 1)]
It’s important to note that reduceByKey() operates on pair RDDs where each element is a tuple or pair of key-value. The function provided to reduceByKey() should be associative and commutative to ensure consistent results.
The reduceByKey() transformation is commonly used for tasks like aggregating data based on keys, such as finding the total count of occurrences, calculating sums, finding maximum or minimum values, or any other reduction operation based on keys.