23 - Create RDD using parallelize method - Theory

Publié le: 11 février 2023
sur la chaîne: Rishi’s programming channel

2,585

‪@backstreetbrogrammer‬

--------------------------------------------------------------------------------
Chapter 04 - Create RDD using parallelize method - Theory
--------------------------------------------------------------------------------
Parallelized collections are created by calling JavaSparkContext’s parallelize() method on an existing Collection in the driver program. The elements of the collection are copied to form a RDD that can be operated on in parallel.

final var data = List.of(1, 2, 3, 4, 5, 6, 7, 8);
final var myRdd = sc.parallelize(data);

RDD created myRdd can be operated on in parallel. These operations can be to reduce, map, etc.

final var max = myRdd.reduce(Integer::max);
final var min = myRdd.reduce(Integer::min);
final var sum = myRdd.reduce(Integer::sum);

One important parameter for parallel collections is the number of partitions to cut the dataset into. Spark will run one task for each partition of the cluster.

We may want 2-4 partitions for each CPU in the cluster. Spark tries to set the number of partitions automatically based on our cluster.

We can get the number of partitions being used by calling this method:
myRdd.getNumPartitions()

However, we can also set it manually by passing it as a second parameter to parallelize() method.
sc.parallelize(data, 10)

We can also create Empty RDDs by calling emptyRDD() method on JavaSparkContext object.
final var emptyRDD = sparkContext.emptyRDD();

Github: https://github.com/backstreetbrogramm...

Apache Spark for Java Developers Playlist:    • Apache Spark for Java Developers
Java Serialization Playlist:    • Java Serialization
Dynamic Programming Playlist:    • Dynamic Programming

#java #javadevelopers #javaprogramming #apachespark #spark

Sur cette page du site, vous pouvez voir la vidéo en ligne 23 - Create RDD using parallelize method - Theory durée heure minute seconde en bonne qualité , qui a été Téléchargé par l'utilisateur Rishi’s programming channel 11 février 2023, Partagez le lien avec vos amis et connaissances, sur youtube cette vidéo a déjà été regardée 2,585 fois et il a aimé 27 téléspectateurs. Bon visionnage!