23 - Create RDD using parallelize method - Theory
@backstreetbrogrammer
--------------------------------------------------------------------------------
Chapter 04 - Create RDD using parallelize method - Theory
--------------------------------------------------------------------------------
Parallelized collections are created by calling JavaSparkContext’s parallelize() method on an existing Collection in the driver program. The elements of the collection are copied to form a RDD that can be operated on in parallel.
final var data = List.of(1, 2, 3, 4, 5, 6, 7, 8);
final var myRdd = sc.parallelize(data);
RDD created myRdd can be operated on in parallel. These operations can be to reduce, map, etc.
final var max = myRdd.reduce(Integer::max);
final var min = myRdd.reduce(Integer::min);
final var sum = myRdd.reduce(Integer::sum);
One important parameter for parallel collections is the number of partitions to cut the dataset into. Spark will run one task for each partition of the cluster.
We may want 2-4 partitions for each CPU in the cluster. Spark tries to set the number of partitions automatically based on our cluster.
We can get the number of partitions being used by calling this method:
myRdd.getNumPartitions()
However, we can also set it manually by passing it as a second parameter to parallelize() method.
sc.parallelize(data, 10)
We can also create Empty RDDs by calling emptyRDD() method on JavaSparkContext object.
final var emptyRDD = sparkContext.emptyRDD();
Github: https://github.com/backstreetbrogramm...
Apache Spark for Java Developers Playlist: • Apache Spark for Java Developers
Java Serialization Playlist: • Java Serialization
Dynamic Programming Playlist: • Dynamic Programming
#java #javadevelopers #javaprogramming #apachespark #spark
En esta página del sitio puede ver el video en línea 23 - Create RDD using parallelize method - Theory de Duración hora minuto segunda en buena calidad , que subió el usuario Rishi’s programming channel 11 febrero 2023, comparta el enlace con amigos y conocidos, en youtube este video ya ha sido visto 2,585 veces y le gustó 27 a los espectadores. Disfruta viendo!
