23 - Create RDD using parallelize method - Theory

Опубликовано: 11 Февраль 2023
на канале: Rishi’s programming channel

2,585

‪@backstreetbrogrammer‬

--------------------------------------------------------------------------------
Chapter 04 - Create RDD using parallelize method - Theory
--------------------------------------------------------------------------------
Parallelized collections are created by calling JavaSparkContext’s parallelize() method on an existing Collection in the driver program. The elements of the collection are copied to form a RDD that can be operated on in parallel.

final var data = List.of(1, 2, 3, 4, 5, 6, 7, 8);
final var myRdd = sc.parallelize(data);

RDD created myRdd can be operated on in parallel. These operations can be to reduce, map, etc.

final var max = myRdd.reduce(Integer::max);
final var min = myRdd.reduce(Integer::min);
final var sum = myRdd.reduce(Integer::sum);

One important parameter for parallel collections is the number of partitions to cut the dataset into. Spark will run one task for each partition of the cluster.

We may want 2-4 partitions for each CPU in the cluster. Spark tries to set the number of partitions automatically based on our cluster.

We can get the number of partitions being used by calling this method:
myRdd.getNumPartitions()

However, we can also set it manually by passing it as a second parameter to parallelize() method.
sc.parallelize(data, 10)

We can also create Empty RDDs by calling emptyRDD() method on JavaSparkContext object.
final var emptyRDD = sparkContext.emptyRDD();

Github: https://github.com/backstreetbrogramm...

Apache Spark for Java Developers Playlist:    • Apache Spark for Java Developers
Java Serialization Playlist:    • Java Serialization
Dynamic Programming Playlist:    • Dynamic Programming

#java #javadevelopers #javaprogramming #apachespark #spark

На этой странице сайта вы можете посмотреть видео онлайн 23 - Create RDD using parallelize method - Theory длительностью часов минут секунд в хорошем качестве, которое загрузил пользователь Rishi’s programming channel 11 Февраль 2023, поделитесь ссылкой с друзьями и знакомыми, на youtube это видео уже посмотрели 2,585 раз и оно понравилось 27 зрителям. Приятного просмотра!

21,717