03 - Apache Spark for Java Developers - Big Data
@backstreetbrogrammer
--------------------------------------------------------------------------------
Chapter 01 - Apache Spark for Java Developers - Big Data
--------------------------------------------------------------------------------
Big data is a term that describes large, hard-to-manage volumes of data – both structured and unstructured – that inundate businesses on a day-to-day basis.
These data sets are so voluminous that traditional data processing software just can’t manage them. But these massive volumes of data can be used to address business problems we would not have been able to tackle before.
Some interesting statistics:
1. More than 1 exabytes worth of data are generated each day
2. There are currently over 64.2 zettabytes of data in the entire digital universe in 2020 and expected to reach 175 zettabytes in 2025
3. The average person generates 1.7 MB of data per second
4. 90% of the world’s data has been created in the last two years
The 5V's of Big Data:
Volume
The amount of data matters. With big data, we’ll have to process high volumes of low-density, unstructured data. This can be data of unknown value, such as Twitter data feeds, click-streams on a web page or a mobile app, or sensor-enabled equipment. For some organizations, this might be tens of terabytes of data. For others, it may be hundreds of petabytes.
Velocity
Velocity is the fast rate at which data is received and (perhaps) acted on. Normally, the highest velocity of data streams directly into memory versus being written to disk. Some internet-enabled smart products operate in real time or near real time and will require real-time evaluation and action.
Variety
Variety refers to the many types of data that are available. Traditional data types were structured and fit neatly in a relational database. With the rise of big data, data comes in new unstructured data types. Unstructured and semi-structured data types, such as text, audio, and video, require additional preprocessing to derive meaning and support metadata.
Value
The bulk of data having no Value is of no good to the company, unless we turn it into something useful. Data in itself is of no use or importance, but it needs to be converted into something valuable to extract information. Hence, we can state that Value is the most important V of all the 5V’s.
Veracity
Veracity refers to inconsistencies and uncertainty in data, that is data which is available can sometimes get messy and quality and accuracy are difficult to control. Big Data is also variable because of the multitude of data dimensions resulting from multiple disparate data types and sources. Example: Data in bulk could create confusion whereas less amount of data could convey half or incomplete information.
Apache Spark for Java Developers Playlist: • Apache Spark for Java Developers
Java Serialization Playlist: • Java Serialization
Dynamic Programming Playlist: • Dynamic Programming
#java #javadevelopers #javaprogramming #apachespark #spark
In questa pagina del sito puoi guardare il video online 03 - Apache Spark for Java Developers - Big Data della durata di ore minuti seconda in buona qualità , che l'utente ha caricato Rishi’s programming channel 14 gennaio 2023, condividi il link con amici e conoscenti, su youtube questo video è già stato visto 4,878 volte e gli è piaciuto 44 spettatori. Buona visione!
