How to programmatically specify a schema?
Official Website: http://bigdataelearning.com
When you don’t know the structure of the data ahead of time, you cannot create a case class. Or, if the structure of data is ever-changing, then also it is not possible to specify a static schema. For e.g. it may be the case that a new column gets added today (or) an existing column can get dropped the day after. In such cases, how do you specify the case class without knowing the schema.
In such instances, you can follow the following 4 steps to programmatically specify the schema, and create a dataframe.
Step - I
======
Remove the header row from the original Rdd and create rddWithNoHeader Rdd. Here I am applying top action to the rdd rdd1. This should pick the first row of the Rdd. I am also passing the row to the parallelize method of spark context. This creates the headerRdd.
picking the first header row using the top action and creating an Rdd out of it
To remove the header row from the original rdd, I am using subtract transformation. Rdd1 dot subtract off headerRdd. This gives us rddWithNoHeader. Removing the header row from the original rdd1, by using the subtract transformation
Step II
======
Now that we have rdd with no header, second step is to generate a schema from the top row. Here I am using the first action to extract the first row from the Rdd as a string. picking the first row as schemaString
Next I am splitting the string based on delimiter and generating the schema. The schema is generated by using StructType and StructField types. Generate the schema based on the schemaString
Step III
======
The 3rd step is to generate the row Rdd from the regular Rdd.
Here I am using map transformation on rddWithNoHeader to create Row Rdd. The elements are split based on comma as delimited to create array of values. I am applying another map transformation on top of it to create a row rdd. convert the regular rdd to row rdd
Step IV
=======
The final step is to create Dataframe by using sqlContext’s createDataFrame method. Here I am passing the rowRdd and schema as parameters. This gives us a new dataframe.
Creating dataframe using createDataFrame method. Using the rowRdd and schema.
Auf dieser Seite können Sie das Online-Video How to programmatically specify a schema? mit der Dauer stunde minuten sekunde in guter Qualität ansehen, das der Benutzer BigDataElearning 05 Dezember 2020 hochgeladen hat, den Link mit Freunden und Bekannten teilen, dieses Video wurde auf Youtube bereits 1,659 Mal angesehen und es wurde von 7 den Zuschauern gefallen. Viel Spaß beim Betrachtenden Zuschauern gefallen!
