Data Engineering with DuckDb Tutorial | PySpark | SQL | Postgres | Python | ETL Data processing
#dataengineering #etl #pyspark #python
Learn DuckDB: A Superfast Python library that beats Pandas and offers Pyspark Capabilities with unlimited possibilities.
In this demo, we will witness how to connect to the Postgres SQL database and query data.
How to read CSV data to perform data analytics and data engineering.
Different transformations and actions of Pysprak and how DuckDB helps integrate spark functionality flawlessly. How to transform and write data to Postgres database. How DuckDB helps install database and connectivity extensions from an extensive collection. How to perform end-to-end ETL using a blazingly fast Python library written in C++ programming language. End-to-end ETL pipeline to connect, extract, transform, and load data from and to Postgres SQL.
Code is available here: https://gist.github.com/Databracket9/...
00:00 - Introduction
01:45 - How to securely read and use environmental variables and secrets in Python using the ConfigParser library.
05:00 - How to install the Postgres extension and load it into DuckDB for connectivity and data analysis.
05:40 - Establising connectivity with Postgre SQL database using the connection string.
06:30 - Query SQL tables from Postgres
09:25 - How to read CSV files from DuckDB and load them as SQL views for data filtering.
12:20 - Import Experimental Pyspark functions to perform ETL data transformation.
14:00 - How to convert DuckDB class object into Pandas Dataframe.
14:18 - Create and Instantiate Pyspark Session.
14:35 - Convert Pandas DataFrame into PySpark Dataframe.
15:00 - Pyspark Transformation to filter and transform data.
18:35 - Write transformed data into Postgre SQL using DuckDB connection.
LET'S CONNECT!
🐦 Gumroad➔ https://databracket.gumroad.com/
📖Medium ➔ / jay-reddy
📲 Substack➔ https://databracket.substack.com
📰 LinkedIn ➔ / jayachandra-sekhar-reddy
💁Fiverr ➔ https://www.fiverr.com/jayreddy9
#pythonprogramming #postgresql #sql #database #cplusplusprogramming #bigdata #data #dataanalytics #dataanalysis
Sur cette page du site, vous pouvez voir la vidéo en ligne Data Engineering with DuckDb Tutorial | PySpark | SQL | Postgres | Python | ETL Data processing durée heure minute seconde en bonne qualité , qui a été Téléchargé par l'utilisateur Databracket 10 mai 2024, Partagez le lien avec vos amis et connaissances, sur youtube cette vidéo a déjà été regardée 2,223 fois et il a aimé 70 téléspectateurs. Bon visionnage!
