Data Engineering Project for Beginners | Airflow + Coder
Data Pipeline using Airflow for Beginners | Data Engineering Project | Airflow, API, GCP, BigQuery, Coder
Real case project to give you a hands-on experience in creating your own Airflow pipeline and grasping what Idempotency, Partitioning, and Backfilling are.
🧭 Plan:
OpenWeather API data → Data in data lake as Parquet files on GCP platform → Staging to Production tables in Data Warehouse (BigQuery)
🏆 Run the pipeline with Airflow using Coder - an open-source cloud development environment you download and host in any cloud. It deploys in seconds and provisions the infrastructure, IDE, language, and tools you want. Used as the best practice in Palantir, Dropbox, Discord, and many more.
Absolutely Free, a few clicks to launch, and super user-friendly.
Link to Github
⚡️ https://github.com/nataindata/data-en...
Coder platform
⚡️ https://rebrand.ly/NatCoder
My blog post
⚡️ https://www.nataindata.com/blog/data-...
GCP Free cloud trial description
⚡️ https://cloud.google.com/free/docs/fr...
Install Docker
⚡️ https://docs.docker.com/desktop/insta...
Comprehensive Data Engineering Roadmap
⚡️ https://www.nataindata.com/data-engin...
🏅 COURSES & CERTIFICATES
Top Data Engineer Courses:
Data Engineer in Python - https://datacamp.pxf.io/e1nbLz
Professional Data Engineer in Python - https://datacamp.pxf.io/55KLRL
Associate Data Engineer in SQL - https://datacamp.pxf.io/GKEqak
Building APIs in Python skill track - https://datacamp.pxf.io/VxgVn3
Top Data Analyst Courses:
Data Analyst with Python - https://datacamp.pxf.io/XmWZ6a
Data Analyst with PowerBI - https://datacamp.pxf.io/BnBAjL
Associate Data Analyst in SQL - https://datacamp.pxf.io/092ObV
Top Data Scientist Courses:
Associate Data Scientist in Python - https://datacamp.pxf.io/nXgba7
Associate Data Scientist in R - https://datacamp.pxf.io/Oez5XN
Industry Recognised Certifications:
Data Engineer Certification Program - https://datacamp.pxf.io/55KLWL
Data Analyst Certification Program - https://datacamp.pxf.io/N9XP5K
Data Scientist Certification Program - https://datacamp.pxf.io/K0g9rx
SQL Associate Certification Program - https://datacamp.pxf.io/xLObe5
WATCH THESE VIDEOS NEXT:
👀 starting a career in Data Engineering 10 things I wish I knew…
• Starting a Career in Data Engineering 10 T...
👀 AI agents tutorial: replacing entire Data team
• AI Agents Tutorial: Replacing an Entire Da...
◽️◽️◽️◽️◽️◽️
TIMESTAMPS:
0:00 Intro
0:20 Project Plan
00:31 Airflow made easy
01:17 Coder platform
01:35 Spinning up Airflow
04:25 Setting up GCP credentials
05:25 Adding connection
05:54 Adding variables
07:10 Creating BQ project
07:24 Pipeline coding
09:05 1st task
10:52 2nd task + partitioning
11:30 3rd task + struct
12:14 4th task
13:20 Backfilling
14:16 Idempotency
14:24 Stopping the project
14:43 Ta-da!
Please tell me if this video was helpful, I do appreciate your feedback!
Share your comments below ⬇️
In questa pagina del sito puoi guardare il video online Data Engineering Project for Beginners | Airflow + Coder della durata di ore minuti seconda in buona qualità , che l'utente ha caricato Nataindata 02 maggio 2024, condividi il link con amici e conoscenti, su youtube questo video è già stato visto 2,771 volte e gli è piaciuto 112 spettatori. Buona visione!
