Data Engineering with Python and AI/LLMs – Data Loading Tutorial

Veröffentlicht am: 16 April 2025
auf dem Kanal: freeCodeCamp.org

157,160

3.5k

Master data ingestion for data engineering with Python. Learn to tackle common pipeline failures like schema changes and API limits by adopting the mindset and practices of a senior platform engineer. This course covers essential techniques including extracting data from APIs, automatic schema management, incremental loading, and orchestrating scalable, automated workflows using modern tools.

Course developed by Alexey Grigorev & Adrian Brudaru.

💻 Code: https://github.com/dlt-hub/dlthub-edu...

🏗️ dlthub.com provided a grant to make this course possible.

⭐️ Contents ⭐️
Alexey's part
0:00:00 1. Introduction
0:08:02 2. What is data ingestion
0:10:04 3. Extracting data: Data Streaming & Batching
0:14:00 4. Extracting data: Working with RestAPI
0:29:36 5. Normalizing data
0:43:41 6. Loading data into DuckDB
0:48:39 7. Dynamic schema management
0:56:26 8. What is next?

Adrian's part
0:56:36 1. Introduction
0:59:29 2. Overview
1:02:08 3. Extracting data with dlt: dlt RestAPI Client
1:08:05 4. dlt Resources
1:10:42 5. How to configure secrets
1:15:12 6. Normalizing data with dlt
1:24:09 7. Data Contracts
1:31:05 8. Alerting schema changes
1:33:56 9. Loading data with dlt
1:33:56 10. Write dispositions
1:37:34 11. Incremental loading
1:43:46 12. Loading data from SQL database to SQL database
1:47:46 13. Backfilling
1:50:42 14. SCD2
1:54:29 15. Performance tuning
2:03:12 16. Loading data to Data Lakes & Lakehouses & Catalogs
2:12:17 17. Loading data to Warehouses/MPPs,Staging
2:18:15 18. Deployment & orchestration
2:18:15 19. Deployment with Git Actions
2:29:04 20. Deployment with Crontab
2:40:05 21. Deployment with Dagster
2:49:47 22. Deployment with Airflow
3:07:00 23. Create pipelines with LLMs: Understanding the challenge
3:10:35 24. Create pipelines with LLMs: Creating prompts and LLM friendly documentation
3:31:38 25. Create pipelines with LLMs: Demo

🎉 Thanks to our Champion and Sponsor supporters:
👾 Drake Milly
👾 Ulises Moralez
👾 Goddard Tan
👾 David MG
👾 Matthew Springman
👾 Claudio
👾 Oscar R.
👾 jedi-or-sith
👾 Nattira Maneerat
👾 Justin Hual

--

Learn to code for free and get a developer job: https://www.freecodecamp.org

Read hundreds of articles on programming: https://freecodecamp.org/news

Auf dieser Seite können Sie das Online-Video Data Engineering with Python and AI/LLMs – Data Loading Tutorial mit der Dauer stunde minuten sekunde in guter Qualität ansehen, das der Benutzer freeCodeCamp.org 16 April 2025 hochgeladen hat, den Link mit Freunden und Bekannten teilen, dieses Video wurde auf Youtube bereits 157,160 Mal angesehen und es wurde von 3.5 tausend den Zuschauern gefallen. Viel Spaß beim Betrachtenden Zuschauern gefallen!

984