How to perform ETL Incremental Data Load using DLT | Data Load Tool | ETL | Python

Publicado el: 14 abril 2024
en el canal de: BI Insights Inc

1,673

In this video we will continue with the data load tool (dlt) library. We will explore how to perform incremental data load using dlt. The incremental data load in ETL (Extract, Transform and Load) is the act of loading only new or changed data. With this approach we process minimal data, use fewer resources and therefore less time. DLT refers to this as the merge write disposition.
We keep the lastest snapshot of a recrod in the data warehouse. We update and/or insert a new record in the dimension table. This is referred to as upsert.

Link to GitHub repo: https://github.com/hnawaz007/pythonda...

Link to previous video (db setup):    • data load tool (dlt) build database d...

Python based Incremental laod (Source Change Detection):    • Python ETL pipeline how to perform in...

Python based incremental load (Destination Change Comparison) :    • How to build ETL pipeline with Increm...

DLT docs on incremental load: https://dlthub.com/docs/general-usage...

Link to Channel's site:
https://hnawaz007.github.io/
--------------------------------------------------------------

💥Subscribe to our channel:
   / haqnawaz

📌 Links
-----------------------------------------
Follow me on social media!

🔗 GitHub: https://github.com/hnawaz007
📸 Instagram:   / bi_insights_inc
📝 LinkedIn:   / haq-nawaz
🔗   / hnawaz100
🚀 https://hnawaz007.github.io/

-----------------------------------------

#ETL #incremental #dlt

Topics in this video (click to jump around):
==================================
0:00 - Introduction to data load tool (dlt) incremental load
0:42 - Source Change Detection: Merge Write Disposition
1:38 - How Merge Write Disposition works
2:08 - Source SQL Server DB setup
2:33 - DLT Incremental Load Function
3:27 - Test Incremental Load Function
5:03 - Update/Insert records in Source SQL DB
5:29 - Run the dlt pipeline
5:36 - Review pipelines results
6:00 - Coming Soon

En esta página del sitio puede ver el video en línea How to perform ETL Incremental Data Load using DLT | Data Load Tool | ETL | Python de Duración hora minuto segunda en buena calidad , que subió el usuario BI Insights Inc 14 abril 2024, comparta el enlace con amigos y conocidos, en youtube este video ya ha sido visto 1,673 veces y le gustó 58 a los espectadores. Disfruta viendo!

48,554

276