Automating Data Pipelines with Python & GitHub Actions [Code Walkthrough]

Published: 30 May 2024
on channel: Shaw Talebi

22,404

704

🤝 Your engineers use Claude but sales, ops and finance don't? I fix that for 50 to 200-person software companies: https://aibuilder.academy/yt/wJ794jLP2Tw

This is the 6th video in a series on Full Stack Data Science. Here, I use Python and GitHub actions to automate a data pipeline for FREE!

More Resources:
💻 Example Code: https://github.com/ShawhinT/data-pipe...
📰 Read more: https://medium.com/towards-data-scien...

🛠️ Data Engineering: • Text Embeddings, Classification, and Seman...
👨🏻‍💻 ML app repo: https://github.com/ShawhinT/yt-search
🔍 ML app UI: https://huggingface.co/spaces/shawhin...

--
Homepage: https://www.shawhintalebi.com/

Intro - 0:00
Motivation - 0:32
2 Ways to Automate - 1:28
Way 1: Orchestration Tool - 2:00
Way 2: Python + Triggers - 3:38
GitHub Actions - 5:56
Example Code: Automating ETL Pipeline - 7:42
1) Create ETL Python Script - 8:33
2) Create GitHub Repo - 12:21
3) Create Workflow .yml File - 13:22
4) Add Repo Secrets - 23:50
5) Commit and Push - 25:59
Final ML App - 28:45

On this page of the site you can watch the video online Automating Data Pipelines with Python & GitHub Actions [Code Walkthrough] with a duration of hours minute second in good quality, which was uploaded by the user Shaw Talebi 30 May 2024, share the link with friends and acquaintances, this video has already been watched 22,404 times on youtube and it was liked by 704 viewers. Enjoy your viewing!

808