Data Engineering Beginner Project - Implementation using Airflow, Python, S3, Snowflake (1/?)

Published: 08 August 2023
on channel: enabledata: Data Engineering and DataOps

199

In this series about providing value with Data Engineering, there will be two kinds of videos: the concept and the implementation.

For every couple of videos, a new iteration will introduce new concepts to the platform - with its respective implementation. With this, more robustness and new features will be added to the platform, as this data product becomes more and more valuable.

This first implementation video will create the base for the next iteration, using technologies such as:
Apache Airflow;
Python (requests, pandas, boto3, awswrangler);
Docker (with Dockerfile customization);
Snowflake.

#DataEngineering #DataPlatform #BusinessIntelligence #DataDrivenFuture #DataPipelines #DataArchitecture

---
CHAPTERS

00:00 Intro
01:04 Create S3 Bucket
01:49 Create Access Key + S3 full policy
04:23 Setup Airflow locally with Docker
08:11 Install requirements for ETL in Jupyter
09:53 Create ETL
17:13 Customize Airflow Dockerfile
20:53 Create External Table in Snowflake
24:13 Next steps

---
📖 REFERENCES

GitHub Repo (w/ commit): https://github.com/dataengexplors/dat...

Install Docker Desktop: https://docs.docker.com/engine/install/

Running Airflow in Docker: https://airflow.apache.org/docs/apach...

Python environment: https://docs.python.org/3/library/ven....

Building a new image with Docker: https://docs.docker.com/engine/refere...

Snowflake External Table with S3: https://docs.snowflake.com/en/user-gu...

---
🔗 LINKS

Follow me on Twitter: / dataengexplors
Follow me on GitHub: https://github.com/dataengexplors

---

ABOUT ME

I have dedicated my career to all aspects of data, encompassing algorithm development for enhanced recommendations, diverse data visualization creation, processing unstructured data, and designing efficient solutions across AWS, Azure, Google Cloud, and on-premises environments. Currently serving as a Data Engineer in a private company, I possess hands-on experience in solving technical challenges while also leading teams in building optimal data structures. My passion for data engineering motivates me to stay updated with the latest industry trends, so I can use my expertise to unlock the full potential of data and deliver impactful outcomes.

For business inquiries: dataengineeringexplorers@gmail.com

On this page of the site you can watch the video online Data Engineering Beginner Project - Implementation using Airflow, Python, S3, Snowflake (1/?) with a duration of hours minute second in good quality, which was uploaded by the user enabledata: Data Engineering and DataOps 08 August 2023, share the link with friends and acquaintances, this video has already been watched 199 times on youtube and it was liked by 5 viewers. Enjoy your viewing!

17,888

128