Data Engineering Beginner Project - Implementation using Airflow, Python, S3, Snowflake (1/?)

Pubblicato il: 08 agosto 2023
sul canale di: enabledata: Data Engineering and DataOps

199

In this series about providing value with Data Engineering, there will be two kinds of videos: the concept and the implementation.

For every couple of videos, a new iteration will introduce new concepts to the platform - with its respective implementation. With this, more robustness and new features will be added to the platform, as this data product becomes more and more valuable.

This first implementation video will create the base for the next iteration, using technologies such as:
Apache Airflow;
Python (requests, pandas, boto3, awswrangler);
Docker (with Dockerfile customization);
Snowflake.

#DataEngineering #DataPlatform #BusinessIntelligence #DataDrivenFuture #DataPipelines #DataArchitecture

---
CHAPTERS

00:00 Intro
01:04 Create S3 Bucket
01:49 Create Access Key + S3 full policy
04:23 Setup Airflow locally with Docker
08:11 Install requirements for ETL in Jupyter
09:53 Create ETL
17:13 Customize Airflow Dockerfile
20:53 Create External Table in Snowflake
24:13 Next steps

---
📖 REFERENCES

GitHub Repo (w/ commit): https://github.com/dataengexplors/dat...

Install Docker Desktop: https://docs.docker.com/engine/install/

Running Airflow in Docker: https://airflow.apache.org/docs/apach...

Python environment: https://docs.python.org/3/library/ven....

Building a new image with Docker: https://docs.docker.com/engine/refere...

Snowflake External Table with S3: https://docs.snowflake.com/en/user-gu...

---
🔗 LINKS

Follow me on Twitter: / dataengexplors
Follow me on GitHub: https://github.com/dataengexplors

---

ABOUT ME

I have dedicated my career to all aspects of data, encompassing algorithm development for enhanced recommendations, diverse data visualization creation, processing unstructured data, and designing efficient solutions across AWS, Azure, Google Cloud, and on-premises environments. Currently serving as a Data Engineer in a private company, I possess hands-on experience in solving technical challenges while also leading teams in building optimal data structures. My passion for data engineering motivates me to stay updated with the latest industry trends, so I can use my expertise to unlock the full potential of data and deliver impactful outcomes.

For business inquiries: dataengineeringexplorers@gmail.com

In questa pagina del sito puoi guardare il video online Data Engineering Beginner Project - Implementation using Airflow, Python, S3, Snowflake (1/?) della durata di ore minuti seconda in buona qualità , che l'utente ha caricato enabledata: Data Engineering and DataOps 08 agosto 2023, condividi il link con amici e conoscenti, su youtube questo video è già stato visto 199 volte e gli è piaciuto 5 spettatori. Buona visione!

21,717