Declarative Pipelines: What’s Next for Apache Spark
Early this year, Spark Declarative Pipelines (SDP) was announced, which has made it dramatically easier to build robust Spark pipelines using a framework that abstracts away orchestration and complexity. The SDP declarative framework extends beyond individual queries to enable a mix of batch and streaming pipelines, keeping multiple datasets fresh.
In this session, Sandy Ryza (Databricks) shares a broader vision for the future of Spark Declarative Pipelines — one that opens the door to a new level of openness, standardization, and community momentum.
Key takeaways:
✅ Core concepts behind Spark Declarative Pipelines;
✅ Where the architecture is headed; and
✅ What this shift means for both existing users and Spark engineers building procedural code.
This talk was part of the “Open Lakehouse + AI Mini Summit,” hosted at the Databricks Mountain View office on November 13, 2025.
00:00 – Intro
02:00 – What “declarative” means; what is a pipeline
04:30 – Why non‑declarative pipelines are hard (ordering, parallelism)
08:00 – Errors, retries, and discrete vs continuous execution
11:30 – Orchestrators (Airflow) pros/cons and limits
14:30 – Introducing Spark Declarative Pipelines (SDP)
17:30 – Core APIs: streaming tables, materialized views, SQL/Python, runner
22:30 – Modes, dependency inference, validation; SDP vs dbt vs Airflow; availability
На этой странице сайта вы можете посмотреть видео онлайн Declarative Pipelines: What’s Next for Apache Spark длительностью часов минут секунд в хорошем качестве, которое загрузил пользователь Open Lakehouse + AI 18 Ноябрь 2025, поделитесь ссылкой с друзьями и знакомыми, на youtube это видео уже посмотрели 1,925 раз и оно понравилось 51 зрителям. Приятного просмотра!
