Description
Guest: Michael Armbrust (Creator of Spark SQL, Delta Lake & Spark Declarative Pipelines)
Hosts: Youssef Mrini (Sr. Developer Advocate, Databricks), Alexandra Chashchina (Specialist Solution Architect, Databricks)
Spark Declarative Pipelines (SDP) let you declare the tables you want and let the engine handle execution, incremental compute, and recovery. Michael Armbrust — creator of Spark SQL, Delta Lake, and Spark Declarative Pipelines — joins host Youssef Mrini and Databricks Specialist SA Alexandra Chashchina to explain how SDP works, how it beats procedural ETL, how it compares to dbt, and where SDP is NOT the right tool.
SDP is the open-source framework Databricks contributed to Apache Spark. Lakeflow is the Databricks-managed product built on it, adding data-quality Expectations, the Enzyme incremental optimizer, a queryable event log, and the Pipeline Editor.
Inside the interview:
• The declarative paradigm — declare target tables; the system handles orchestration, incrementalization, and failure recovery
• The Enzyme optimizer — incremental updates with standard SQL semantics
• SDP vs dbt Core — pure SQL/Python vs Jinja templating and procedural incremental logic
• Auto CDC and SCD Type 1, Type 2, and bitemporal history
• Pipeline rewind & replay — point-in-time fixes across schema drift in one click
• Data-quality Expectations and the OBSERVE operator
• Serverless autoscaling and the multi-cluster roadmap on Spark Connect
• When NOT to use SDP — orchestration, sub-second latency, and simple one-off jobs
Chapters
00:00 Introduction: Michael Armbrust & guests
01:27 What are Spark Declarative Pipelines (SDP)?
02:01 SDP vs Notebooks, Jobs & raw PySpark
03:52 Streaming vs batch, and the Enzyme optimizer
06:00 Where SDP runs inside Databricks: System Tables, Lakebase sync, Vector Search
07:17 SDP vs dbt Core
08:28 Who SDP is for: SQL analysts vs data engineers
13:44 Generative AI, vibe coding & declarative pipelines
14:24 Cost optimization & TCO (Auto CDC)
17:11 Pipeline rewind & replay
18:47 Developer experience: Lakeflow Pipeline Editor, local dev & DABs
21:10 Unity Catalog: streaming tables & materialized views
23:18 Change Data Capture & SCD Type 1, 2 & bitemporal
27:58 Debugging: Expectations & the OBSERVE operator
29:56 Scaling pipelines & multi-cluster auto-partitioning
33:30 Recent Lakeflow updates & wrap-up
Resources
Bringing Declarative Pipelines to Apache Spark (blog): https://www.databricks.com/blog/bringing-declarative-pipelines-apache-spark-open-source-project
Lakeflow pipelines docs (formerly DLT): https://docs.databricks.com/aws/en/ldp/
Spark Declarative Pipelines concept (docs): https://docs.databricks.com/aws/en/ldp/concepts/spark-declarative-pipelines
Databricks Asset Bundles (DABs): https://docs.databricks.com/aws/en/dev-tools/bundles/
Apache Spark: https://spark.apache.org
Delta Lake: https://delta.io
Spark Connect: https://spark.apache.org/docs/latest/spark-connect-overview.html