Lakeflow Declarative Pipelines vs dbt on Databricks
A fair, up-to-date comparison of the two leading ways to transform data on Databricks: what each does well, where each falls short, and how to choose.
Anton Corredoira
If you build data pipelines on Azure Databricks, you eventually face the same question: do you transform your data with Lakeflow Declarative Pipelines or with dbt? The honest answer: neither wins outright. Which one fits depends on your team, your data, and how much of the stack you want to own. But the trade-off can be made sharply, especially with the 2026 landscape in mind.
First, the naming (a lot has changed)
The space moved fast in the last year, so let’s get current:
- Delta Live Tables (DLT) was renamed to Lakeflow Declarative Pipelines. Its core engine was open-sourced and contributed to Apache Spark as Spark Declarative Pipelines, so the framework is no longer a closed, Databricks-only spec. The documentation has since started calling the product “Lakeflow Spark Declarative Pipelines”.
- dbt Labs and Fivetran completed their merger on 1 June 2026, bringing ingestion and transformation under one roof.
- dbt shipped dbt Core v2.0: open source under Apache 2.0 and built on the foundations of the Fusion engine, a Rust rewrite that parses faster, runs faster and actually understands SQL. The commercial Fusion distribution adds proprietary features on top. Databricks is supported by the new engine.
What each one actually is
Lakeflow Declarative Pipelines (LDP) is a declarative framework for building batch and streaming pipelines in SQL or Python. You declare streaming tables, materialized views and views; Databricks works out the execution order, handles incremental processing, retries failures and manages the compute. It covers the full ETL path, including ingestion and data quality.
dbt is a transformation framework. You write modular SQL SELECT statements (models); dbt resolves dependencies with ref()/source(), materializes them as tables, views or incremental models, and adds tests, documentation and a lineage graph. On Databricks it runs through the dbt-databricks adapter against SQL warehouses or clusters. It is the T in ELT; ingestion and orchestration sit elsewhere.
Lakeflow Declarative Pipelines
Strengths
- End-to-end ETL in one framework. Ingestion (Auto Loader, message buses), transformation and serving live together, not as three separate tools.
- Batch and streaming in the same model, with an AUTO CDC API for change data capture and SCD type 1/2 with very little code.
- Built-in data quality. Expectations validate rows and can warn, drop or fail. Since early 2026 they can also be stored and versioned in Unity Catalog and shared across pipelines.
- Automatic orchestration. Dependency resolution, retries and incremental recomputation are handled for you, with managed (including serverless) compute.
- Deep Unity Catalog integration for lineage, audit and permission propagation, enforced at runtime.
Trade-offs
- Databricks-bound. Even with the open Spark spec, the managed runtime and tooling live on Databricks. If you may move platforms, that is real lock-in.
- A different mental model. Streaming tables versus materialized views versus flows takes time to internalise.
- A smaller ecosystem. Fewer community packages, and testing/documentation conventions are less standardised than dbt’s.
- Cost discipline required. Continuous pipelines left running can get expensive if you don’t stay on top of them.
dbt
Strengths
- SQL-first with a huge community. A mature ecosystem of packages, generic and singular tests, auto-generated docs, exposures and a semantic layer.
- Portability. The same project can target Databricks, Snowflake, BigQuery and more, which reduces lock-in and protects your investment.
- Strong engineering workflow. Modularity, version control, CI and testing are first-class, which keeps large projects maintainable.
- The Fusion engine brings faster local development, real-time type checking and quicker runs, and dbt Core v2.0 is open source.
Trade-offs
- Transformation only. dbt does not ingest data or do streaming natively; you need Auto Loader, Fivetran or similar for the EL, and streaming is not a first-class concept.
- Needs separate orchestration. Scheduling comes from the dbt platform, Databricks Workflows or Airflow, not dbt itself.
- You manage the compute. On Databricks you still size and run the SQL warehouses or clusters dbt executes against.
- Some capabilities are paid. The cloud IDE, scheduler and semantic layer sit behind dbt’s commercial product.
Cost
Cost is where the two diverge most clearly, and it pays to be precise.
Lakeflow Declarative Pipelines has no separate licence. You pay only for Databricks compute (DBUs). The rate depends on cloud, edition and whether you run classic or serverless; expect roughly $0.20 to $0.40 per DBU, with the details on the pricing page. There is no per-seat or per-developer fee on top, and development happens in the Databricks workspace you already pay for.
dbt is free to run, but the comfortable UI is not. dbt Core is open source (Apache 2.0): you develop locally or in the VS Code extension, orchestrate with Databricks Workflows or Airflow, and pay only compute. The moment you want the managed experience with the IDE, scheduler and semantic layer, you land on the commercial platform: the Starter plan costs $100 per developer per month, and enterprise pricing is custom. That comes on top of the compute you still pay Databricks.
In short: with Lakeflow you pay compute and nothing else. With dbt you can too, as long as you stay on Core. But the UI-based workflow most teams eventually want carries real per-seat licensing on top.
Head to head
| Dimension | Lakeflow Declarative Pipelines | dbt |
|---|---|---|
| Scope | Ingestion + transformation + serving | Transformation (T) only |
| Streaming | First-class, native | Not a core concept |
| Data quality | Built-in expectations (UC-managed) | Tests (generic + singular) |
| Orchestration | Built in | External (Workflows, Airflow, dbt platform) |
| Languages | SQL and Python | SQL (plus Jinja, Python models) |
| Portability | Databricks (open Spark core) | Multi-warehouse |
| Ecosystem | Growing | Large and mature |
| Governance | Unity Catalog, runtime-enforced | dbt lineage graph + integrations |
| Cost model | Compute only (DBUs), no licence | Core free; UI paid per seat |
How to choose
Lean towards Lakeflow Declarative Pipelines if you are all-in on Databricks, need ingestion, streaming, CDC and data quality in one managed framework, and want minimal orchestration overhead with Unity Catalog-native governance.
Lean towards dbt if you have a SQL-centric team, value multi-warehouse portability, and want the mature testing, documentation and package ecosystem with a strong software-engineering workflow, while ingestion is handled elsewhere.
They are not mutually exclusive
The most pragmatic answer is often “both”. A very common pattern: use Lakeflow Declarative Pipelines (or Auto Loader) for ingestion and streaming into your bronze and silver layers, then use dbt to model the curated gold and marts in SQL, with its tests and docs. You get native ingestion and streaming where it matters, and a portable, well-documented transformation layer on top.
In closing
Pick the tool that fits the job in front of you, not the one with the loudest launch. And if you would like a second opinion on your specific setup, get in touch.