A reliable Lakehouse on Azure Databricks
How a medallion architecture with Delta Lake and Lakeflow Declarative Pipelines gives you data you can build on.
Anton Corredoira
Ask a team why they traded their data warehouse for a data lake, and you’ll hear: flexibility. Then ask why nobody trusts the numbers coming out of it, and you’ll hear the same answer. A Lakehouse on Azure Databricks can combine flexibility with reliability, but only if the foundations are right. Here is the approach that proves itself in practice.
Start with the medallion architecture
Split your pipelines into three layers:
- Bronze: raw data, exactly as it arrived, append-only. Your audit trail and safety net.
- Silver: cleaned, deduplicated and conformed. This is where data becomes reliable.
- Gold: business-ready tables, modelled for reporting and analysis.
That separation makes mistakes recoverable: if something breaks in Silver, you rebuild from Bronze without re-querying the source.
Let Lakeflow do the heavy lifting
With Lakeflow Declarative Pipelines (formerly DLT) you describe what a table should be, not how to populate it step by step. Databricks handles dependencies, incremental processing and recovery from failures. You add expectations to enforce data quality: rows that don’t meet them are flagged or dropped, not silently let through. If you’re weighing Lakeflow against dbt for the transformation layer, I’ve worked through that trade-off separately.
Ingestion that scales
For files trickling in, Auto Loader is the default: it incrementally tracks what’s new without expensive directory listings. For changes from source systems, change data capture feeds Silver. In both cases the rule is the same: process idempotently, so a restart never produces duplicate data.
Performance and cost are one conversation
Turn on Photon for SQL and DataFrame workloads and choose job clusters over always-on all-purpose clusters. Partition only on what you actually filter, and keep in mind that tables under a terabyte usually need no partitions at all; with Liquid Clustering you no longer have to set that choice in stone anyway. Then measure in the system tables what each pipeline costs; what you don’t measure, you don’t optimise.
In closing
A good Lakehouse is boring in the best sense: pipelines that run, data that’s correct, and costs that are predictable. Start small with one medallion flow, make it watertight, and build out from there. And sort out governance from day one: restructuring afterwards is expensive.