Terraform vs Bicep for Azure Databricks: which tool for which layer

Platform teams that have standardised their Azure landing zone on Bicep keep asking me the same question: can’t we just deploy Azure Databricks with Bicep too? The short answer: Bicep gets you to the workspace door and not one step further. If you want the whole platform managed as code, you end up at Terraform. There is one exception, which I cover at the end, and it demands a heavier dose of Databricks Asset Bundles than most teams expect.

Bicep stops at the workspace door

The Microsoft.Databricks resource provider exposes exactly five resource types: the workspace itself, access connectors, private endpoint connections, private link resources and VNet peerings. That is the complete list. No clusters, no jobs, no catalogs, no permissions.

For that slice, Bicep is fine. A workspace with VNet injection and an access connector for Unity Catalog is a handful of lines:

resource workspace 'Microsoft.Databricks/workspaces@2024-05-01' = {
  name: 'dbw-dataplatform-prd'
  location: 'westeurope'
  sku: { name: 'premium' }
  properties: {
    managedResourceGroupId: managedRg.id
    parameters: {
      customVirtualNetworkId: { value: vnet.id }
      customPublicSubnetName: { value: 'snet-dbw-public' }
      customPrivateSubnetName: { value: 'snet-dbw-private' }
      enableNoPublicIp: { value: true }
    }
  }
}

Together with the VNet, NSGs, private endpoints and Key Vault, this is the layer ARM governs, and therefore the layer where Bicep belongs.

Nothing inside speaks ARM

The real work starts after the workspace exists: the metastore assignment, storage credentials, external locations, catalogs and grants, cluster policies, SQL warehouses, groups and service principals. None of these objects live in Azure Resource Manager. They sit behind the Databricks REST APIs, and Bicep has no way in.

The Databricks Terraform provider aims to support all of those REST APIs, and in practice covers everything a platform build needs. The pattern I set up at clients: the azurerm provider creates the workspace and the access connector, the databricks provider builds everything inside it, and Terraform passes the references across automatically:

resource "databricks_storage_credential" "lake" {
  name = "cred-datalake-prd"
  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.uc.id
  }
}

resource "databricks_catalog" "sales" {
  name         = "sales_prd"
  storage_root = "abfss://catalogs@stdataplatformprd.dfs.core.windows.net/sales"
}

resource "databricks_grant" "sales_engineers" {
  catalog    = databricks_catalog.sales.name
  principal  = "data-engineers"
  privileges = ["USE_CATALOG", "USE_SCHEMA", "SELECT"]
}

Try this in Bicep and you end up with deployment scripts that curl the Databricks API. That is no longer infrastructure as code, that is a shell script wearing a YAML coat.

Three layers, three owners

The discussion gets cleaner once you cut the platform into layers:

• The landing zone: VNet, subnets, NSGs, private endpoints, the workspace itself, the access connector. This is ARM territory. Bicep if that is your organisation’s standard, otherwise azurerm in Terraform.
• The platform layer: metastore assignment, storage credentials, external locations, catalogs, grants, cluster policies, warehouses, groups. No Bicep option exists here. This is the Databricks Terraform provider, owned by the platform team.
• The project layer: jobs, pipelines, dashboards and a single team’s schemas. This does not belong in central Terraform state, because development teams iterate here daily. This is the domain of Databricks Asset Bundles, since renamed to Declarative Automation Bundles: YAML next to the code, deployed through the Databricks CLI in the team’s own CI/CD pipeline.

Terraform in the project layer is possible, but I advise against it: pushing every job change through the platform team’s state and pull request queue turns your fastest layer into your slowest.

The DAB-heavy route: can you skip Terraform?

There is one configuration where you can skip Terraform, and it is heavier on bundles than most teams realise. Bundles can define far more than jobs and pipelines these days: schemas, volumes, SQL warehouses, external locations and even catalogs. Bicep for the landing zone plus bundles for nearly everything inside is technically feasible.

But the same documentation is honest about the gaps: bundles cannot manage the metastore, storage credentials, users and groups, or connections. The very foundations of your governance end up managed by hand or by loose scripts. And bundles are scoped per project: ten teams means ten places where platform configuration can drift apart, with no central state to make that drift visible.

Pick this route only if Terraform is genuinely off the table in your organisation and your account-level configuration is small and stable. It is a workable exception, not a recommendation.

Never Bicep alone

The decision rule that remains is short. No Bicep mandate? Do everything in Terraform, with azurerm and databricks side by side, and bundles for the project layer. A Bicep mandate? Also fine: Bicep for the landing zone, Terraform for the platform layer, passing the workspace URL and access connector ID through as outputs. Both combinations work well.

What does not work is Bicep alone. Then you have an empty workspace as code and everything valuable inside it as click-ops, and click-ops is exactly what governance from day one was supposed to prevent.

In closing

Bicep and Terraform are not competitors here, they serve different layers: ARM outside the workspace, the Databricks APIs inside it. If you are unsure where to make the cut in your organisation, get in touch.