Databricks Architecture Explained: Control Plane, Compute Plane, Delta Lake, and Unity Catalog

# Overview

# Foundation: your cloud account Databricks runs on Azure, AWS, or Google Cloud. Your cloud account is the foundation: object storage (ADLS Gen2 on Azure or S3 on AWS) holds your raw files and remains under your control. The raw data stays in your account and does not move into Databricks-managed infrastructure.

# Control plane (Databricks-managed) The control plane is hosted and managed by Databricks. It contains the web application, notebooks, the job scheduler, and the cluster manager. The control plane accepts user commands and orchestrates work, but it does not run the heavy compute on your data by itself in the classic setup.

# Compute plane (your cloud account or Databricks account for serverless) The compute plane — also called the data plane — is where actual compute runs. In the classic model clusters run inside your cloud account and your virtual network. With serverless compute, the compute plane runs in the Databricks account instead. Compute reads and writes data stored in your cloud object storage.

# Data lake and Delta Lake At the base is your data lake: object storage in your cloud account. On top of that sits Delta Lake, a storage layer that turns Parquet files into a lakehouse by adding atomic transactions, schema enforcement, and time travel. Delta Lake is the layer that gives a data lake database-like behavior.

# Bronze / Silver / Gold zones Teams typically structure data within Delta Lake in three zones:

• Bronze: raw ingested files.
• Silver: cleaned and joined datasets suitable for downstream use.
• Gold: business-level aggregates and tables prepared for reporting and BI.

This layered organization separates ingestion, transformation, and consumption responsibilities.

# Unity Catalog: governance across everything Unity Catalog sits across workspaces and catalogues the data. It provides unified governance: a single catalog, permissions management, and lineage for assets. Unity Catalog makes it possible to enforce policies and grant consistent access across workspaces and compute engines.

# What happens when you run a notebook

# Consumption patterns

# Practical takeaway Design around separation of control and compute: keep sensitive data in your cloud account, use Delta Lake for reliable transactional storage and versioning, and use Unity Catalog to centralize permissions and lineage across teams and workspaces.