Databricks Lakehouse — A Field Guide

The "lakehouse" pitch is simple: keep data in open files on cheap object storage, but get the transactional guarantees and query performance people built data warehouses for. Databricks didn't invent that idea alone, but it's the platform that pushed it into the mainstream by open-sourcing Delta Lake and building the whole workspace — SQL, notebooks, streaming, ML, and BI — on top of one table format. Here's what actually holds that story together, and where a lakehouse still isn't the right call.

The problem a lakehouse solves

Data lakes are cheap and flexible: dump Parquet or JSON into S3 or ADLS, schema-on-read, no gatekeeper. The catch is that a folder of files has no transactions. Two jobs writing at once can corrupt a table. A failed job can leave half-written files with no way to roll back. There's no single source of truth for "what did this table look like as of last Tuesday." Data warehouses solve all of that with ACID transactions and fast SQL, but at the cost of proprietary storage, expensive compute-storage coupling, and a much harder time handling unstructured data or ML workloads.

The lakehouse pattern keeps data in open formats on object storage — so it's cheap and usable by any engine — but adds a transaction layer on top so it behaves like a warehouse table. In Databricks that layer is Delta Lake.

Delta Lake: the transaction log

A Delta table is a directory of Parquet files plus a _delta_log folder — a sequence of JSON commit files that record every add, remove, and metadata change. Readers don't scan the directory and guess; they replay the log to know exactly which files make up the current (or a historical) version of the table. That log is what gives you atomic commits, isolation between concurrent writers, and time travel.

-- Upsert changed rows from a staging table
MERGE INTO orders AS target
USING orders_staging AS source
ON target.order_id = source.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

-- Query the table as it looked yesterday
SELECT * FROM orders VERSION AS OF 42;
SELECT * FROM orders TIMESTAMP AS OF '2026-07-20';

-- Compact small files, then drop old versions no longer needed
OPTIMIZE orders ZORDER BY (customer_id);
VACUUM orders RETAIN 168 HOURS;

MERGE INTO is the workhorse for upserts and slowly changing dimensions — something plain Parquet on a lake can't do without rewriting whole partitions. OPTIMIZE compacts the small files that streaming or micro-batch writes tend to produce, and ZORDER co-locates related data to speed up filtering. VACUUM physically deletes files no longer referenced by the log, which is also what eventually caps how far back time travel can go.

Medallion architecture: bronze, silver, gold

Databricks documentation leans on a three-layer naming convention for organizing tables, not a required feature — you can build a lakehouse without it, but it's a useful default:

• Bronze — raw data landed as-is from source systems, kept for lineage and reprocessing. Minimal transformation, append-only.
• Silver — cleaned and conformed: types fixed, duplicates removed, joins applied. Still fairly granular, one table per entity.
• Gold — business-level aggregates built for a specific purpose, like a BI dashboard or a reporting table.

The point of the layering is that each stage is independently reprocessable. If a silver transformation had a bug, you fix the logic and replay from bronze rather than trying to patch the gold table by hand.

Unity Catalog: governance across the workspace

Unity Catalog is Databricks' governance layer: a three-level namespace (catalog.schema.table) with centralized access control, so permissions are defined once and apply across SQL, notebooks, and jobs instead of being re-implemented per cluster. It also tracks lineage — which notebooks and jobs read and wrote a given table — and gives you a searchable catalog of tables, views, and ML models across workspaces. Before Unity Catalog, permissions were largely enforced at the cluster or table-ACL level per workspace, which didn't scale once an organization had more than a handful of teams sharing data.

Batch, streaming, ML, and BI on one copy of the data

Because everything reads and writes the same Delta tables, the same data serves very different workloads without copying it into separate systems. Structured Streaming can write micro-batches into a Delta table while a scheduled job runs OPTIMIZE on it and a BI tool queries it through Databricks SQL — all against the same underlying files, coordinated by the transaction log.

(spark.readStream
    .format("delta")
    .table("bronze.events")
    .withWatermark("event_time", "10 minutes")
    .groupBy(window("event_time", "5 minutes"), "event_type")
    .count()
    .writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "/chk/silver_event_counts")
    .toTable("silver.event_counts"))

A notebook doing feature engineering for a model can read from the same silver table a dashboard queries, so there's no separate ETL pipeline just to get data into a warehouse for BI and a different one to get it into a feature store for ML. That's the practical payoff of "lakehouse" — fewer copies of the data, fewer pipelines keeping those copies in sync, and one governance model instead of three.

When a lakehouse makes sense — and when it doesn't

A lakehouse earns its complexity when you have a genuine mix of workloads: streaming ingestion, ML training, and BI reporting all needing the same underlying data, or data volumes and variety (semi-structured, unstructured) that a traditional warehouse handles poorly. It's also the right call when vendor lock-in to a proprietary storage format is a real concern, since Delta tables are just Parquet plus a log that other engines (Trino, Presto, Snowflake via Delta connectors) can read.

It's overkill if you have a single BI workload against a few gigabytes of clean, structured data — a managed warehouse like Snowflake or BigQuery will be simpler to operate and probably cheaper at that scale, with none of the cluster-sizing and job-orchestration decisions a Spark-based platform asks you to make. The lakehouse pattern pays for itself at the point where "just export it to a warehouse" stops being a one-line answer.