Big Data Architecture Today: The Lakehouse and Medallion Layers

Big Data Architecture is the plan for moving data from where it’s created to where it’s used. Today that plan has four jobs: ingest, store, process and serve. A fifth, governance, runs underneath all of them.

Start With Questions, Not Tools

Before I draw any boxes, I ask five questions. How big is the data now, and how big will it be in three years? How fresh must the answers be: seconds, hours or a day? What happens if the platform is down for an hour? Who gets to see which data? What can we spend each month?

The answers shape every Big Data Architecture choice that follows. A report that can wait until morning needs a nightly batch and a small bill. A fraud alert that must fire in seconds needs a streaming path and a bigger bill. Write the answers down first, and the tools nearly choose themselves.

Sources and Ingest

Sources are the systems that create data: operational databases, apps, files from partners, and event streams from websites or sensors. Ingest is the job of bringing that data in. It happens in three ways.

Batch ingest copies files or tables on a schedule, such as every night. Change data capture, called CDC, reads a database’s transaction log and ships only what changed since the last read. Streaming ingest takes events one at a time through a log such as Apache Kafka, within seconds.

Store: Object Storage and Open Tables

A modern platform keeps its data as files in cheap object storage. Most analytic data uses Parquet, a columnar file format. It stores each column together, so a query that needs three columns reads three columns, not every row in full.

Plain files have a weakness. Two jobs that write at once can leave a half-finished result, and nothing records the schema or the history. Open table formats such as Delta Lake and Apache Iceberg fix that.

They add a metadata layer on top of the Parquet files. Delta Lake records every change in a transaction log. Apache Iceberg tracks the table in metadata files that point to manifest files, which list the data files. Either way, you get ACID transactions, schema checks, and the ability to read an older version of a table.

A folder of files now behaves like a database table. That’s the core of the lakehouse. It pairs the cheap, open storage of a data lake with the reliable tables of a warehouse.

The Whole Picture

The diagram reads from left to right. Sources are databases, apps, files and events. Ingest brings them in by batch, change data capture or streams. Storage sits on object storage in three layers, bronze, silver and gold, held in an open table format.

Processing engines, Spark and SQL engines, move data from one layer to the next. Serving delivers it to reports, data science and apps. A governance band runs under every stage. SQL Server appears twice: as a source on the left and as a serving database on the right.

Bronze, Silver and Gold

The medallion pattern gives each layer one purpose. Bronze holds data exactly as it arrived. Silver holds it cleaned and joined. Gold holds it shaped for one business question. Here’s one grocery order moving through the layers.

Keeping bronze untouched is the key habit. If a cleaning rule turns out wrong, you rebuild silver from bronze. You don’t have to ask the source system for the data again. Each step also gets its own quality check, so a bad batch stops at the layer that caught it.

Medallion is a convention, not a standard. Some teams use two layers and some use four. What counts is that every layer has one clear purpose.

Process: Engines, Batch and Streaming

Processing engines read tables in one layer and write the next. Apache Spark splits the work across many machines. MapReduce Explained: Map, Shuffle and Reduce Step by Step shows that idea on a small example. SQL engines run the same steps as plain SQL.

Batch processes a large chunk of data on a schedule. Streaming processes events as they arrive. Spark shares many transformations between the two modes. Streaming adds its own rules for state, output and joins, so check what a streaming query supports. I start with batch and add streaming only where waiting costs money.

Serve: Where the Value Shows

Serving is where people finally see the data: dashboards, notebooks, machine learning models and applications. Gold tables feed most of it. Dashboards need fast, steady answers for many users at once. For that reason, many teams copy gold tables into a relational database built for the job.

Where SQL Server 2025 Fits

SQL Server is a source first. Change data capture records inserts, updates and deletes from the transaction log. SQL Server 2025 also adds change event streaming, which sends those changes out as events. Azure Event Hubs is a supported destination. It’s a preview feature, and it needs the PREVIEW_FEATURES database scoped configuration. Check its status before you plan around it. SQL Server can also write tables to Parquet files and read Parquet files from object storage.

Both need PolyBase and external data set up first. Exporting with CREATE EXTERNAL TABLE AS SELECT also needs the allow polybase export option turned on.

It also works as a serving database. A gold table is a summary, so it’s small next to bronze. A relational engine answers many small, repeated queries quickly. It controls access by row and column, and it speaks T-SQL, which analysts already know.

Governance Underneath

A catalog lists every dataset with its owner and meaning. Access control decides who reads what, down to the column. Lineage shows where a number came from. Quality checks and retention rules, which say how long data stays, finish the list.

Add these at the start of the platform. Adding them later means finding out that a sensitive column was already copied into many tables.

Is This Too Much Machinery?

The case against all of this is cost. If your data fits on one SQL Server and reports run in seconds, a lakehouse adds cost with no gain. For many small teams, that settles it.

The lakehouse earns its place in three cases. The data grows past one machine, it arrives in many formats, or many teams read it. The five questions at the top tell you which case you’re in.

What to Remember

Think of Big Data Architecture as four jobs and one band: ingest, store, process, serve, and govern. Keep bronze raw, make silver trustworthy, and shape gold for the question. Add streaming only to the paths that need it.

Three of the five questions, size, freshness and spend, are the volume, velocity and value of data. The 5 Vs of Big Data: Volume, Velocity, Variety, Veracity and Value covers all five Vs. Answer the questions first, then pick the tools.

Big Data Architecture is not a shopping list of tools, it is how raw data becomes trusted data.

Published by Pinal Dave on SQLAuthority. More of my work at pinaldave.com.