Evolution of Big Data: From Flat Files to the Lakehouse

Evolution of Big Data comes down to one pattern that repeats. Data grew, the old tool strained, and someone built the next one. Flat files led to relational databases, then warehouses, distributed systems, the cloud and the lakehouse. Each step solved a real problem and left a new one behind.

Flat Files: The 1960s

Early business programs kept data in flat files, which are plain records in a file or on tape. Only the program that wrote a file understood its layout. Finding one record could mean reading the whole file. Two programs sharing a file had to agree on every byte.

Integrity was a habit, not a feature. Hierarchical databases came next. IBM’s IMS, from the late 1960s, stored records in a tree. It was faster, but you had to know the path through the tree to reach a record. Changing the structure meant changing the programs.

The Relational Model: 1970

In 1970, Edgar F. Codd, a computer scientist at IBM, proposed a different idea. Keep data in tables, and let people ask for what they want without saying how to find it. Codd later wrote 12 rules that define what counts as a relational database.

SQL grew out of that work at IBM in the 1970s. Commercial relational databases arrived in the late 1970s, and by the 1980s they were the default for business data. Entity-relationship modeling, a way to draw tables and their links, grew up beside them. This was the stable era. Tables, keys and transactions gave businesses data they could trust.

The Whole Story on One Line

So far the story runs from flat files to relational tables. The timeline below shows the full evolution of Big Data by decade, from the 1960s to the 2020s. Each stop names the problem that pushed the next step. The sections that follow take the remaining stops one at a time.

Data Warehouses: Late 1980s and 1990s

As businesses stored more data, a new problem appeared. One database served the order screen and the monthly report, and the report slowed the orders down. The answer was a data warehouse: a separate database, loaded on a schedule, shaped for questions instead of transactions.

Bill Inmon and Ralph Kimball wrote the books that taught the design, in 1992 and 1996. Warehouses used ETL, which stands for extract, transform, load, to copy and clean data from the source systems. They worked well for structured data, and they still do. What they didn’t expect was the data that came next.

Too Much, Too Fast: The 2000s

Web companies in the early 2000s collected clicks, search logs and messages. This data was huge, it arrived fast, and it didn’t fit tables neatly. No single server, however large, could keep up.

Google published papers on its file system in 2003, on MapReduce in 2004 and on Bigtable in 2006. Hadoop, an open-source system built on those ideas, spread the work across many cheap machines. MapReduce Explained: Map, Shuffle and Reduce Step by Step walks through the core idea on a small example. NoSQL databases followed, trading some relational features for scale.

The Cloud and the Lake: The 2010s

Cloud storage changed the bill. Cloud data warehouses arrived in the early 2010s. Several of them separated storage from compute. You paid for disks and processors separately, and could grow either one alone.

Around the same time, Apache Spark began to replace older batch tools. It keeps working data in memory and handles batch and streaming. Kafka moved events between systems, and Parquet gave analytic data a compact, columnar file format. The term data lake appeared in 2010. The idea was to keep raw data in cheap storage and shape it when a question needed it.

The lake had a flaw. Without schemas, transactions or quality checks, many lakes turned into swamps. Companies ended up with a lake and a warehouse side by side. A pile of jobs copied data between them.

The Lakehouse: Today

Delta Lake and Apache Iceberg appeared in the late 2010s with an answer. They add a metadata layer on top of Parquet files in cheap storage. Delta Lake keeps a transaction log. Apache Iceberg keeps metadata files that point to manifest files. Either way, you get tables, ACID transactions and the ability to read an older version of a table.

One copy of the data now serves both the report writer and the data scientist. Teams organize it in bronze, silver and gold layers, from raw to ready. Big Data Architecture Today: The Lakehouse and Medallion Layers shows how those layers fit together.

Why Learn the History?

The case against this history lesson is simple: learn today’s tools and move on. But every layer from the evolution of Big Data still exists. CSV exports are flat files. Relational databases run the order screen, warehouses feed the reports, and lakehouses hold the raw material.

Knowing which problem each layer solved tells you when not to use it. A small company with 20 GB of data doesn’t need a lakehouse. A relational database or a small warehouse will do the job for less money.

What to Remember

Every step in the evolution of Big Data solved the last step’s biggest problem. Flat files lacked structure, so we got tables. Tables strained under reports, so we got warehouses. Warehouses couldn’t hold the flood, so we got distributed systems and the cloud. Lakes lacked reliability, so we got the lakehouse.

For any new data tool, ask which old problem it fixes. Then ask which new one it brings. That question works for every tool in this timeline, and it will work for the next one.

Big Data is not a new kind of data, it is the same old problems arriving at a larger scale.

Published by Pinal Dave on SQLAuthority. More of my work at pinaldave.com.