What Is Big Data? A Simple Explanation for Database People

What Is Big Data? It’s data that has outgrown what one machine and one database can handle in a useful time. Size alone isn’t the test. The test is the point where a single server stops being enough.

Why There Is No Magic Number

People who ask what is big data usually want a number, such as a terabyte or a petabyte. That number keeps moving. A 1 GB hard drive once felt endless, and a phone now holds hundreds of times more. A single server today can hold many terabytes.

So size alone is a poor test. A well-designed SQL Server database handles tables with billions of rows. Data becomes big when an ordinary approach starts to fail, and the failure shows up in one of four ways.

The first is storage: the data no longer fits on one machine. The second is time: a scan that should take minutes takes a day. The third is speed: data arrives faster than one server can write it. The fourth is variety: the data is JSON, logs, text and images, not neat rows. Only the first three are about size. Variety is about shape, and it calls for flexible formats, not more machines.

Three of the five Vs name these problems: volume, velocity and variety. The 5 Vs of Big Data: Volume, Velocity, Variety, Veracity and Value covers them in full.

What Each Failure Looks Like

Picture a pharmacy chain. Ten years of sales records from every till no longer fit on one server, which is a storage problem. The nightly report over all of them takes a whole day, which is a time problem.

Now picture a bike rental app with 200,000 bikes, each sending a location every second. That’s 200,000 writes a second, a load that one ordinary server struggles to keep up with. Then the company adds web orders as JSON, reviews as free text and photos of damaged bikes. The variety grows next to the speed.

Scale Up or Scale Out

When one machine runs out, you have two choices. Scaling up means buying a bigger machine, with more memory, faster disks and more cores. It’s simple, and your code doesn’t change. But there’s a ceiling, set by cost and by what the hardware can do.

Scaling out means adding more ordinary machines, called nodes, and giving each one a slice of the data. The ceiling is far higher, because you can keep adding nodes. The price is that the work must now be split and combined, which is harder than it sounds.

Scaling out also raises a safety question. Any node can fail at any time. So these systems keep copies of each slice on more than one node. That costs extra storage, but the data survives a failure.

One Worked Example

The diagram below uses 40 TB of data. On the left, one server is made bigger and bigger until it hits its ceiling. On the right, the same 40 TB is cut into 8 equal slices of 5 TB, one slice per node. A coordinator sends the query to every slice. Each node works on its own 5 TB and returns a small partial answer. The coordinator gathers the partial answers into one result.

Now the arithmetic. Say one machine reads 1 GB of data per second. Scanning 40 TB, which is 40,000 GB, takes 40,000 seconds, about 11 hours. Eight nodes scan their 5 TB at the same time. That takes 5,000 seconds, about 83 minutes.

The speed comes from the split, not from faster hardware. Real systems fall short of a perfect eight times, because the coordinator and the network take time too. The split and combine idea is the heart of tools like Spark. MapReduce Explained: Map, Shuffle and Reduce Step by Step walks through it on a small example.

How Today’s Tools Hide the Split

You’ll rarely split data by hand today. Apache Spark cuts large files into partitions and schedules a task for each one. Distributed SQL engines and lakehouse platforms do the same behind a SQL interface.

Formats such as Parquet store the slices as columnar files, so each node reads only the columns a query needs. You write SQL, and the platform plays coordinator.

Is Big Data a Marketing Label?

A fair objection: big data is a marketing word, and a tuned SQL Server solves most problems people call big. There’s a lot of truth in that. Before scaling out, check the indexes, the query plans, partitioning and columnstore.

Partitioning and columnstore can push one server a long way. Partitioning lets you move old data out in one fast step. Columnstore compresses data and scans only the columns a query uses. Try both first, and measure again.

Scaling out adds real cost. You take on a coordinator, a network and shuffles of data between nodes. Before anyone answers what is big data for your team, ask for proof that the single machine failed. Then ask whether it was storage, time or speed.

What to Remember

The answer to what is big data comes down to one test: does one machine still do the job? When storage, time or speed break the single machine, you have it. Scale up first when you can, and scale out when you must. Variety is a separate problem that needs flexible formats.

Ask what failed on the single server, and get a number. If nobody can name the failure, start with the query plan before adding nodes.

Big data is not a number of terabytes, it is the point where one machine stops being enough.

Published by Pinal Dave on SQLAuthority. More of my work at pinaldave.com.