Volume is not about how much you have. It is about how much you can afford to keep. Every byte stored is a byte that must be justified. Most bytes are not.
Volume is the most obvious dimension of data and the least interesting. It is what people mean when they say "big data" — petabytes, exabytes, the incomprehensible scale of modern data generation. A single Boeing 787 generates approximately 500 GB of sensor data per flight. A fleet of 1,000 aircraft flying two flights per day generates 1 PB per day. The Large Hadron Collider generates 90 PB per year. YouTube ingests 500 hours of video per minute. The numbers are large. The numbers are not the point.
The point is that volume forces choices. You cannot keep everything. The physics of storage — the cost per terabyte, the I/O throughput of the disk subsystem, the network bandwidth between storage and compute — imposes constraints that no amount of cloud elasticity can eliminate. Elasticity changes the cost curve from stepwise (buying servers) to continuous (paying per gigabyte). It does not make storage free. The cost is linear with volume. The value of data is not. The oldest data has the lowest value and the same cost as the newest data. The divergence between cost and value is the engineering problem of volume.
The solution is tiered storage. Hot data — queried frequently, needed in milliseconds — lives on SSDs or in-memory caches. Warm data — queried occasionally, needed in seconds — lives on object storage with fast retrieval tiers. Cold data — queried rarely, needed in minutes or hours — lives on the cheapest object storage tier, or on tape, or is deleted. The tiers are distinguished by cost, latency, and throughput. The assignment of data to tiers is an optimization problem: minimize total cost subject to latency constraints for each query class. The optimization is continuous because data ages. Yesterday's hot data is today's warm data. Today's warm data is next month's cold data. The tiering must be automated. The automation must be correct. A misclassified dataset — cold data on hot storage — is a cost inefficiency. Hot data on cold storage is a latency violation.
The engineering discipline of volume is the discipline of deciding what to throw away. Most organizations never make this decision. They keep everything. The storage bill grows. The value of the oldest data approaches zero. The cost is constant. The gap between cost and value widens. The gap is waste. The waste is invisible because the storage bill is aggregated. Nobody sees the line item for "data that has not been queried in three years." The line item exists. It is the majority of the bill. The majority of the bill is waste.
See: James Hamilton, "Internet-Scale Storage" (AWS Reinvent, 2014) — on the physics of storage at hyperscale. Alex Petrov, "Database Internals" (O'Reilly, 2019), Chapter 3, on B-Trees and LSM-Trees — the data structures that make volume queryable. David DeWitt and Jim Gray, "Parallel Database Systems: The Future of High Performance Database Systems" (Communications of the ACM, 1992) — the paper that predicted the distributed, partitioned architecture that makes petabyte-scale queries possible.
This post is part of a series on The Many Vs of Data, originating from Doug Laney's 2001 Gartner note. Each V names a dimension of why data is hard.