Books / How Computers Work
Chapter 14

The Storage Pyramid

The Storage Pyramid

Fast storage is expensive. Slow storage is cheap. This has been an iron law of computing since the 1950s, and nothing has changed it. If a storage technology were simultaneously fast, cheap, and large, every other kind would vanish overnight. The laws of physics say that can't happen. So instead, computers are built around a pyramid: fast and expensive at the top, slow and cheap at the bottom, with each layer compensating for the weaknesses of the one below it.

Understanding the pyramid explains why CPUs need caches, why virtual memory exists, why databases manage buffer pools so carefully, and where the bottlenecks in any system actually live.

Core Concepts

The Complete Storage Pyramid

                      ┌────────┐
                      │Registers│  < 1 KB      ~0.3 ns     equivalent ~$10,000+/GB
                      └───┬────┘
                   ┌──────┴──────┐
                   │  L1 Cache   │  32–64 KB    ~1 ns       equivalent ~$1,000/GB
                   └──────┬──────┘
                ┌──────────┴──────────┐
                │     L2 Cache        │  256 KB–1 MB  ~4 ns
                └──────────┬──────────┘
             ┌─────────────┴─────────────┐
             │          L3 Cache         │  4–64 MB    ~30 ns
             └─────────────┬─────────────┘
          ┌────────────────┴────────────────┐
          │         Main RAM (DRAM)          │  8–128 GB  ~100 ns  ~$3–5/GB
          └────────────────┬────────────────┘
        ┌──────────────────┴──────────────────┐
        │            NVMe SSD                  │  500 GB–8 TB  ~0.1 ms  ~$0.10/GB
        └──────────────────┬──────────────────┘
      ┌────────────────────┴────────────────────┐
      │         HDD (spinning disk)              │  1–20 TB  ~10 ms  ~$0.02/GB
      └────────────────────┬────────────────────┘
    ┌──────────────────────┴──────────────────────┐
    │           Tape Archive                       │  100+ TB/cartridge  ~$0.002/GB
    └─────────────────────────────────────────────┘

Real Numbers for Each Layer

Registers

L1 Cache

L3 Cache

Main RAM (DRAM)

NVMe SSD

HDD (Hard Disk Drive)

Tape Archive

Why You Can't Use Just One Kind of Storage

What if you built a 256 GB computer using only SRAM (the technology of L1 cache)?

At SRAM's cost density: roughly $500,000. And if you built everything from tape, the CPU would wait minutes for each instruction fetch.

Here's a famous way to visualize the latency ratios. If one CPU cycle (0.3 ns) = 1 second of human time:

L1 Cache hit       =  1 second     ← looking up a dictionary
L2 Cache hit       =  4 seconds    ← grabbing a book from the shelf
L3 Cache hit       =  40 seconds   ← walking to the hallway bookcase
RAM access         =  3 minutes    ← walking to a nearby convenience store
SSD access         =  1.5 hours    ← taking the subway across town
HDD random access  =  16 hours     ← taking a high-speed train to another city
Tape (after load)  =  weeks        ← mailing a letter overseas

These ratios explain every architectural decision in computing: each layer exists to prevent the CPU from needing to go to the next layer down.

Locality Runs Through the Entire Pyramid

The whole pyramid works because the principle of locality holds at every level:

Every layer is the cache for the layer below it. The entire pyramid is a recursive caching system.

Try It Yourself

Measure the latency difference between cache-friendly and cache-hostile access in Python:

import time, array, random

SIZE = 1024 * 1024   # 1M integers (~4 MB — fits in L3, not L1/L2)
data = array.array('i', range(SIZE))

N = 100_000

# Sequential access — excellent spatial locality, high cache hit rate
t = time.perf_counter()
s = 0
for i in range(N):
    s += data[i]
print(f"Sequential {N} accesses: {(time.perf_counter()-t)*1000:.1f} ms")

# Random access — poor locality, frequent cache misses
indices = random.sample(range(SIZE), N)
t = time.perf_counter()
s = 0
for i in indices:
    s += data[i]
print(f"Random     {N} accesses: {(time.perf_counter()-t)*1000:.1f} ms")
# Random is typically 5–20× slower

Inspect your machine's cache hierarchy:

# Linux
getconf -a | grep -i cache
# or browse:
ls /sys/devices/system/cpu/cpu0/cache/
cat /sys/devices/system/cpu/cpu0/cache/index0/size   # L1 data cache size

# macOS
sysctl -a | grep cache

# Memory speed
sudo dmidecode --type 17 | grep -E "Speed|Size|Type"

🔬 Going Deeper

Intel Optane: bridging the gap between RAM and SSD

Between RAM (byte-addressable, nanosecond latency, volatile) and SSD (block-addressable, microsecond latency, persistent) lies a huge gulf. Intel Optane, based on 3D XPoint technology, tried to fill it: byte-addressable, persistent, with ~300 ns latency (about 3× slower than DRAM, ~300× faster than SSD). Databases could map their entire working set directly into Optane as persistent memory—no warmup needed after a restart. Optane was discontinued in 2022 when Intel restructured, but it proved that the market for a "persistent memory" tier between RAM and SSD is real. Competing technologies (PCM, FeRAM, ReRAM) continue to be researched.

CXL: expanding memory over a network-like link

CXL (Compute Express Link) is a next-generation memory expansion protocol that runs over PCIe physical links. It allows multiple servers or accelerators to share a common pool of memory, or attach additional DRAM/flash that the CPU addresses as if it were local memory. CXL memory latency falls between 200–400 ns (2–4× local DRAM), but it dramatically expands the total memory capacity available to a machine. Meta, Microsoft, and Google are deploying CXL memory expansion in data centers to break through the memory-capacity wall for large AI model inference.

The pyramid's future: layers blurring together

The traditional boundaries between layers are becoming less sharp:

The pyramid won't collapse into a single layer—physics prevents it—but the distinctions between levels will keep blurring as new technologies arrive.

Where to learn more

Rate this chapter
4.6  / 5  (20 ratings)

💬 Comments