Memory Hierarchy: L1 to HBM — Circuit Copilot Glossary

There's no such thing as memory that's fast, big, and cheap. You get to pick two. The layers of a modern chip are how we live with that trade-off.

The fundamental trade-off

Memory speed comes from being close to the compute unit and from being small enough that the address lookup is fast. Memory capacity comes from being far away and dense. You can't have both.

So every modern chip stacks memory in tiers: fast/small at the top, slow/big at the bottom. The processor tries to keep the most-used data in the fastest tier.

The tiers, top to bottom

Registers — inside the CPU itself. ~1 ns access, but only kilobytes. This is where the current instruction's operands live.
L1 cache — 32-128 KB per core. ~1 ns. Split into instruction and data caches.
L2 cache — 256 KB - 2 MB per core. ~4 ns.
L3 cache — 8-96 MB shared across cores. ~10-40 ns.
Main memory (DDR5) — 8-256 GB. ~100 ns.
NVMe SSD — 500 GB - 8 TB. ~10 µs (10,000 ns).
Network storage — unlimited. Milliseconds.

Each tier is roughly 10× slower and 10× bigger than the one above it. This ratio hasn't changed much in 30 years.

Why caching works

Programs exhibit locality: they tend to access the same memory repeatedly (temporal locality) and nearby memory sequentially (spatial locality). Caches exploit this — a small fast tier can hold the "working set" and satisfy the vast majority of memory requests.

A well-tuned program with high cache hit rate can run 10-100× faster than a poorly-tuned one, on the exact same hardware.

HBM — the GPU exception

GPUs need enormous memory bandwidth to feed their thousands of cores. Standard DDR memory can't keep up. Enter HBM (High Bandwidth Memory): DRAM stacked vertically, mounted on the same package as the GPU, with an ultra-wide bus (1024 to 4096 bits wide, vs 64-bit for DDR).

An H100 has 80 GB of HBM3 with 3 TB/s bandwidth. That's 30× the bandwidth of a fast desktop CPU. This is why AI training happens on GPUs — the memory system is designed for it.

The programmer's job

Write code that respects the hierarchy. Access memory sequentially where possible. Reuse data before evicting it. Process data in cache-sized chunks (called blocking or tiling). Most performance-critical code — from matrix multiplication to database queries — is really about managing what's in cache at what time.