The Latency Ladder, by Feel
A small interactive tool for getting the right gut-feel about how slow each rung of the storage hierarchy actually is.
There is a list every backend engineer eventually memorizes — the “latency numbers every programmer should know” table. It’s useful, but a table of numbers is the wrong representation for what we actually care about, which is shape: how a real operation’s wall time decomposes across the storage hierarchy.
I built this little widget for myself, mostly to stop kidding myself about where the time in a RAG query actually goes.
- L1 cache reference1 ns ea
- Branch mispredict3 ns ea
- L2 cache reference4 ns ea
- Mutex lock/unlock17 ns ea
- Main memory reference100 ns ea
- Compress 1 KB w/ Snappy×1 · 2.0 µs
- Send 2 KB over 1 Gbps net20 µs ea
- SSD random read (4 KB)×1 · 16 µs
- Read 1 MB sequentially RAM3.0 µs ea
- Read 1 MB sequentially SSD49 µs ea
- Round trip within a datacenter×1 · 500 µs
- Read 1 MB sequentially HDD825 µs ea
- HDD seek10 ms ea
- CA -> Netherlands round trip150 ms ea
Pick a different recipe and watch which rung dominates. A few observations I keep coming back to:
L1 is not “fast”, it’s a different universe
A tight loop over an L1-resident array runs at ~1 ns per op. A single intra-DC round trip is 500,000 ns. That’s a factor of half a million.
If your hot loop is doing one network call per iteration, no amount of fiddling with the CPU side will help. The bar chart above flatlines on everything but the network row.
Random SSD reads are not free
A 4 KB random SSD read is ~16 µs. That is genuinely good — an order of magnitude faster than a spinning disk for the same operation. But it is still 16,000× slower than an L1 reference. Every B-tree lookup that misses buffer pool bottoms out here.
This is the reason people write LSM trees instead of B-trees when write volume gets serious: amortize random writes into sequential ones, then live with the read amplification.
RAG queries are network-bound, not compute-bound
Look at the RAG recipe. The embedding round trip, the vector DB round trip, and the LLM call dominate everything else by two orders of magnitude. The embedding model’s GPU time is real, but it’s not what’s making your end-to-end latency look bad.
When people complain that RAG is “slow”, they’re usually complaining about a problem that more SSD or faster CPUs cannot fix. The fix is colocating the services, batching, or caching the embeddings.
The CA → Netherlands rung is humbling
150 ms. Once. Half the budget for a “snappy” web request, spent on a single round trip across an ocean. Every architecture diagram with two regions and a synchronous arrow between them is, secretly, a 150 ms tax on every request.
The numbers in the widget are coarse and a little dated — they came from Jeff Dean’s original list with minor updates. The point isn’t precision. The point is that the ratios between rungs are stable across hardware generations, and they’re what your intuition needs to be calibrated against.
If your mental model can’t explain why one recipe is ten thousand times slower than another at a glance, the numbers haven’t sunk in yet.