lru_cache hit vs miss: Cache Cost Lab

Intro — what this post promises

functools.lru_cache on a cheap function is not free. This lab times hits, miss-and-insert, and a plain uncached call on Linux localhost, and varies maxsize.

Related links:

• copy vs deepcopy localhost lab
• logging vs print localhost lab
• set vs list membership localhost lab
• string concat vs join localhost lab
• dataclass vs slots vs dict localhost lab
• json vs orjson vs msgpack localhost lab
• sorted vs heapq vs bisect localhost lab
• python re vs str methods localhost lab

Lab honesty (1 Oct 2026 IST): Python 3.13.5. Function is (n * 17) ^ (n >> 3) — CPU only, no I/O. Affiliates: 0. Hits use a key set that fits in the cache. Misses pass new integers so every call inserts (and may evict).

Verdict up front: uncached is ~11.35M/s (88.1 ns). A hit at maxsize 128 is ~17.12M (58.4 ns, ~1.51×). A miss-insert is slower than uncached: ~6.17M (162 ns). Cache the expensive call, not the cheap one.

Arms

maxsize=None is unbounded. Hit key space is 8 when maxsize is 8, else 64.

Lab topology

N = 200000 calls / repeat
maxsize in {8, 128, 1024, None} plus maxsize=0
Metric: p50 wall → ops/s and ns/op

Script: lab-evidence/52-lru-cache-hit-miss/results/run_lab.py.

Lead table — ops/s (p50)

What the curve says

• Hits are only ~1.5–1.6× the bare call because the function is already ~88 ns. The cache lookup itself is tens of nanoseconds. If your real function is milliseconds (SQL, parse, HTTP), that overhead vanishes.
• Miss-and-insert is a tax. At maxsize 128 it is ~0.54× uncached. You pay hash, store, and the function.
• Tiny maxsize + unique keys is the worst: maxsize 8 miss is ~258 ns — every insert evicts.
• maxsize=None on a miss storm keeps every key (currsize 200k). Slightly slower than a bounded cache and it grows RAM.
• maxsize=0 is slower than no decorator (103.8 ns vs 88.1 ns). It does not cache; it still wraps.

Pitfalls

1. Caching a 100 ns function “just in case” — a miss makes it slower.
2. Unbounded cache on user input — memory leak with a decorator.
3. Mutable args / unhashable — lru_cache will throw; this lab uses ints only.
4. Assuming hit ratio from folklore — call cache_info().

When to pick what

Reproduce

python3 lab-evidence/52-lru-cache-hit-miss/results/run_lab.py

Evidence: /workspace/lab-evidence/52-lru-cache-hit-miss/results/.

Closing

On this cheap function, a hit is ~58 ns vs ~88 ns uncached, and a miss-insert is ~162 ns (worse at maxsize 8: ~258 ns). maxsize does not change hit speed once the key fits; it changes eviction and RAM. Cache work that is actually expensive.