SO_REUSEPORT vs Single Listen: Independent Queues and ListenOverflows

Intro — what this post promises

SO_REUSEPORT lets multiple processes each bind() + listen() on the same IP:port. Nginx, Envoy, and many multi-worker servers expose it as reuseport. The folklore says it “scales accept.” This lab measures what actually moved on one box.

This is a localhost lab with measured numbers:

1. What ss shows: one shared listen queue vs N independent queues.
2. A fair echo microbench: single-listen workers vs SO_REUSEPORT workers.
3. Accept affinity (which worker PID answers).
4. A slow-accept flood where backlog pressure creates ListenOverflows.
5. What this does not claim (WAN, magic RPS multipliers, older-kernel thundering-herd lore alone).

Related links:

• HTTP Keep-Alive vs Connection: close lab
• Nginx limit_req rate-limit lab
• Why your average latency graph is lying (p50 / p95 / p99)
• How to read server monitoring graphs

Lab honesty (30 Sep 2026 IST): Shared Linux lab box (8 cores, kernel 6.12). Python 3.13.5 multi-process TCP servers bound to 127.0.0.1 only. Modes: inherited single listen() FD vs per-worker SO_REUSEPORT. No Docker. No public bind. Affiliates: 0.

Verdict up front: on this kernel, echo RPS was tied (~9.8k) and accept affinity was already fair without reuseport. The clear, measurable win was independent listen queues in ss and fewer ListenOverflows under a tight backlog + slow accept (e.g. +476 → +394 overflows; 62% → 72% client OK on the backlog=8 flood).

What SO_REUSEPORT changes

Without it, N workers typically share one listening socket (inherited FD). With it, each worker owns a listening socket on the same tuple; the kernel steers new SYNs across those sockets.

Related links:

• man 7 socket — SO_REUSEPORT
• nginx listen … reuseport

Mental model that matched our lab:

Pair this with the listen-backlog lab: reuseport does not remove the need for accept capacity; it shards the shock absorber.

Lab topology

ThreadPool clients
        │
        ▼
4 worker processes on 127.0.0.1:PORT
   mode=single     → one listen FD, accept() in each worker
   mode=reuseport  → each worker bind+listen with SO_REUSEPORT

Arms:

1. Fast 64-byte echo, n=8000, concurrency=64.
2. Affinity probe: server replies with os.getpid(), n=4000.
3. Slow accept flood: sleep(delay) after accept, small backlog, concurrent connect flood; deltas from /proc/net/netstat ListenOverflows / ListenDrops.

ss: the picture worth saving

With backlog=2 and 4 workers:

That is the operational tell. If you expected “four queues” but ss shows one line, you are on shared-listen, not reuseport.

Fast echo: RPS basically tied

n=8000, concurrency 64, payload 64 B, 4 workers:

No meaningful win. The client was opening a fresh TCP connection per op against a cheap handler — we were not backlog-bound. Honest labs say so.

Affinity: both modes spread work

n=4000 probes, reply = worker PID:

On this 6.12 kernel, shared accept() was already balanced. Do not sell reuseport as the only path to “fair workers” without measuring your kernel and workload.

Slow-accept flood: where reuseport showed up

Same flood shape, 4 workers, intentional accept delay so the listen queue matters.

backlog=8, delay=25 ms, n=400 connects (200 client threads):

backlog=2, delay=40 ms, n=300:

Healthy control — backlog 128, delay 0, n=300: both modes 300/300, overflow delta 0.

Reading: reuseport’s extra queues absorbed more of the same shock before the kernel dropped. Client OK % still tracks accept capacity (the sleep). Raising backlog or speeding accept remains mandatory — reuseport is not a substitute.

Pitfalls we hit (or avoided)

1. Claiming an RPS win from a connect-heavy echo — we measured a tie and kept it.
2. Forgetting to look at ss — the multi-LISTEN fingerprint is the config proof.
3. Ignoring overflow counters — success % alone understates how hard the kernel worked (ListenOverflows).
4. Treating reuseport as a WAN feature — it is same-host / same-VIP accept steering.
5. Skipping percentiles — flood survivors had ugly p95; see evidence JSON.

Related links:

• Why your average latency graph is lying (p50 / p95 / p99)

Practical checklist

• Multi-worker proxy/app: confirm reuseport (or equivalent) if you want per-worker listen queues.
• Verify with ss -ltnp — expect N LISTEN lines on the port, not one.
• Under incidents, check ListenOverflows / ListenDrops alongside CPU.
• Size backlog and accept/worker capacity; reuseport shards the queue, it does not create accept CPUs.
• Benchmark your handler; do not cite our echo RPS as a universal multiplier.

Verdict

SO_REUSEPORT on this box bought independent listen queues and fewer ListenOverflows under tight backlog + slow accept (72% vs 62% OK on the backlog=8 flood; ~41% fewer overflows on the backlog=2 arm). It did not multiply tiny-echo RPS, and accept affinity was already fair. Use it when you want sharded accept queues; still fix accept capacity and backlog.

Evidence path on the lab box: lab-evidence/16-so-reuseport/results/. Affiliates: 0.