mmap vs read (+ O_DIRECT): Localhost Sequential Scan Lab

Intro — what this post promises

Need to scan a file you already opened? Folklore splits into two camps: read() into a buffer vs mmap and touch the pages. A third knob — O_DIRECT — skips the page cache on purpose. This lab measures all three on a warm box.

This is a hands-on lab with measured numbers:

1. Sequential buffered read vs mmap page-touch on 64 MiB and 256 MiB files.
2. Chunk-size sensitivity (4 KiB / 64 KiB / 1 MiB).
3. Best-effort O_DIRECT via aligned libc.read (cache bypass).
4. An honesty arm: mmap that only reads the first and last byte (fake GB/s).

Related links:

• Pipe vs tmpfile IPC localhost lab
• Unix Domain Socket vs TCP localhost lab
• Why your average latency graph is lying (p50 / p95 / p99)
• How to read server monitoring graphs

Lab honesty (30 Sep 2026 IST): Shared Linux lab box (8 cores, kernel 6.12). Python 3.13.5. Payloads under /workspace (overlay). Warm page cache — we could not drop_caches without root. O_DIRECT used a 4096-aligned buffer + libc.read. No Docker. No GPU. Affiliates: 0.

Verdict up front: at 256 MiB / 64 KiB chunks, mmap touch hit 11.5 GB/s vs buffered read 5.4 GB/s (~2.1×). O_DIRECT sat near 0.9 GB/s because it refused the warm cache. Endpoint-only mmap printed fantasy terabyte/s numbers — do not ship that graph.

What we compared

Related links:

• man 2 mmap
• man 2 open — O_DIRECT

Lab topology

payload_64mb.bin / payload_256mb.bin
rounds=5 each · chunks 4 KiB, 64 KiB, 1 MiB
Arms: read_buffered | mmap_touch | mmap_endpoints_only | read_odirect
Metric: wall GB/s (p50 / mean) for a full sequential pass

Arm A — 64 MiB / 64 KiB chunks (p50)

Arm B — 256 MiB / 64 KiB chunks (p50)

Warm-cache mmap wins here because the kernel can fault pages already resident without an extra copy into a Python bytes object on every read. Buffered read still copies. That gap is real for scan-shaped work on hot files; it is not a claim about cold HDD/NVMe.

Chunk size still matters

Buffered read liked mid-size chunks. O_DIRECT hated tiny reads (alignment + syscall density). mmap touch was less sensitive to the userspace chunk because the walk step was fixed at 4 KiB pages.

When buffered read or O_DIRECT still win

• Short-lived one-shot reads of small files: read is simpler; mmap setup is overhead.
• Streaming into another API that wants a bytes/bytearray anyway: you will copy once either way.
• Isolating storage from cache effects (fio-style): O_DIRECT is the point — “slow” on a warm box is success.
• Untrusted file size / sparse horror: mmap of attacker-controlled paths needs care (not this lab’s threat model).

Pipes and tmpfiles are a different IPC question — see the pipe vs tmpfile lab.

Related links:

• Pipe vs tmpfile IPC localhost lab

Pitfalls we hit (or avoided)

1. Publishing mmap without page faults — endpoints-only looked 100× “faster”; we kept it as a warning.
2. Calling O_DIRECT “broken” on a warm cache — it is supposed to skip cache.
3. No drop_caches — cold-cache ranking may differ; we said warm explicitly.
4. Python os.read + O_DIRECT — unaligned destination → EINVAL; aligned libc.read fixed it.
5. Confusing this with sendfile — kernel sendpath to a socket is a different lab.

Practical checklist

• Scanning a hot file in-process: try mmap (or memoryview) and measure against read.
• Report touched bytes, not map size, when you quote GB/s.
• Use O_DIRECT when you intend to bypass cache; do not use it as a general “go faster” flag.
• Tune read chunk size; 4 KiB is often leave-performance-on-the-table.
• Do not treat loopback/overlay warm-cache numbers as cold-NVMe gospel.

Verdict

For full sequential touches on warm 64–256 MiB files, mmap beat buffered read ~1.8–2.2× (7.0 vs 4.0 GB/s at 64 MiB; 11.5 vs 5.4 GB/s at 256 MiB, 64 KiB chunks). O_DIRECT ~0.9 GB/s showed cache bypass, not failure. Endpoint-only mmap is a lie detector for your methodology — if you did not fault the pages, you did not measure the scan.

Evidence path on the lab box: lab-evidence/21-mmap-vs-read/results/. Affiliates: 0.