struct.pack vs to_bytes vs memoryview Lab

Intro — what this post promises

Packing a fixed record — u32, u16, u16, u64 (16 bytes, little-endian) — with struct.pack, Struct.pack_into, int.to_bytes, or a memoryview into a bytearray. This lab reports records/s and MB/s on Linux localhost.

Related links:

• base64 vs hex encode localhost lab
• json vs orjson vs msgpack localhost lab
• string concat vs join localhost lab
• copy vs deepcopy localhost lab
• logging vs print localhost lab
• SHA-256 vs BLAKE2 vs xxHash localhost lab
• uuid4 vs secrets token localhost lab
• atomic rename vs overwrite localhost lab

Lab honesty (1 Oct 2026 IST): Python 3.13.5. 200,000 records per batch. Affiliates: 0. No C extension beyond stdlib struct.

Verdict up front: Struct.pack_into ~125.7 MB/s (8.24M rec/s, 121 ns). struct.pack + b''.join is ~74.7 MB/s. to_bytes + memoryview is ~35.2 MB/s (~0.47× pack). Hand-rolled shifts are the slowest (~14.5 MB/s).

Layout

struct fmt: <IHHQ
bytes: 4 + 2 + 2 + 8 = 16
N = 200000 → 3.05 MiB per batch

Arms: struct.pack then join; compiled Struct.pack then join; pack_into a pre-sized bytearray; four to_bytes pieces joined; to_bytes written through a memoryview; manual byte shifts into a bytearray.

Script: lab-evidence/53-struct-pack-vs-to-bytes/results/run_lab.py.

Lead table — p50

Reading it

• pack_into wins because it does not allocate a 16-byte bytes per record and then join them. One buffer, one write cursor.
• Struct vs struct.pack is a small win (80.9 vs 74.7 MB/s) — the format is compiled once. Still slower than pack_into because of the per-record object + join.
• int.to_bytes builds four temporary bytes per record. memoryview avoids the final join but not those temps: 35.2 vs 30.4 MB/s.
• Manual shifts look “closer to the metal” and lose (1049 ns/rec). Python bytecode per byte is the tax; struct is C.

Pitfalls

1. Timing pack without the join — you still have to assemble the buffer in real code.
2. Native endian (@ / =) — this lab is explicit <.
3. Assuming memoryview makes to_bytes free — the allocations are inside to_bytes.
4. One-off headers — clarity of to_bytes is fine; this gap shows up at hundreds of thousands of records.

When to pick what

Reproduce

python3 lab-evidence/53-struct-pack-vs-to-bytes/results/run_lab.py

Evidence: /workspace/lab-evidence/53-struct-pack-vs-to-bytes/results/.

Closing

Write into a buffer. On this box pack_into ~126 MB/s beats struct.pack+join ~75 and to_bytes ~30–35. Manual shifts landed at ~15 MB/s. Use struct for layouts; use pack_into when the record rate is the point.