itertools.batched vs Manual Chunking: Localhost Lab

Intro — what this post promises

Split a sequence into fixed-size batches via itertools.batched vs manual list slices / islice chunking. This lab reports items/s on Linux localhost.

Related links:

• graphlib topo vs manual localhost lab
• functools cache vs lru localhost lab
• tomllib vs json localhost lab
• path glob vs fnmatch localhost lab
• dataclass replace vs manual localhost lab
• zoneinfo vs utc offset localhost lab
• heapq merge vs sorted localhost lab
• islice vs list slice localhost lab

Lab honesty (1 Oct 2026 IST): Python 3.13.5. Affiliates: 0. batched is stdlib since 3.12.

Verdict up front (100 k items / batch 100): batched consume ~274.18 Mitems/s; batched→list ~247.69; list slices ~242.25; islice chunks ~104.07. Prefer batched for clarity on iterables; slices fine for concrete lists.

Arms

Lab topology

10k×32 · 10k×256 · 100k×100 · 100k×1000 · 7 rounds · p50
metric: items/s = n / p50_s

Script: lab-evidence/120-itertools-batched-vs-chunk/results/run_lab.py.

Lead table — 100 k / batch 100 (p50)

Scale sketch

Tuple vs list batches

batched yields tuples. Wrap list(chunk) only if a callee requires lists — SQL executemany usually accepts tuples.

Remainder policy

Both styles emit a final short batch. If your protocol forbids short batches, pad or drop explicitly.

Reading it

• Any iterable / streaming source → batched.
• In-memory list → slices remain competitive.
• Avoid hand-rolled islice chunkers unless you need custom policies.
• Not the same as lab 109 (window/skip); this is tiling a sequence.

Why batch at all

Network APIs, executemany, and bulk inserts all prefer finite pages. Chunking wrong (off-by-one, dropped remainders, huge last page) causes retries and timeouts. batched makes the page loop obvious in code review.

Memory note

list(batched(...)) retains every page. For multi-million row exports, iterate for page in batched(rows, n): sink(page) without collecting. The consume arm in this lab models that path.

vs islice windows

Lab 109 measured skipping to a mid-list window. This lab tiles the whole sequence. Reusing an islice-based chunker is fine functionally but trailed batched and slices here (~104 Mitems/s vs ~240+ Mitems/s at 100 k/100).

Pitfalls

• Calling batched on Python ≤3.11 without a polyfill.
• Materializing all batches when streaming to a sink.
• Mutating the source while iterating batches.
• Confusing batch size with worker pool size.

Reproduce

python3 lab-evidence/120-itertools-batched-vs-chunk/results/run_lab.py

Evidence: summary.json, summary.txt.

Document batch size next to timeout and payload limits so operators can tune pages without reading the chunker implementation.

Limits

One Linux box. Integer lists in RAM. Not async or multiprocess batching.

Keep the comparison honest: measure your resource type (dummy vs FD) before optimizing.

Takeaway

At 100 k / 100, batched ~274.18 Mitems/s matches list slices ~242.25; DIY islice ~104.07 trails. Default to itertools.batched.