string concat vs join vs StringIO: Build Lab

Intro — what this post promises

Building a big string in a loop: s += chunk, parts.append + "".join, or io.StringIO? Folklore says “never +=” and “always join.” This lab builds fixed sizes from 64-byte chunks on Linux localhost and reports MB/s (p50).

Related links:

• sorted vs heapq vs bisect localhost lab
• pathlib vs os.path localhost lab
• json vs orjson vs msgpack localhost lab
• base64 vs hex encode localhost lab
• python re vs str methods localhost lab
• dataclass vs slots vs dict localhost lab
• tempfile NamedTemporaryFile localhost lab
• SHA-256 vs BLAKE2 vs xxHash localhost lab

Lab honesty (1 Oct 2026 IST): Python 3.13.5. Chunk = 64×"x". Affiliates: 0. CPython may over-allocate on += — this is not a guarantee for other runtimes.

Verdict up front: at 1 MiB, list+join ~2767 MB/s beats += ~781 (~0.28×) and StringIO ~1113 (~0.40×). At 32 MiB, join still leads (~1304 MB/s) over += (~860) and StringIO (~600). Prefer join; don’t assume StringIO is faster.

Arms

Each arm must produce a string of exact target length — asserts catch under-builds.

Lab topology

chunk = 64 bytes ASCII
Targets: 4KiB, 64KiB, 1MiB, 8MiB, 32MiB
Repeats: 7 (3–5 for largest)
Metric: p50 wall → MB/s and chunks/s

Script: lab-evidence/49-string-concat-vs-join/results/run_lab.py.

Lead table — MB/s (p50)

1 MiB wall times (p50): join ~0.36 ms, += ~1.28 ms, StringIO ~0.90 ms.
32 MiB: join ~24.5 ms, += ~37.2 ms, StringIO ~53.3 ms.

Prealloc list helped a little at mid sizes (~1.03× join at 1 MiB; ~1.13× at 8 MiB) and was roughly tied at 32 MiB — join itself is the win, not fancy pre-sizing.

Why += is not always a catastrophe (here)

CPython can over-allocate string storage so some += loops avoid full quadratic copies. You still see ~0.28× join at 1 MiB and ~0.66× at 32 MiB — slower, not explosive. Do not rely on that across PyPy/other versions; "".join is the portable habit.

StringIO was not the winner: write + getvalue() copy landed ~0.40× join at 1 MiB and ~0.46× at 32 MiB. Use it for file-like APIs, not as a magic faster builder.

Bytes contrast (1 MiB): BytesIO then decode ~1596 MB/s; bytes join then decode ~1262 MB/s — decode tax included; prefer staying in str or bytes end-to-end.

At 8 MiB the gap narrows some (+= ~0.46× join) but join still leads; allocator / cache effects show up in absolute MB/s — trust ratios, re-measure on your box.

Pitfalls

1. Microbenching tiny strings — 4 KiB finishes in microseconds; trust 1 MiB+.
2. Assuming StringIO ≈ join — measured slower here.
3. += in non-CPython — may be truly quadratic.
4. Building then decoding — pay once; don’t churn encodings in the loop.
5. Joining with a separator by accident — ",".join is a different problem (still usually better than loop += with commas).

When to pick what

Reproduce

python3 lab-evidence/49-string-concat-vs-join/results/run_lab.py

Evidence: /workspace/lab-evidence/49-string-concat-vs-join/results/.

Closing

Join wins. On this box 1 MiB list+join ~2767 MB/s vs += ~781 vs StringIO ~1113. At 32 MiB: join ~1304, += ~860, StringIO ~600 MB/s. Default to join; keep StringIO for interfaces; treat += as acceptable only for tiny or one-off appends — and measure if the loop is hot.