csv.reader vs split: Parse Lab

Intro — what this post promises

Parsing CSV without pandas: is naive str.split(',') worth it over csv.reader, and what does csv.DictReader cost? This lab times rows/s on a generated fixture with quoted commas on Linux localhost.

Related links:

• json dumps compact vs indent localhost lab
• pickle vs json roundtrip localhost lab
• str translate vs replace localhost lab
• glob vs rglob vs walk localhost lab
• Counter vs dict tally localhost lab
• setdefault vs defaultdict localhost lab
• bytes vs bytearray localhost lab
• perf_counter vs time localhost lab

Lab honesty (1 Oct 2026 IST): Python 3.13.5. Fixtures under lab-evidence/78-csv-reader-vs-split/results/fixture_*.csv. Affiliates: 0. No pandas.

Verdict up front (10 000 rows consume): split ~5422k/s vs csv.reader ~2345k (~2.31×) vs DictReader ~948k (reader ~2.47× DictReader). But split had the wrong field count on ~62% of rows — fast and wrong. Prefer csv.reader / DictReader.

Arms

Lab topology

rows: 1000 / 10000 / 50000; header + quoted notes with commas
metric: p50 rows/s

Script: lab-evidence/78-csv-reader-vs-split/results/run_lab.py.

Lead table — consume (p50 rows/s)

Correctness tax

Quoted "hello, ""world"" #i" style notes are normal CSV — split cannot win.

DictReader cost

Building a dict per row costs about ~2.5× vs csv.reader lists at n=10 k. File open on the same fixture was nearly free vs in-memory StringIO (mem÷file ~1.02×).

Reading it

• Speed without a parser is a trap when fields can quote commas.
• csv.reader is the correct default for row lists.
• DictReader buys named columns for ~2.5× time — often worth it in app code.
• Materializing huge lists of rows adds allocation (consume vs list gap grows at 50 k).

Why quoted fields exist

CSV exporters quote when a field contains the delimiter, quotes, or newlines. Real billing/export files do this constantly. A microbench on comma-only synthetic rows would crown split and teach the wrong lesson. This fixture forces ~60%+ arity failures so the speed story cannot hide the correctness story.

Consume vs materialize

At 50 k rows, building a Python list of every row slows csv.reader more than streaming consume. If you aggregate on the fly (counts, filters), iterate the reader without retaining all rows. DictReader amplifies that allocation cost because each row is a new dict.

Pitfalls

1. line.split(',') on vendor CSV — silent column shifts.
2. Ignoring newline='' when opening files for csv module.
3. DictReader when you only need positional fields — pay for dicts unnecessarily.
4. Assuming pandas is required — stdlib csv covers a lot.

When to pick what

Header discipline

Always decide whether the first line is a header. DictReader assumes it; csv.reader does not. Mixing next(reader) skips with split-based header parsing is a common off-by-one source when rewriting parsers. Keep one code path.

Reproduce

python3 lab-evidence/78-csv-reader-vs-split/results/run_lab.py

Evidence: /workspace/lab-evidence/78-csv-reader-vs-split/results/.

Closing

Don’t split production CSV. On this box split looked ~2.3× faster than csv.reader at 10 k rows while corrupting ~62% of records; DictReader traded ~2.5× for ergonomics. Correctness first — then measure.