Benchmarking a knob you don't control
I needed to know how to configure the write path of a storage engine a colleague maintains. The answer took an afternoon. Making the answer survive questioning took the rest of the day, and that second part is the only bit worth writing down.
The shape of the problem
The engine streams rows into a write-ahead log, and periodically checkpoints: merges the accumulated log into columnar data files. Two settings govern it — how many bytes of log accumulate before a checkpoint, and how many merge tasks run concurrently. Defaults were 1 GiB and 16, the latter hardcoded rather than derived from the machine.
I wanted a production recommendation for hundred-million-row writes. What I did not want was to present a number and be unable to answer the first three questions anyone sensible would ask.
The checks, and why each exists
One corpus, generated once
Sixty-four columns — a date over two years, an id, thirty low-cardinality dimensions, thirty-two double metrics — generated once at a hundred million rows and reused for every run. Row i is a pure function of i, so "the first N rows" is a legitimate N-row dataset rather than a truncation artefact. Generation time is excluded from every figure.
This exists because otherwise you are benchmarking your data generator, and the scale comparison between 50M and 100M becomes meaningless.
Verify the output on every run
Row count checked equal to N, every time. A configuration that is fast because it dropped rows will otherwise look like your winner.
Prove your condition sets are comparable before you combine them
The sweeps ran under two different environments — different memory limits, different thread caps. Treating them as one surface is only legitimate if they agree where they overlap, so I ran the same settings under both: 8.10 against 8.06 million rows per minute. Within noise.
Without that overlap point I'd have had two datasets and no honest way to draw one conclusion from them. It costs one extra run. It is the single highest-value thing in the whole exercise.
Show that nothing was resource-bound
Peak resident memory sat between 26 and 30 GB across every combination, with no swap and hundreds of gigabytes free throughout. That's what licenses the claim that the numbers measure the settings rather than the machine — and it's the first thing a reviewer should ask.
Report the anomaly
The budget sweep is not monotonic: 24 GiB came out slightly worse than 16 GiB, against the trend. I don't have an explanation. It's in the write-up with that said plainly.
Smoothing it would have made a tidier graph and a less trustworthy one. A benchmark with no unexplained points has usually been tidied.
What it found
Checkpointing is the bottleneck, always. Between 70% and 90% of wall time in every single combination. The write phase is untouched by either knob, at a steady two-ish seconds per million rows. So every gain available is a checkpoint gain — which immediately tells you where not to spend effort.
The log budget is worth about 3×, and saturates. Raising it from 1 GiB to 32 GiB took checkpoint time from 1,993 seconds to 524.
The mechanism matters more than the number: at a small budget the log is checkpointed in many batches, and each batch re-merges against an output table that is already growing. Cost compounds super-linearly. The evidence for that being the real mechanism, rather than a story I told after the fact, is a scale comparison — at the default budget, per-million checkpoint cost rose from 12.46 seconds at 50M rows to 19.93 at 100M. Sixty percent worse for twice the data. Raise the budget and that super-linearity disappears: both scales converge at about 5.8 s/M. If batch re-merge weren't the cost, widening the batch wouldn't flatten the curve.
Concurrency is the bigger lever, and it wasn't saturated. 16 → 32 → 64 gave 8.06 → 9.73 → 10.58 million rows per minute, still climbing at the top of the range I tested. The hardcoded default of 16 badly under-uses a many-core machine.
The two knobs interact. At low concurrency, raising the budget is worth about 6%. At high concurrency it's worth 2% — because plentiful merge parallelism already absorbs most of the batch-remerge penalty the budget was fighting. Report a single-axis sweep and you will state one of those two numbers as though it were general.
The recommendation is not the maximum
Peak throughput was 10.58 M rows/min at the largest budget and highest concurrency.
The setting I recommended was neither: half that checkpoint memory, at 10.39 M rows/min — 98% of peak. On a shared machine, the second one is obviously right, and it is only visible because the sweep recorded memory alongside throughput.
"Which configuration is fastest" and "which configuration should we ship" are different questions. A benchmark that only answers the first has done about half the job.
The general version
A benchmark's purpose is not to produce a number. It is to produce a number that is still true after someone knowledgeable has attacked it.
Every check above corresponds to an attack: was it memory-bound? — no, here's the RSS across all runs. Are those two sets comparable? — yes, here's the overlap. What about that dip? — I can't explain it, it's in the data. Does this hold at other scales? — here are two, and the difference between them is what identifies the mechanism.
Write the defence into the experiment and you will usually find the defence is the finding. The two most useful results here — that concurrency dominates, and that the batch-remerge effect is real — both came out of checks I ran to withstand questions, not out of the sweep I originally planned.
The storage engine is a colleague's work; the harness, the measurements, the analysis and the recommendation are mine.