Write the budget before the fix
A customer's product feed had grown to seventeen million objects a day, and the job that ingested it had a four-hour timeout it was no longer meeting. Everyone had a theory. The thing that actually moved us was writing down, in one message, how fast each step was allowed to be.
The arithmetic
The job splits the feed into 500 partitions and runs eight at a time. Of the four hours, about 3.75 are available for the step that matters — matching each row to the advertising object it refers to, which we call dereference. So: 500 partitions at parallelism 8 is 62.5 rounds; 3.75 hours over 62.5 rounds is 3.6 minutes per partition; each partition makes 34 requests of a thousand rows; 3.6 minutes over 34 requests is about six seconds per request.
The median request was taking twenty.
That one line did three things. It told us the target was 3×, not 10% — so no amount of index tuning would be enough on its own. It told us where the target lived: in the request path, not the Spark side. And it turned a vague "deref is slow" into a number a fix either met or didn't.
What the budget pointed at
A stall we had been living with. Seven months earlier a colleague had plotted request durations and found them bimodal: about a tenth of a second, or about sixty. Nobody had chased the sixty. With the budget in hand it was the obvious first target, and it took a day: a metrics-reporter thread in the request path took a full minute to stop when a request finished early. Removing the reporter from that path removed the mode.
A cache that was slower than the database. Profiling the request showed the per-object cache layer making things worse, not better — direct lookups took seconds, the cached path minutes, because the cache was read one key at a time. Batched reads and writes fixed the shape of it.
A payload nobody was looking at. Each request carried about 3.5 MB of JSON, and the web framework's parameter parser was spending ~700 ms on it before our code ran. Sending the data field as a pre-serialised string, so the framework passed it through untouched, saved that on every one of ~17,500 requests a day — roughly three and a half hours of server time per run.
A batch size on the far side. Once dereference was fast, the annotation system that syncs the resulting objects became the long pole. Raising its batch from 1,000 to 50,000 took it from about 1,100 to about 6,700 objects a second.
Dereference finished in 53 minutes. The job finished in 1.1 hours.
What I got wrong on the way
I also tried to prove a load hypothesis by re-running other jobs at night and reading the speed-up as evidence. A senior colleague's reply was one line — correlation is not causation; depend on the metrics and the profile — and he was right; the profile is what found the stall. And a change I made to the cache path a month later introduced a memory leak he caught in review. Both are in the record, and both are the kind of thing a written budget doesn't protect you from. It only tells you where to look.
Did it hold?
This is the question I care about more than the 53 minutes, because a good week is easy. Over the following ten months the feed grew from eight to twelve million rows a day, with a 21.7-million-row day at the peak. The job's mean time went from 140 minutes in July to 54 by January and stayed there; after August no run touched the four-hour timeout. The budget had been written for seventeen million objects and it held at twenty-one.
The general version
Before profiling, before proposing, write the budget: how much time is there, how does it divide across the units of work, and what does that make the allowed cost of one unit. It is ten minutes of arithmetic. It converts "slow" into a number, it tells you which order of magnitude you are chasing, and it hands you the target the profile has to hit. When someone later asks whether the fix worked, the budget is also the only honest way to answer.
Job durations and volumes are from the production job and import-report tables, re-queried in August 2026. The stall and the cache findings were team work; the budget, the payload change and the batch-size change were mine.