A day and a half to ninety minutes
A five-million-row weekly ingest took about thirty-six hours. The important thing about that number isn't that it's slow. It's that a job taking thirty-six hours cannot meet a daily deadline, at any level of effort, because the next one is due before this one finishes. That reframes the work from optimisation to redesign.
What the job did
Customers connect their own data — a CSV on S3, a BigQuery table, a Google Sheet, a Parquet export from their warehouse. We pull it, and for each row work out which advertising object it refers to (a campaign, an ad) by matching on a set of dimensions, then write the result so it can be joined against the spend and performance data we already hold. Call that second step dereference. It is most of the cost.
The pipeline that did this had grown up inside a Rails application over several years. It was not badly written. It was structurally in the wrong shape, in three specific ways — and each fix turned out to be about moving work somewhere else rather than making any code faster.
1. Stop transcoding to CSV
The old pipeline required CSV. A Parquet source or a BigQuery table was therefore converted to CSV first, then read back.
This is worse than the wasted I/O suggests. Transcoding is single-threaded and does not parallelise — it's a funnel every large source has to squeeze through before any real work starts. And it discards everything the source knew about itself. Parquet is columnar and carries statistics; BigQuery will happily apply a predicate server-side. Flatten both to CSV and you have thrown away the ability to not read most of the data.
Reading each format natively meant filters could push down to the source. The fastest rows to process are the ones you never fetch, and that is not a saying — for a date-filtered job against a two-year table it is most of the work.
2. Batch the dereference
Dereference was a lookup per row. At five million rows, per-request overhead is the workload; whatever the lookup itself costs is a rounding error next to the cost of asking five million times.
We batched at a thousand rows per request, across thirty-two parallel executors.
The number isn't the point, and I'd be suspicious of anyone who presented it as one. The point is recognising which regime you're in. I'd learned the same lesson a year earlier tuning consumer batch sizes on a message queue: at 100 KB batches we were spending about 85% of round-trip time purely on networking — asking for the next batch, committing offsets — and at 500 KB that overhead fell below 30%. Same shape of problem, different system. When per-item overhead dominates, the fix is never a faster item; it's fewer, larger requests.
3. Make the retry unit small
The old job retried as a unit. A failure eighty percent of the way through a thirty-six hour run cost you thirty-six hours.
Once the data is partitioned across executors, a failure can retry a chunk. That sounds like an operational nicety and is actually a throughput change: if failures are routine — and against a dozen third-party APIs and customer-managed storage, they are — then your effective throughput includes the cost of retries. Cutting the retry unit from a day to a few minutes moved the number more than some of the direct optimisations did.
The result, and how I made myself believe it
The weekly five-million-row job went from about thirty-six hours to about ninety minutes. A 500,000-row daily job went from an hour to ten minutes on the same changes. Later, a two-year backfill of forty million rows ran in an hour and a half.
I did not trust any of it. A twentyfold speedup on a data pipeline is exactly the shape of result you get when you have quietly stopped doing some of the work.
So before merging: clone a live customer channel, run the clone through the new pipeline and the original through the old one, and compare the rendered dashboards — not row counts, not checksums over intermediate files, but the numbers a customer would actually look at. Exact match.
That comparison is the only part of this I'd insist on if you took nothing else. A performance claim you haven't tried to falsify is a rumour, and a data pipeline is the one place where "faster" and "wrong" are easy to confuse.
What I'd got wrong the previous time
Four years earlier I'd faced a related problem — work that was too slow, in a runtime that couldn't parallelise it — and I tried to solve it by changing the runtime under the whole platform. That took six months and was reverted.
The difference isn't that Spark is better than JRuby. It's that the second time I changed where the work happened — pushed filters into the source, moved lookups into batches, cut the unit of failure — rather than hoping the same work would go faster underneath. Every win listed above is of that kind. None of them made a line of processing code faster.
I don't think I'd have seen that without having got it wrong first, expensively, in public.
The pipeline reached full coverage of the relevant channels and its predecessor was retired. It was a team system; I founded it and wrote more of it than anyone else, but not most of it.