Name the properties before you choose the architecture
We had a system that applied user-defined tags to tens of millions of objects, and it was wrong often enough that nobody could say when it was right. The useful thing I did was not the design. It was refusing to write one until we had agreed what the system had to be true of.
The system
Customers of a marketing analytics platform define their own dimensions — "Funnel Stage", "Region", "Brand vs Performance" — as rules over their advertising objects. A rule is a filter plus a value: if the campaign name matches this pattern, tag it "Acquisition".
Three things make that harder than it sounds.
Objects form a hierarchy — account, campaign, ad group, ad — and tags cascade down it. Rules can depend on other rules' output, so the rule set is itself a dependency graph, not a list. And the object set changes continuously as we ingest, while the rule set changes whenever a user edits something.
The system we had did work, in the sense that tags mostly appeared. But:
- Two rules in the same group could both match an object; whichever ran last won. Delete the winner and the object didn't fall back to the other rule — it fell to "(not set)".
- Because processing ran in independent phases, users could observe intermediate states — and those intermediate states got cached downstream, so a transient "(not set)" could outlive the run that produced it.
- A failure part-way through meant the next run restarted from the last fully successful snapshot. Under sustained failure the window grew, so the system got further behind the more trouble it was in.
- There was no way to answer "is this object correctly tagged right now?" other than looking.
The incident record over three months had no pattern to it. That was the actual problem: not a bug, but the absence of any property you could rely on.
What we wrote down first
The temptation with a system like this is to start sketching. Queues here, workers there. I've since come to think that's almost always premature, because you can't evaluate a sketch without a standard to evaluate it against, so you end up choosing on aesthetics and calling it judgement.
So the specification opened with the properties, each stated concretely enough to be checkable:
- Deterministic. A given version of the rule graph always yields the same rules with the same dependencies. The processor is a pure function of (object attributes, rules) — same input, same tags, however many times it runs.
- Atomic. Either all of an object's tags persist, or none do. No object is ever half-tagged.
- Eventually causally consistent. The tag version applied across a hierarchy corresponds to one rule-graph version. A child is never tagged against a graph its parent hasn't seen.
- Isolated. One customer's or one channel's volume cannot degrade another's.
- Visible. Progress and status of every pipeline is observable while it runs, not inferable afterwards.
- Predictable. At any moment you can say which tag version is applied to a given object.
- Scalable. A new customer's data does not slow the existing workload.
- Fault-tolerant and checkpointed. Any fault short of a code bug is recoverable, and recovery resumes from the last successful point rather than the beginning.
We also wrote the non-goals, which turned out to matter just as much. We explicitly did not solve the conflicting-rules problem: if two rules both match an object and the dimension can hold one value, one of them loses, and we were not going to pretend otherwise. Writing that down stopped it being relitigated in every review.
Three architectures, and a failure-mode analysis
Only then did we draw anything. Three candidates, all of which would "work":
1. Asynchronous persist, trigger the next stage with the tags
A stage computes tags, publishes them to a durable queue for persistence, and — without waiting — publishes the tags again as a trigger for the next stage down the hierarchy. Fast, because nothing blocks on the database.
Failure analysis kills it. The output of a stage now lives in two places: the database, and a queue that is standing in for the database for as long as the child stage takes. When a tag is wrong, you cannot say where to look. You are one system away from being able to answer the question, and the answer might be in a message backlog you'd have to dump to read. It fails predictable, and it fails debuggability, which was the whole motivation.
2. Synchronous persist, then signal
Persist the stage's output, then signal the next stage that it may start. One source of truth. Causal consistency comes free: the child cannot start until the parent is durable.
Better, but the worker cannot consider itself done until the signal is published, so a persistence backlog stalls the whole pipeline behind it. It also still requires a message system whose only job is sequencing — a component that can fail and take the pipeline down with it, in exchange for coordination we might be able to get another way.
3. Independent per-stage pipelines over consistent metadata
No queue between stages at all. Each level runs as its own perpetual pipeline. Output is written synchronously, and the metadata recording which object is tagged at which version is updated in the same atomic operation. A worker starts at the top level, takes whatever visible work exists, persists it, and moves down; a child reads its parents' current tag version from metadata when it processes.
This one has an obvious weakness, and I want to state it as plainly as the specification did: correctness now rests entirely on the metadata being consistent and available. There is no second copy to fall back on. If the metadata is wrong, the system is wrong, quietly.
We took it anyway, because everything else about it was better. One datastore, so a question about a tag has exactly one place to be answered. No inter-stage component that can fail. Higher levels — the ones customers look at most, with the least data — get to the newest rule version soonest, because they aren't waiting for the deepest level to finish. And the overhead of skipping objects that haven't changed since they were last tagged is trivial.
The part worth stealing
A properties list is not documentation. It is a decision procedure, and it does most of the deciding for you.
Notice that I did not argue architecture 1 was slow, or ugly, or unfashionable. I argued it violates predictable, a property we'd written down before we knew which designs we'd be comparing. That's the whole trick: commit to the standard while you are still ignorant of which option it will favour, and the comparison stops being about taste.
It also changes what a design review is for. Reviewers who are handed a diagram argue about the diagram. Reviewers handed a properties list argue about the properties — which is the argument you actually want, because it's the one where being wrong is cheap.
The engine that came out of this is still the platform's annotation system six years later, and I wrote very little of the code. Most of it was built by the team over the following years. What survived was the list.
Validating the part most likely to break
A properties list only helps if you then go and check the assumptions it rests on. Six months later I did, and the shape of that exercise mattered as much as the design had.
I want to be precise about what it was, because it is easy to oversell. Architecture 3 put Postgres in the critical path as the work executor: each worker asks the database which objects it should claim next, and the consistency of that metadata is what correctness rests on. So the question was narrow — can that layer sustain the rate the design assumes? This was design validation of one component, not an end-to-end measurement of the tagging pipeline. Those are different numbers, and conflating them would flatter the result.
It could, once one thing was fixed. Holding everything else constant and toggling only Postgres's merge-join planner flag took the executor from 2–3k objects/sec to 17–19k — six to eight times, from one setting, isolated because nothing else moved between the two runs. Worth pausing on: the design was not wrong and the code was not slow. A query planner was choosing a join strategy that happened to be catastrophic for this access pattern, and no amount of reading my own code would have found it.
Three other things I would do the same way again:
- Treat the hardware as a variable, not a nuisance. The disk on the test box failed intermittently mid-exercise. Rather than discarding those runs or quietly averaging them in, I recorded disk condition as a column and split the results into two sections. On a degraded disk the same workload ran at 2–3.5k/sec. Without that split, a hardware fault would have been indistinguishable from a regression in my own code — and I would probably have gone looking for the bug.
- Track the curve, not a number. I measured across successive versions of the rule graph as the working set grew: 17–19k, then 15–16k, 13–14k, and eventually around 7k. A single spot measurement would have said "fast". The curve said "fast now, and here is the slope" — which is the number you actually need to plan against.
- Isolate the outlier query. One finalisation step was taking 20.5 minutes while the rest of the run cruised. Splitting it out and reworking the index it depended on brought it to 111 seconds. It would have been invisible inside an aggregate throughput figure; the only reason I saw it was that the harness timed phases separately.
That last one generalises. Aggregate throughput hides exactly the thing you most want to find, because a single pathological phase averages away against everything that is working fine.
Adapted from an internal technical specification I wrote in December 2019 and a pair of benchmark write-ups I published to the team's engineering docs in May–June 2020. Written up here in 2026. The engine that resulted was built by the team over the following years; the specification and the benchmarks are mine.