The second reviewer
Twice in three months a migration I was responsible for duplicated customer data. Both were caught, fixed and written up. But the second one happened after I had written the lesson from the first, which means the lesson was wrong — or at least, incomplete.
The first
We were giving customer-uploaded ad accounts a proper unique identifier; until then an account was whatever its name said it was, and near-duplicate names had been quietly creating duplicate accounts for years. The project ran for a quarter. On the last action item, a manual SQL update meant for inactive accounts matched active ones too. Budgets and metrics doubled for five customers overnight.
I found it the same morning, said so publicly within hours — "this happened due to me … accidentally updating all accounts instead of inactive, archived ones" — marked the duplicates inactive, re-ran the affected channels and had a customer-visible fix out the same day and a code fix two days later. The same update also touched every account's timestamp, which caused the tagging engine to re-evaluate 210 million objects and left the annotation database with billions of dead rows and an outage of its own.
The lesson I wrote that day: never perform a large-scale destructive operation without another pair of eyes reviewing it.
The second
Two months later, migrating the last family of these channels onto the new pipeline, the same class of change — an identifier strategy applied per channel — was applied to channels it didn't fit. This time there were reviewers. It still duplicated accounts for eight customers, and it took ten days and a customer's report to notice. When we fixed it, a subset duplicated again, because the list of channels to fix had been generated before the mappings it depended on were updated.
I wrote that root-cause analysis too, and this time the lesson had to be different, because "another pair of eyes" had been present and hadn't been enough.
What was actually missing
Reading the two write-ups side by side, the second reviewer was the least important of three things, and I'd put them in this order now:
- Written success criteria, before running. Not "migrate the channels" but "after this, the count of accounts per channel equals the count of distinct identifiers, and this query returns zero rows." The first migration had no such statement; the second had one that didn't cover the channels it turned out to affect. A reviewer can only check against a criterion that exists.
- A detection query, scheduled, before running. Ten days to notice is the real failure in the second incident, not the bug. A duplicate-account check now runs in production and pages the on-call; it exists because of these two weeks and it has fired since, correctly.
- A named second reviewer who has read the first two items and signs the SQL. Not a pair of eyes on a call — a name on the plan.
And one thing that isn't a rule: the note I wrote to a colleague the day after the second incident. "Consider that mistakes are unavoidable … list the scenarios of what can go wrong. After the first migration went cleanly I became confident for the rest and didn't consider the chance of failure at all." The migration that hurts is the one after the one that went well.
Why not "be more careful"
Because I was careful. Both projects had requirement docs, scoped stories and a plan. Care is a disposition; it varies with fatigue, deadline and confidence, and the second incident happened at the end of a quarter-long project under a customer deadline, when confidence was highest and fatigue was too. The three items above are artefacts. They exist or they don't, and a reviewer can ask for them by name. That is the whole difference between a lesson and a process.
Both incidents have internal root-cause analyses I wrote at the time; the quotes are from them and from the incident threads. The detection monitoring that came out of them was built by colleagues on the successor team.