Writing

Infrequent, and mostly about things that went wrong. Grouped by when the work happened, not when I got round to writing it up — each piece says both.

The through-line

Compile, don't configure

Ten years of the same move at rising levels of abstraction: a formula, a rule graph, a channel configuration, a channel definition, a dataflow program. Why "compiled" and not "configured" is the distinction that matters, and the three properties it buys.

A number is only worth quoting if it is a rate

680 million rows a day sounds like a capacity; it is a demand. Finding the shape under a total, the flow under a stock, and the median under a headline — on my own systems' numbers.

2016 – 2019 · expressions

Resolving metric dependency & expression with DAG & AST Clarisights blog · Apr 2019

How we let customers define their own KPIs as formulas over base metrics, and evaluated them at interactive speed: an abstract syntax tree per expression, a directed acyclic graph for the dependencies between metrics, and a resolution order derived from the graph. A single query could touch tens of millions of rows per source and perform hundreds of thousands of expression evaluations, all inside an interactive SLA.

The earliest piece of the line the rest of my work follows — written at the time, not in hindsight.

What I learned migrating a Rails platform to JRuby, and back again

The expression graph above eventually outgrew a single-threaded interpreter. I tried to fix that by changing the runtime. It measured faster and still failed, four separate ways, and we reverted after six months.

2019 – 2020 · rules

Name the properties before you choose the architecture

Redesigning a tagging engine that had no predictable failure rate. Rather than opening with a design, we wrote down the ten properties the system had to have, then put three candidate architectures against a failure-mode analysis. Two of the three eliminated themselves.

2022 – 2025 · pipelines

A day and a half to ninety minutes

A batch job whose duration exceeds its own period cannot meet a daily deadline, however much you tune it. Rewriting an ingestion pipeline where every real win came from moving work somewhere else rather than making the code faster.

Write the budget before the fix

A seventeen-million-object daily feed was timing out a four-hour job. Ten minutes of arithmetic turned "slow" into "six seconds per request", pointed at a stall the team had lived with for seven months, and held for the next ten.

Mechanism and policy

A design rule handed to me in a code review, and the one test that enforces it. How it turned raw SQL filters into a language, and shaped three decisions in the system that came after.

The second reviewer

Two migrations I led duplicated customer data three months apart — the second after I'd written the lesson from the first. Why the lesson had to become three artefacts instead of one disposition.