Mechanism and policy
In early 2024 I was worried the ingestion pipeline I'd founded was heading the way of the monolith it replaced — special cases accreting into the core until nobody could change it. A colleague on the team answered with a principle older than either of us, and I've built on it ever since.
The principle
Infrastructure provides mechanism: the generic capability, the knobs. Applications set policy: the values, the specific behaviour. The mechanism does not name, and does not know about, the policies that use it. It is the separation operating-system designers have argued for since the 1970s; what I was handed was the reminder that it applies just as much to a data pipeline's operators as to a kernel's scheduler.
The test that makes it operational: if a second user of this appeared tomorrow, would I have to refactor? If yes, mechanism and policy are tangled, and the fix is to find the generic capability under the specific behaviour, build that, and let the specific behaviour sit on top as data.
Where it bit first
The pipeline's filters were the immediate case. A customer's filter had been arriving as a raw SQL fragment spliced into the plan — policy and mechanism in one string, unanalysable, unquotable, and different for every customer. The mechanism underneath was "evaluate a predicate over typed columns". Once that was the thing the pipeline provided, filters became a small expression language compiled to typed column expressions, and the per-customer variety moved entirely into data.
The colleague's own warning came with it: the language will become too powerful. He was right that it's a risk; a mechanism that grows a feature per policy has simply moved the tangle. The discipline is that the mechanism grows a capability, never a case.
Where it shaped the next system
In the pipeline compiler I'm building now, the rule is stated in the repository as a design invariant, and three decisions follow from it directly.
- The meter operator. The compiler provides a step that records a measurement about a dataset as it's produced — row counts, freshness, whatever the author declares. What is measured, and what to do about it, belongs to the application that reads it. The compiler owns the mechanism; the app owns the policy. When a colleague built the operator, that sentence was the acceptance criterion.
- Runtime parameters as holes. A program declares that it needs a value under a name; the compiler never knows what the name means — a credential, a bucket, a date — only that it is a hole to be filled at run time by whoever drives the system. So the same program runs in a test, a REPL and production without the compiler learning about environments.
- Multi-tenancy as a scoped context. The most recent piece: one process serves many workspaces by deriving everything per-workspace — the metadata connection, the config — from an ambient tenant context that the caller sets and the compiler only reads. The registry of workspaces, and how it's populated, is deliberately deferred: it's policy, and building the mechanism first means it plugs in later without re-architecting.
Why it matters more in data systems
Application code has customers who complain when it's wrong. Infrastructure has engineers who work around it when it's wrong, and the workarounds become the system. A pipeline that knows about its channels, a tagging engine that knows about its customers, a scheduler that knows about a particular job's deadline — each is a small convenience that, ten years on, is the reason nothing can change. Mechanism and policy is the one rule I know that prevents that at design time rather than diagnosing it afterwards.
I credit the colleague who handed it to me every time I explain it, because the second-best thing about a good principle is being able to say where you got it.