Content identity, or why the compiler is pure
The most consequential decision in the pipeline compiler I'm building is one that looks wasteful: it compiles the entire program from scratch every time, and persists nothing about the compiled graph except a fingerprint per node. This is why.
The failure it is designed against
I spent years inside a platform where changing a schema meant a full recompute. Not because anyone designed it that way, but because nothing recorded what depended on what, so the only safe response to any change was to rebuild everything. The cost was not the compute. It was that change became unaffordable, so it stopped happening, and the platform ossified around whatever shape it had on the day each piece was first built. Every engineer who has worked on a mature data system knows this feeling: the migration you don't run because you can't afford to find out what it touches.
So the property I wanted first was: an edit costs only what actually changed. Everything else in the design is downstream of that.
The mechanism
An author writes a program — sources, objects, datasets as pipes of steps, destinations. The compiler lowers it through a fixed sequence of pure passes: validate, resolve, analyse, plan, then a pass that stamps every node and every field with a content identity — a Merkle hash of the node's own definition and the identities of everything it depends on. Codegen, lineage and enrichment follow.
Deployment is then a diff, not a build: compare the fresh identities against the last committed generation manifest, and only nodes whose identity changed are rebuilt. At runtime the same idea recurs one level down: a run-ledger records which (artifact × partition) cells were produced at which identity, and the executor reproduces only the dirty ones. When a run succeeds, a new generation is published and the manifest swaps atomically; readers see the old generation or the new, never a half-built one.
Why pure, and why nothing stored
The tempting shortcut is to store the compiled graph and update it incrementally. Every incremental system I've worked on drifted: stored state and source disagreed, and the disagreement was invisible until it wasn't. A pure compiler cannot drift, because there is nothing to drift — the graph is re-derived from the program each time, and the only persistent artefact is the identity, which is a function of the program. Compilation is cheap; the expensive thing is the data, and the identity is what lets us avoid touching it.
Two details show how far the principle goes. Runtime parameters — credentials, bucket names — are declared as holes in the program, and the identity hashes the hole's name, never its value, so rotating a credential never rebuilds anything. And when a colleague proposed calling a persistence step a "checkpoint" by analogy with Spark, I renamed it: Spark's checkpoint truncates lineage; this system never does. The lineage is the point.
What it cost
A clean-slate rewrite. The first version of the compiler had the identity idea but wrapped it in a column registry and a deploy state machine that turned out to be exactly the drift-prone stored state the principle forbids. In June I wrote the redesign specs and rebuilt the core in three weeks on a branch, then cut over. Most of the operators and the executor carried across; the registry and the state machine did not. Being wrong about the boundary of purity cost about three months.
It also costs a compile on every run. I measured what that actually amounts to — reopening the DuckDB file per operation, the case people worried about, was 0.1% of a run — and wrote the number into the repository next to the decision so nobody re-litigates it from intuition.
Status
Merged and running end to end in an alpha environment, with two years of one customer's data through it. Not in production, and I've stopped predicting the date. The property, though, already holds: changing a program and redeploying rebuilds what changed and nothing else, and I can show the diff that says so.