Compile, don't configure
The landing page of this site says I build the thing that compiles the thing. This is the argument for why that is the right instinct in data infrastructure, and the honest account of each time it was forced on me by the failure of the level before it.
Level one: a formula (2017)
Customers of a marketing-analytics platform wanted to define their own metrics — a ratio, a conditional, a metric derived from other derived metrics. The first implementation was code: numerators and denominators as columns, a special case per kind. It couldn't express "if the campaign objective is conversions, then…", and every new shape was an engineering ticket.
So the formula became the artifact. A customer types cost / conversions or
if(dim('Objective') == 'Conversions', spend, 0); an expression parser produces a tree; and
I wrote the compiler from that tree into the database's aggregation language — conditionals to
$cond, dimension lookups to joins on the tag store, date variables to the query's range.
A year later a dependency graph over metrics let nested formulas resolve in topological order, with
cycles surfaced to the user. Rates and ratios stopped being kinds and became expressions. That
compiler is still the path a customer's formula takes.
The lesson I took: once the behaviour is an expression rather than a code path, adding capability means extending the language, and the compiler pays for every consumer at once.
Level two: a rule graph (2019–20)
Tagging rules — "if the campaign name matches this, tag it that" — had grown into a system with no predictable failure rate. The redesign made the set of rules the artifact: a versioned graph, snapshotted, so a tag on any object can be attributed to a specific version. The engine that evaluates it is a deterministic function of (object, version). The version number is the compile step; the executor is the runtime. Written up here.
Level three: a channel configuration (2022–25)
Customer data arrived through a Ruby framework where each source was a code path and each transformation was a method. The replacement I founded made a channel a configuration — source, columns, filters, dereference keys, sink — compiled into a distributed Spark plan. Twelve source types and ten processors sit behind one config surface; adding a source is a config change.
Two refinements later made the point sharper. Filters had been raw SQL strings pasted into the plan — brittle, unquotable, unanalysable; in 2025 I replaced them with a small filter language compiled to typed Spark column expressions. And the reconciliation step that repairs object hierarchies became a generator emitting multi-CTE SQL from a declared spec of levels and paths, rather than a hand-written query per shape.
Level four: a channel definition (2025)
The next attempt declared the whole channel — levels, column groups, streams, the joins between them — in a typed framework, with an s-expression DSL for stream expressions and an array-based AST for join rules, and compiled the streams into pipeline jobs. It reached staging for two channels and was paused before production when the company chose to rebuild the platform from scratch. The design survived into the next level; the code did not.
Level five: a dataflow program (2026)
The current system takes a whole program — sources, objects, datasets as pipes of steps, destinations — and lowers it through eleven pure passes into a task graph. Every node and field carries a content identity, so deployment is a diff against the last committed generation and only what changed is rebuilt. The design is argued separately. It is in alpha, not production; I say so wherever I describe it.
Why this and not configuration
"Configuration" is the word people reach for when behaviour leaves code, and it undersells what happens. A configuration is read. A program is compiled: validated, analysed, planned, fingerprinted, optimised. The difference is where the intelligence lives. A configured system pushes the hard cases back into code paths and special flags; a compiled one keeps them in passes that apply to every program equally.
Three properties follow, and they are the ones data infrastructure most lacks:
- Change becomes cheap. The customer's formula, the operator's rule, the engineer's pipeline — each is an edit to an artifact, not a deploy of a code path.
- Correctness becomes checkable. A pass can reject a cyclic metric, a rule graph can be versioned, a program can be type-checked for nullability before anything runs.
- The runtime becomes replaceable. The same formula compiled to MongoDB in 2017 could target a columnar store later; the same channel config ran on a Kubernetes Spark cluster and then on bare metal. Behaviour that lives in the artifact survives the engine underneath it.
Each level here was reached because the previous one ran out. I would not claim to have seen the ladder in 2017. I would claim that, having climbed it, I now start at the artifact.