Data infrastructure & compilers · Bengaluru
I build the thing that compiles the thing.
For a decade I've worked on the ingestion and computation layer of a marketing analytics platform spanning 80+ advertising and analytics sources. One instinct runs through all of it: take behaviour that was written as imperative code, move it into a declarative artifact, and build the compiler that turns that artifact into an executable plan.
Metric formulas compiled to database aggregation pipelines. Tagging rules compiled to a deterministic per-object work plan. Channel configuration compiled to a distributed Spark plan. Now: an EDN program compiled, through a pure pass pipeline, into a content-addressed task graph.
Each step was forced by the failure of the one before it. The most useful of those failures is written up below.
Writing
Write the budget before the fix
A seventeen-million-object daily feed was timing out a four-hour job. Ten minutes of arithmetic turned "slow" into "six seconds per request", pointed at a stall the team had lived with for seven months, and held for the next ten — while the feed grew by half.
More in the archive, including the argument for compiling rather than configuring, and the postmortem of a runtime migration I proposed and then reverted.
Elsewhere
- Writing — essays spanning the same decade as the work.
- Selected work — the systems, and the line that connects them.
- CV — two pages, prints cleanly.
- GitHub — mostly small patches to other people's projects: JRuby, Rails, rake-compiler, concurrent-ruby, Doom Emacs, flycheck.