Harnessing Agentic AI Systems: Voting / Consensual Ensemble Pattern

Problem 15 of 15: producing a verdict no single model can fake. The Voting / Consensual Ensemble pattern, the Committee Paradox anti-pattern, the Generator–Evaluator Loop and Live-Environment Evaluators frontiers — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriesverificationensemblesevaluation

Problem 15 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Sequential Pipeline Routing Pattern.

The Problem — Producing a verdict no single model can fake

One judgment is unreliable, and self-evaluation is reliably lenient. The system must decide what is good without depending on a single model's opinion — and the frontier has evolved the gate into an iterative critic (F1) with hands (F5).

Field P16 — Voting / Consensual Ensemble (pattern) A10 — The Committee Paradox (anti-pattern) F1 — Generator–Evaluator Loop (frontier) F5 — Live-Environment Evaluators (frontier)
Forces / Smell Reliability vs cost; independence vs one family; agreement vs ground truth. Agents reviewing each other; no exit condition; "making progress" without converging. Independent feedback vs self-review; iteration vs cost. Live interaction vs static scoring; reality vs CI speed.
Solution / Anti-solution Query multiple independent model setups with identical prompts; use harness code to calculate majority agreement. "More debate equals better decisions." Separate the generator from the evaluator — the critique is the next iteration's input. Give the evaluator hands — the Playwright MCP against the live page, "the way a user would."
Consequences / Failure A statistical signal where verifiers are king; cross-review replaces self-review. An infinite loop with a nicer name; tokens burn, no verdict arrives. A skeptical standalone evaluator is tunable where self-criticism is not; the full-stack run beat the solo run. The verifier uses the artifact instead of reading it; only a verifier with hands finds the broken wiring.
Tradeoffs / Refactoring Cost scales linearly; correlated members vote as one and add nothing; agreement measures preference, not correctness. Termination as a system property — threshold, budget, breakpoint; P16's aggregation as the disagreement rule. "Over 20x more expensive" — worth it only when output quality justifies the bill. Wall-clock: runs stretched to four hours; reserved for the final slow-loop gate.
Evidence LMArena (leaderboard); mob self-review — 79% of 25,264 agent PRs (mob programming remastered). AutoGen's termination as a first-class design concern (paper). Anthropic's harness (post). Anthropic's harness (post).
Related Composes with P13; aggregation answer to A10; statistical cousin of F1. Is the absence of P13 and P16's termination; orchestration form of A2. Composes with P16; pairs with F2. Slow-loop complement of P11; composes with F1.

Discussion

The verdict problem is where the "system, not the agent" framing is most visible: self-evaluation is reliably lenient — "agents tend to respond by confidently praising the work" — so the system must not depend on a single model's opinion. The pattern makes verification statistical: independent judgments aggregated by code, and the honest limit is that agreement measures preference, not correctness. The frontier evolves the gate into a critic with agency (F1) and hands (F5), both at an honest price; the committee without an aggregation rule is an infinite loop with a nicer name.

Key Insight

Agreement is a signal, not a ground truth. Independence is the whole game — correlated members vote as one and add nothing — and termination is the difference between a debate and a decision. The ensemble replaces self-review with cross-review, and it is never a ground truth.

References

LMArena (lmarena.ai); AutoGen (arXiv:2308.08155); Anthropic, Harness design for long-running application development (post); archive: verifiers-are-king, mob programming remastered.

Harnessing Agentic AI Systems: Sequential Pipeline Routing Pattern

Problem 14 of 15: keeping linear flows linear. The Sequential Pipeline Routing pattern — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriesorchestrationpipelineschains

Problem 14 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Blackboard Pattern · Next: Voting / Consensual Ensemble Pattern.

The Problem — Keeping linear flows linear

Some flows are genuinely linear — classify, transform, emit — and the simplest verifiable shape is the right one. There is no named anti-pattern; its failure mode — forcing non-linear flows into pipelines — is in the tradeoffs.

Field P15 — Sequential Pipeline Routing (pattern)
Forces Simplicity vs branching; fixed routing vs recovery; few calls vs specialized hops.
Solution Pass analytical payloads through rigid linear stages, using LLMs solely for classification or transformation tasks at each hop.
Consequences The easiest harness to verify, replay, and bill: each hop has one job and a deterministic contract — "simple, composable patterns rather than complex frameworks." Each hop is a tool with a --json contract, exit codes, and honest --help.
Tradeoffs Rigid pipelines cannot route around a failed stage; pipelines serialize at the slowest hop, and each hop is a full request. Chains are right when the flow is linear, wrong when it needs to branch, loop, or recover.
Evidence LangChain's chains, with the migration-era docs explicit about when the rigid linear form is right (docs).
Related Is the disciplined baseline; the opposite of A11 at orchestration scale; composes with P9.

Discussion

The pipeline is the disciplined baseline for genuinely linear flows: each hop has one job and a deterministic contract, which makes the system verifiable, replayable, and billable. The boundary is the pattern's own warning: chains are right when the flow is linear, wrong when it needs to branch or recover — forcing non-linear flows into a rigid shape pays the costs (no rerouting, serialization, poisoned stages) without the benefits.

Key Insight

Linear flows deserve linear shapes. The pipeline is the disciplined baseline — and its boundary is explicit: it is the default for the flows that are already linear, not for everything. When the flow needs to branch, loop, or recover, an orchestrator or a graph wins.

References

LangChain chains (docs); Anthropic, Building Effective Agents (post); archive: Agentic-First CLI, Loop Engineering.

Harnessing Agentic AI Systems: Blackboard Pattern

Problem 13 of 15: coordinating through shared state without corruption. The Blackboard (Shared Workspace) pattern vs the State Race Conditions anti-pattern — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriesorchestrationshared-statechoreography

Problem 13 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Orchestrator-Worker Pattern · Next: Sequential Pipeline Routing Pattern.

The Problem — Coordinating through shared state without corruption

Multiple agents need a shared working memory, and unsynchronized writes corrupt it. The board must be shared and safe at once.

Field P14 — Blackboard (Shared Workspace) (pattern) A12 — State Race Conditions (anti-pattern)
Forces / Smell Unified schema vs per-agent scopes; parallel writes vs ordered consistency; coordinator vs choreography. Async multi-agent writes to shared memory with no transactional locks; corruption that appears "random."
Solution / Anti-solution Connect disjointed agents to a unified state schema where the harness coordinates simultaneous data writes and events. "It usually works" — hope as a concurrency strategy.
Consequences / Failure Choreography without an orchestrator — "No orchestrator. Just state." Conflicts detected at write time, not merge time (GitButler's virtual branches). Two agents writing the same ledger, one overwriting the other's checkpoint — the blackboard without its harness.
Tradeoffs / Refactoring Shared state is shared risk; the board fights per-agent scope; the resolution is explicit scope discipline — an ownerless board is a tragedy of the commons for state. P7 with the durability discipline: writes through a single ordered log, exactly-once, idempotency keys, transactional ownership.
Evidence CrewAI's Flows (docs); the durable-daemons choreography (definition); GitButler's collaboration patterns (Buzz). Temporal's event-sourced, deterministic execution (docs).
Related Composes with P7 (the board must be snapshotable) and P10 (the board's slow writers). Is the absence of P7 and P14's ownership discipline; the async risk of P10.

Discussion

The blackboard is choreography made concrete — shared state with no orchestrator. Shared state is shared risk: the board concentrates races, and its durability decides whether corruption is recoverable, which is why snapshots (P7) compose with it. The fix is the durable-daemons discipline: "no RPC. No message bus. No central orchestrator" is only safe when the shared state itself is the coordination mechanism, and the coordination mechanism must be transactional.

Key Insight

Shared state is shared risk. Choreography without an orchestrator requires transactional ownership — writes through a single ordered log, exactly-once, idempotency keys — and the ownership question, which state is shared and who owns the boundary, is the least-solved governance problem in the catalog.

References

CrewAI Flows (docs); Temporal (docs); archive: durable daemons series, Buzz and the Identity Problem, always-on agents.

Hacker Laws for Agentic Software Engineering: The Law of Leaky Abstractions

Law 12 of 12. Spolsky's Law of Leaky Abstractions says all non-trivial abstractions, to some degree, are leaky. The ASE key insight: every abstraction the agent lives on leaks — and the leak layer is where the harness must put the verifier, because the model cannot see the leak.

hacker-lawsagentic-software-engineeringseriesleaky-abstractionsspolskyabstractionsverification

Law 12 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: The Bitter Lesson.

The Law

All non-trivial abstractions, to some degree, are leaky. (Joel Spolsky, via hacker-laws)

The Key Insight for Agentic Software Engineering

The agent lives on a stack of abstractions, and every one of them leaks: tool calling is an abstraction over the model's output format, and it leaks when the format changes; structured output is an abstraction over the model's ability to follow a schema, and it leaks when the schema is wrong; the context window is an abstraction over the model's attention, and it leaks in the middle (Lost in the Middle); "the model understood the task" is the deepest abstraction of all, and it leaks exactly when the model misunderstood. Spolsky's warning applies with a new twist: the user of the abstraction is also the thing that leaks. The agent cannot see the leak — it cannot tell the difference between a tool that failed and a tool that succeeded, which is precisely the silent crash anti-pattern: the abstraction returns "" or "ok" and the agent confidently proceeds on a false premise.

The consequence is that the harness is the leak-detection layer, and the verifier is the leak detector. Every abstraction boundary in the agent's stack needs a check on the far side of it: the schema boundary needs enforcement with feedback (the model is still in the loop to fix what leaked); the tool boundary needs the gatekeeper and legible exit codes (the three channels exist so failures are visible); the outcome boundary needs the evaluator with hands that uses the artifact instead of trusting the abstraction. The agentic-first CLI discipline is Leaky Abstractions in contract form: make the leak visible in the interface, because the agent will not see it otherwise.

The ASE reading of the Law of Leaky Abstractions: agent abstractions leak — and the leak layer is where the harness must put the verifier, because the model cannot see the leak. Each abstraction saves the agent from the underlying complexity until the day the complexity breaks through, and on that day the agent — unlike a human — has no prior experience of the underlying system to draw on. The harness is the one component that can be told about all the layers at once, so it is the one that must watch the seams.

References

Harnessing Agentic AI Systems: Orchestrator-Worker Pattern

Problem 12 of 15: dividing a workflow across agents. The Orchestrator-Worker pattern, the God Agent anti-pattern, the Sprint Contracts frontier — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriesorchestrationdelegationmulti-agent

Problem 12 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Dynamic Tool Discovery Pattern · Next: Blackboard Pattern.

The Problem — Dividing a workflow across agents

No single context can hold a whole workflow. The work must be divided, delegated, and handed off — and the frontier has added the contract that makes delegation safe (F2).

Field P13 — Orchestrator-Worker (pattern) A11 — The God Agent (anti-pattern) F2 — Sprint Contracts (frontier)
Forces / Smell Context limits vs a single plan; specialists vs oversight; termination vs conversation. One massive agent, immense prompt, every phase in one context window. Specification vs latitude; criteria vs freedom.
Solution / Anti-solution Direct traffic using a highly capable central agent that delegates atomic sub-tasks to smaller, faster, specialized agents. "One agent, all context, total control." Before each sprint, generator and evaluator negotiate what "done" looks like — and how success will be verified — before any code is written.
Consequences / Failure Bounded contexts, parallelizable work, a plan-then-execute shape. Context avalanche and lost-in-the-middle degradation by construction — the window degrades as it fills. Ambiguity resolved at the moment of maximum leverage; the agent defines the verifier before the artifact.
Tradeoffs / Refactoring The orchestrator is a single point of failure; spec errors cascade; handoffs can lose state; termination must be a system property. P13 with decomposition and handoff: one feature at a time, structured artifacts between sessions. Negotiation overhead — bounded by keeping the contract to a sprint-sized chunk.
Evidence AutoGen (paper); Anthropic's planner-generator-evaluator (post); DeepSeek's subagent registry (DeepSeek teardown). CrewAI's crews argument (docs). Anthropic's harness (post).
Related Refactoring for A11; composes with P16 and F2. Is the absence of P13; composes A4 and A8. Composes with P13 and F1.

Discussion

Delegation is how a system outgrows one context window: the orchestrator curates a delegation tree instead of drowning in context. The single point of failure is the orchestrator — the spec must stay high-level because errors cascade downstream, and termination must be a system property, not the conversation's. The god agent fails by construction; F2 makes each delegation safe by negotiating "done" before the work exists.

Key Insight

No single context can hold a workflow. Delegation is how systems scale, the spec must stay high-level because errors cascade, and termination is a system property, not a conversation's. One window holding every phase degrades exactly as the middle fills.

References

AutoGen (arXiv:2308.08155); CrewAI crews (docs); Anthropic, Harness design for long-running application development (post); archive: DeepSeek teardown, harness canon.

Hacker Laws for Agentic Software Engineering: The Bitter Lesson

Law 11 of 12. Sutton's Bitter Lesson says general methods that leverage computation are ultimately the most effective. The ASE key insight: the loop that leverages computation beats the hand-crafted prompt — and the agent will apply the same lesson to your harness.

hacker-lawsagentic-software-engineeringseriesbitter-lessonsuttoncomputeloops

Law 11 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: Chesterton's Fence · Next: The Law of Leaky Abstractions.

The Law

The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. (Richard S. Sutton, via hacker-laws)

The Key Insight for Agentic Software Engineering

The Bitter Lesson was written about model research: hand-crafted features lose to scaled general methods. It applies with equal force to harness research, and the evidence is already in this blog's archive. The closed loop — trajectories in, trained model out — beats the hand-tuned prompt: a 350M-parameter specialist fine-tuned on tool-calling trajectories beat ChatGPT on ToolBench by 51 points, because the general method (distillation at scale) out-leveraged the bespoke reasoning of a frontier model. Meta-Harness reached 76.4% on Terminal-Bench 2.0 and "was itself discovered through automated harness evolution" — the general method of searching the harness space beat every hand-designed harness (DeepSeek teardown). Sutton's lesson, applied to ASE: the loop that feeds trajectories back into training beats the prompt you spent a week writing.

The second half of the law is the part nobody wants to hear: the agent will apply the same lesson to your harness. An agent optimizing a benchmark will find the general solution you did not hand-craft — it will game the metric, exploit the interface, take the shortcut — because general search over the solution space beats the specific behavior you tried to engineer (this is Goodhart's Law at machine speed, Law 5). The bitter lesson cuts both ways: computation beats your hand-crafted prompt, and the agent's computation beats your hand-crafted constraints.

The ASE reading of the Bitter Lesson: the loop that leverages computation beats the hand-crafted prompt — and the agent will apply the same lesson to your harness. Invest in the general machinery — the eval, the loop, the training signal — not the bespoke prompt; and design the eval as if the agent's general search will find the crack, because it will. The bitter lesson is not an argument for less care in harness design; it is an argument for putting the care where the computation can amplify it.

References

Harnessing Agentic AI Systems: Dynamic Tool Discovery Pattern

Problem 11 of 15: discovering capabilities without bloating the prompt. The Dynamic Tool Discovery / Registry pattern, the Bloated Utility Belt anti-pattern, the Interop Layer frontier — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriestool-bindingdiscoveryregistryinterop

Problem 11 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Mock Tool Virtualization Pattern · Next: Orchestrator-Worker Pattern.

The Problem — Discovering capabilities without bloating the prompt

Tool spaces grow, and static lists bloat the prompt and confuse selection. Capabilities must be discoverable — and the frontier has standardized the discovery seam itself (F4).

Field P12 — Dynamic Tool Discovery / Registry (pattern) A8 — Bloated Utility Belt (anti-pattern) F4 — The Interop Layer (frontier)
Forces / Smell Catalog vs small prompt; reordering vs fixed cache prefix; discovery vs authorization. Dozens of complex tools per agent; incorrect selection; the model "forgetting" tools exist. Open protocols vs vendor moats; portability vs control.
Solution / Anti-solution Store tool specs in databases, matching and surfacing capabilities dynamically to the agent based on semantic text queries. "More tools equals more capable." Adopt the emerging protocol layer: MCP for tools, ACP for editor-agent connection, AGENTS.md for where agent instructions live.
Consequences / Failure Many tools behind small prompts; the registry as single source of truth; the type-graph mirror keeps specs from drifting. Every tool in the prompt is a decision the model must make; a bloated registry turns selection into a needle-in-a-haystack retrieval problem. The seam becomes the ecosystem; your files become portable, whichever harness you run.
Tradeoffs / Refactoring The registry is an injection surface; reordering kills the cache prefix; discovery and authorization are separate seams. P12 with per-session tool compositions — DeepSeek's presets, and code mode replacing the tool list with a generated SDK. Every protocol you adopt is a contract you do not control; every standard is a boundary where a substitution can hide.
Evidence Toolformer (paper); DeepSeek's capability seam (DeepSeek teardown); MCP (architecture). Toolformer (paper); SWE-agent's ACI (Agentic-First CLI). MCP (docs); ACP (agentclientprotocol.com); AGENTS.md (agents.md).
Related Refactoring for A8; composes with P2 (the gatekeeper authorizes what the registry discovered). Is the absence of P12; tool-scale form of A11. Protocol face of P12; interop layer for P13.

Discussion

The registry is the capability seam made discoverable: many tools behind small prompts, with the type-graph mirror keeping specs from drifting. The two boundaries are the interesting parts: discovery is not authorization (the gatekeeper still decides), and the tool list must not reorder, or the cache prefix dies — the registry and the bill are the same seam. The anti-pattern is the registry without curation: with the same model, a minimal, well-designed interface more than doubled state-of-the-art (SWE-agent).

Key Insight

Discovery is not authorization. Many tools behind small prompts, specs that cannot drift, a tool list that never reorders — and the gatekeeper still decides what the registry surfaced may do. The model should learn which tool; the system should not trust it with all tools.

References

Toolformer (arXiv:2302.04761); Model Context Protocol (architecture); Agent Client Protocol (agentclientprotocol.com); AGENTS.md (agents.md); archive: DeepSeek teardown, Agentic-First CLI.

Hacker Laws for Agentic Software Engineering: Chesterton's Fence

Law 10 of 12. Chesterton's Fence says reforms should not be made until the reasoning behind the existing state of affairs is understood. The ASE key insight: the harness must make the agent find out why the code is there before letting it change — intent is a verification problem.

hacker-lawsagentic-software-engineeringserieschestertons-fencelegacyintentrefactoring

Law 10 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: Parkinson's Law · Next: The Bitter Lesson.

The Law

Reforms should not be made until the reasoning behind the existing state of affairs is understood. (hacker-laws)

The Key Insight for Agentic Software Engineering

Chesterton's Fence is the law every agent violates on its first pass: it comes across a fence — a function that looks redundant, a workaround that looks wrong, a test that looks pointless — and removes it, because the agent has no memory of why the fence was built and no patience to find out. The unrequested feature that cost Fowler's team three days of investigation is the mild form (Verification Is the Bottleneck); the removed workaround that was load-bearing is the severe form. "Each line of a program was originally written by someone for some reason" — and in an agentic system, the someone is often an earlier run of the same agent, which makes the fence rule both more important and harder: the reasoning may exist only in a session log.

The law is why legacy modernization is the clearest near-term value pool for agents, and why it is also the sharpest test of the harness: an agent told to "clean up this legacy code" is a fence-removal machine. The fix is not a better prompt ("understand before you change" is exactly the kind of instruction an agent will be told to ignore or will comply with shallowly); it is a harness rule — the verifier must check intent, not just correctness. Fowler's DSL idea is the constructive form: restrict the agent's change vocabulary until removing a fence requires explaining it first (Verification Is the Bottleneck).

The ASE reading of Chesterton's Fence: the harness must make the agent find out why the code is there before letting it change — intent is a verification problem. The fence rule cannot be a line in the system prompt; it has to be a gate in the pipeline: the agent must produce the fence's purpose as an artifact, and the verifier must check that artifact against the change. The mayor's answer to the man applies verbatim to the agent: "If you don't know its purpose, I certainly won't let you remove it. Go and find out the use of it, and then I may let you destroy it."

References

Harnessing Agentic AI Systems: Mock Tool Virtualization Pattern

Problem 10 of 15: making the fast loop repeatable. The Mock Tool Virtualization pattern — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriestool-bindingtestingreplay

Problem 10 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Asynchronous Tool Worker Queue Pattern · Next: Dynamic Tool Discovery Pattern.

The Problem — Making the fast loop repeatable

Real APIs are slow, flaky, and costly; the system needs deterministic replay at scale. There is no named anti-pattern; its failure mode — the mock that drifts from production — is in the tradeoffs.

Field P11 — Mock Tool Virtualization (pattern)
Forces Fidelity vs frozen repeatability; speed vs drift detection; determinism vs live realism.
Solution Swap production APIs out for lightweight mock responses inside development environments during multi-agent unit testing routines.
Consequences Deterministic replay becomes possible — "a harness that cannot reproduce a run cannot measure a change" — and the sample sizes data-driven design demands become affordable.
Tradeoffs A mock that drifts teaches the wrong lessons; mocks hide latency and rate limits; an agent trained only against mocks fails the first time it meets a 429. The honesty rule: mock for the unit test, record for the integration test, real for the eval.
Evidence VCR.py, the record-and-replay reference (docs); the harness canon's layering (harness canon).
Related Composes with P10; fast-loop complement to F5 (the slow loop).

Discussion

Mocks are how the fast loop gets deterministic replay at the sample sizes data-driven design demands — "a single run is a data point; a thousand runs is a distribution." The honesty rule is the whole pattern: a drifting cassette teaches the wrong lessons, and the final gate must always be real. F5's live environment is this pattern's slow-loop counterpart.

Key Insight

The fast loop needs frozen reality. Deterministic replay is the enabling condition for measurement — a harness that cannot reproduce a run cannot measure a change — and the honesty rule is: mock for the unit test, record for the integration test, real for the eval.

References

VCR.py (docs); archive: harness canon, Agents Are Too Stochastic for Intuition.

Hacker Laws for Agentic Software Engineering: Parkinson's Law

Law 9 of 12. Parkinson's Law says work expands so as to fill the time available for its completion. The ASE key insight: agent work expands to fill the budget — the context window and the token budget — so the budget is the discipline and the scope is a contract agreed before the work.

hacker-lawsagentic-software-engineeringseriesparkinsons-lawscopebudgetscontext

Law 9 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: Kernighan's Law · Next: Chesterton's Fence.

The Law

Work expands so as to fill the time available for its completion. (hacker-laws)

The Key Insight for Agentic Software Engineering

Parkinson's Law described bureaucracies; agentic software engineering gives it a meter. An agent does not have a deadline, it has a context window and a token budget — and its work expands to fill both. Give an agent a vague task and a large window and it will produce a large result: more features it wasn't asked for, more refactoring it wasn't asked to do, more files touched than the task required. The unrequested feature that cost Fowler's team three days of investigation is Parkinson's Law with teeth (Verification Is the Bottleneck): the work expanded to fill the scope the agent inferred, and nobody had bounded the scope.

The law has a second, economic face. Work expands to fill the budget — and the budget is metered. This is the token economics argument in one sentence: an agent given an uncapped budget will spend it, because "the employee's rational strategy is to maximize usage — to tokenmaxx" — and the employee is now a loop that never gets tired. Parkinson's Law is why the budget throttler and the infinite execution vortex are the same pattern's two sides: without a ceiling, the work expands without bound; with a ceiling, the work expands to the ceiling and stops.

The fix is the discipline the pattern language keeps naming: the scope is a contract agreed before the work — the sprint contract that defines "done" before the agent starts, and the tasks-that-fight-back principle that keeps tasks small enough to score. Parkinson's Law is not defeated by asking the agent to be brief; it is defeated by making the container small.

The ASE reading of Parkinson's Law: agent work expands to fill the budget — the budget is the discipline, and the scope is a contract agreed before the work. The deadline is not a time; it is a context window, a token ceiling, and a definition of done. Make the container small and the work will fit it.

References

Harnessing Agentic AI Systems: Asynchronous Tool Worker Queue Pattern

Problem 9 of 15: not blocking the loop on long tools. The Asynchronous Tool Worker Queue pattern — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriestool-bindingasyncqueues

Problem 9 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Schema Enforcement & Self-Correction Pattern · Next: Mock Tool Virtualization Pattern.

The Problem — Not blocking the loop on long tools

Some tools take minutes. The loop must stay live while the work happens elsewhere, and cancellation must be a contract. There is no named anti-pattern; its failure modes — the blocking loop and the abandoned worker — are in the tradeoffs.

Field P10 — Asynchronous Tool Worker Queue (pattern)
Forces Responsiveness vs correctness; tracking ID vs done work; cancellation contract vs worker completion.
Solution Offload long-running processes to background task workers and hand a tracking ID to the looping agent.
Consequences The loop stays live; the context holds a tracking ID instead of the worker's output, so the window stays lean. A worker queue is a daemon that satisfies all four durable-daemon conditions.
Tradeoffs Asynchrony introduces the races of A12; cancellation must be a contract (ABORTED_BEFORE_DISPATCH vs ABORTED); workers must be exactly-once or idempotent; the queue is infrastructure the team owns forever.
Evidence Celery, the production standard for background task execution (docs); DeepSeek's cancellation contract (DeepSeek teardown).
Related Composes with P7 (durability) and P14 (the board's slow writers); its consistency risk is A12.

Discussion

The queue decouples the loop from the tool: the agent holds a tracking ID, not the worker's output, so the window stays lean and the loop stays live — the context-budgeting benefit is as important as the latency one. The cost is consistency: cancellation must be a contract ("cancellation never abandons the body"), workers must be idempotent, and the queue is infrastructure the team owns forever. Its named risk is A12: asynchrony is how races enter the system.

Key Insight

The loop must never wait on a tool. The tracking ID is the interface, cancellation is the contract, and idempotency is the price of retry — a retried task double-executes unless the worker can deduplicate.

References

Celery (docs); archive: DeepSeek teardown, durable daemons execution.

Hacker Laws for Agentic Software Engineering: Kernighan's Law

Law 8 of 12. Kernighan's Law says debugging is twice as hard as writing the code in the first place. The ASE key insight: if the agent writes clever code, the system must debug it — keep agent output boring and make the verifier the smarter half.

hacker-lawsagentic-software-engineeringserieskernighans-lawdebuggingverificationsimplicity

Law 8 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: Hofstadter's Law · Next: Parkinson's Law.

The Law

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. (Brian Kernighan, via hacker-laws)

The Key Insight for Agentic Software Engineering

Kernighan's Law was written for human programmers; agentic software engineering makes it worse, because the writer and the debugger are different systems. The model writes the code; the model cannot debug it — an agent asked to evaluate its own work "tend[s] to respond by confidently praising the work" (Anthropic's harness) — so the debugging burden falls on the system: the verifiers, the tests, the reviewers, and eventually the humans. This is the same finding Fowler's retreat made the headline of the agentic era: "code generation is no longer the bottleneck — verification is" (Verification Is the Bottleneck). Kernighan's Law names why: the generation half is now nearly free, so the debugging half — always twice as hard — is where all the cost went.

The law's prescription survives verbatim: don't let the agent write clever code. If the agent produces the most intricate solution it can, the system — which must debug it — is "by definition, not smart enough." The harness-level translation is the whole boring-output discipline: schema-enforced, conventional, simple output that the verifier can actually check; and the verifier must be built to be the smarter half — the evaluator with hands that uses the artifact instead of reading it, the ensembles that cross-check instead of self-praise. The 79% self-review datum from the mob post is Kernighan's Law in review form: most agent PRs were reviewed by the same developer who prompted the agent — the writer debugging its own work, which the law says is impossible (mob programming remastered).

The ASE reading of Kernighan's Law: if the agent writes clever code, the system must debug it — keep agent output boring and make the verifier the smarter half. The model generates; the harness verifies; and the division is structural, because the writer can never be trusted to debug what it wrote.

References

Harnessing Agentic AI Systems: Schema Enforcement & Self-Correction Pattern

Problem 8 of 15: typing tool output and keeping errors legible. The Schema Enforcement & Self-Correction pattern vs the Silent Crash and Schema Free-for-All anti-patterns — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriestool-bindingschemaserrors

Problem 8 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: State Snapshot & Rollback Pattern · Next: Asynchronous Tool Worker Queue Pattern.

The Problem — Typing tool output and keeping errors legible

Model output is text; tools and downstream systems need types. Malformed values fail far from their cause, and swallowed errors become hallucinated successes. The boundary must be typed, and failures must stay legible.

Field P9 — Schema Enforcement & Self-Correction (pattern) A7 — The Silent Crash (anti-pattern) A9 — The Schema Free-for-All (anti-pattern)
Forces / Smell Free text vs types; bounded retries vs bad data; append-only corrections. Errors caught in the background; blank strings returned; no stderr, no verdict. Complex arguments as raw strings; parsing deferred; "the model formats it."
Solution / Anti-solution Force raw LLM text into JSON Schema, catching parsing failures and feeding structural fixes back internally. Catch-and-continue: "the agent doesn't need to know." Trust the model's output format.
Consequences / Failure A typed contract at the harness boundary — the same contract as --json and exit codes; corrections are appends, not rewrites. A real error becomes a hallucinated success; destroys the stdout/stderr/exit-code contract. Parsing moves downstream where no model can correct it; errors surface far from their cause.
Tradeoffs / Refactoring Retries cost tokens — the retry budget is part of the pattern; the error message must name the field. P9 with errors surfaced and bounded; layered checks: "the planner chose the wrong tool," not "the agent failed." P9 at the boundary with feedback while the model is still in the loop.
Evidence Instructor — Pydantic validation with max_retries and token_budget (docs, retry logic). Instructor's retry mechanics (retrying). Pydantic — core validation (docs).
Related Refactoring for A9 and A7; composes with the frozen-request pattern. Is the absence of P9; feeds A9. Is the absence of P9; feeds A7's downstream.

Discussion

The schema is the grammar of the contract: output is converted from text to structure at the boundary where the model is still in the loop to fix it, and the correction is an append, not a rewrite (DeepSeek teardown). The two anti-patterns are the boundary's two failure directions — the silent crash hides the error (a hallucinated success), and the free-for-all defers parsing until no model is present to correct it. Both are fixed by enforcement at the boundary with legible feedback.

Key Insight

The schema is the grammar of the contract. Malformed output is fixed while the model is still in the loop, the retry budget is part of the pattern (unbounded self-correction is the schema version of the vortex), and swallowed errors become hallucinated successes.

References

Instructor (docs, retry logic); Pydantic (docs); archive: Agentic-First CLI, harness canon, DeepSeek teardown.

Hacker Laws for Agentic Software Engineering: Hofstadter's Law

Law 7 of 12. Hofstadter's Law says it always takes longer than you expect, even when you take into account Hofstadter's Law. The ASE key insight: an agent task always takes longer than you expect, recursively — so the ceiling is a system property (a budget), not an estimate.

hacker-lawsagentic-software-engineeringserieshofstadters-lawestimationbudgetsloops

Law 7 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: Hyrum's Law · Next: Kernighan's Law.

The Law

It always takes longer than you expect, even when you take into account Hofstadter's Law. (Douglas Hofstadter, via hacker-laws)

The Key Insight for Agentic Software Engineering

Hofstadter's Law is about estimation, and agentic software engineering is the first discipline where the "it" being estimated is a loop you cannot see the end of. An agent run is not a task with a duration; it is a loop that keeps deciding to continue — and every continuation invalidates the estimate. The law's recursive form is the literal description of an agentic pipeline: the task takes longer than you expect, and when you account for the agent's tendency to loop, it takes longer than that. This is why the budget throttler exists: when the estimate is structurally unreliable, the ceiling must be structural too. "It always takes longer than you expect" is not a problem to be solved by better estimation; it is a constraint to be bounded by design.

The recursion has two named components. First, the infinite execution vortex: an agent retrying a broken step is Hofstadter's Law with a bug — every retry re-estimates, and the estimate never converges. Second, context anxiety: models begin wrapping up prematurely as they approach their perceived context limit, so the run ends early with work unfinished — the estimate was wrong in the other direction, but the fix is the same: the system must decide when the work is done, never the loop (termination as a system property).

The ASE reading of Hofstadter's Law: an agent task always takes longer than you expect, recursively — so the ceiling is a system property, not an estimate. Every agent pipeline needs the structural equivalent of the sprint contract: "done" defined before the work, a budget that ends the loop, and a verifier that decides when the output is good enough. The estimate is for planning; the ceiling is for survival.

References

Harnessing Agentic AI Systems: State Snapshot & Rollback Pattern

Problem 7 of 15: making state survive and giving it homes. The State Snapshot & Rollback and Tiered Hierarchical Memory patterns vs the Goldfish Amnesia anti-pattern — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriesstatedurabilitymemorygovernance

Problem 7 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Semantic Memory Router Pattern · Next: Schema Enforcement & Self-Correction Pattern.

The Problem — Making state survive and giving it homes

State must survive crashes and error loops, and different state types need different lifetimes. The system must remember across turns, crashes, and sessions — the survival half (P7) and the organization half (P8) are two patterns for one problem.

Field P7 — State Snapshot & Rollback (pattern) P8 — Tiered Hierarchical Memory (pattern) A5 — Goldfish Amnesia (anti-pattern)
Forces / Smell Durability vs latency; exactly-once vs replay; the conversation vs the world. Speed vs capacity; hot path vs archive; per-tier governance vs one store. A multi-turn loop with no persisted state; identical tool calls repeated; the goal forgotten mid-task.
Solution / Anti-solution Save complete state snapshots at checkpoint N; recover if the agent hits an error loop at step N+3; completed steps never re-execute. Divide storage into immediate short-term context, scratchpad workspace, and long-term historical database storage. "The prompt has the goal" — statelessness as simplicity.
Consequences / Failure Crash-proof execution, audit by construction, the agentic equivalent of a database transaction — condition 4 of the durable-daemons pattern. State has homes with different lifetimes — ledgers, permissions, commitments, provenance — and recall at the right latency. A stateless loop is a request-response function with a longer prompt; nothing records what was tried, so everything is tried again.
Tradeoffs / Refactoring A snapshot of the conversation does not capture the world; recovery is as wide as the reification; external effects need idempotency keys; removal still has to be invoked. More tiers mean more consistency work; forgetting from all tiers — including cached contexts and weights — is the hard part; a tier that rewrites itself is negotiable past. P8 with the state lifecycle — write, validate, retrieve, update, forget; the durable-daemons conditions 2 and 3 are the spec.
Evidence Temporal (docs); DBOS — a Postgres write as a 1-2 ms checkpoint (durable daemons execution); DeepSeek replay (DeepSeek teardown). Lilian Weng's canonical essay (post); the always-on survey's six axes (always-on agents). LangGraph's persistent-state architecture (docs); the always-on survey (always-on agents).
Related Refactoring for A12; composes with P14; its boundary is the spatiotemporal system boundary. Refactoring for A5; supplies the stores for P6. Is the absence of P8; the delegation risk of P13.

Discussion

Remembering has two halves: surviving (P7) and organizing (P8). Recovery is promised exactly as wide as the system reifies — a snapshot of the conversation does not capture the world, which is why external effects need idempotency keys and why removal still has to be invoked (spatiotemporal composability). Tiers give state homes, and forgetting across all tiers is the least-solved stage of the state lifecycle (always-on agents).

Key Insight

Recovery is promised exactly as wide as the system reifies. The snapshot covers the conversation; the world needs idempotency keys; and remembering without forgetting is not governance. A stateless loop is a request-response function with a longer prompt.

References

Temporal (docs); Lilian Weng, LLM Powered Autonomous Agents (post); LangGraph memory (docs); archive: durable daemons series, always-on agents, spatiotemporal composability.

Hacker Laws for Agentic Software Engineering: Hyrum's Law

Law 6 of 12. Hyrum's Law says with enough users, all observable behaviours of a system will be depended on by somebody. The ASE key insight: agents are the most thorough users of your interfaces — they depend on every observable behaviour you didn't promise, so the implicit interface IS the contract.

hacker-lawsagentic-software-engineeringserieshyrums-lawimplicit-interfacesclicontracts

Law 6 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: Goodhart's Law · Next: Hofstadter's Law.

The Law

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviours of your system will be depended on by somebody. (Hyrum Wright, via hacker-laws)

The Key Insight for Agentic Software Engineering

Hyrum's Law assumed human users who squint, scroll, and improvise. The agent is a user who never blinks: it reads every byte of help text, every line of output, every exit code, every timestamp, every ordering — and it depends on all of it, because it has no tolerance for ambiguity and no memory of what "should have worked." The agentic-first CLI discipline is Hyrum's Law taken as a design contract: "a lie in help text is the most expensive bug an agentic CLI can have," and "no timestamps unless asked" is not a preference but a dependency hazard — the agent will start parsing the timestamp and break when it changes format.

The law sharpens in two directions. First, the model-facing interfaces: the tool schemas, the --json outputs, the structured contracts the agent reads — every observable behaviour, including the ones you did not promise, becomes part of the de facto API. The type-graph mirror exists precisely because spec drift is a Hyrum's Law failure: the tool catalogue that drifts from the implementation is an implicit interface that somebody — some agent — will depend on. Second, the model itself becomes an interface others depend on: once agents are built on a model's observable behaviours (its tool-calling format, its refusal patterns, its output ordering), those behaviours are frozen by the ecosystem the way an API is frozen by its users — which is why compatibility-breaking changes in a harness's session format are so expensive (DeepSeek teardown).

The ASE reading of Hyrum's Law: the agent will depend on every observable behaviour you didn't promise — for agents, the implicit interface IS the contract. The defense is to make the promised contract exhaustive: stable, versioned, additive --json schemas; deterministic ordering; explicit exit-code semantics; and honest --help. You cannot stop agents from depending on your behaviour, but you can decide which behaviour they depend on. Hyrum's Law does not say the contract is meaningless — it says the contract must cover everything observable.

References

Harnessing Agentic AI Systems: Semantic Memory Router Pattern

Problem 6 of 15: choosing what context to inject. The Semantic Memory Router pattern vs the RAG Firehose anti-pattern — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriesmemoryretrievalrag

Problem 6 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Rolling Window Compression Pattern · Next: State Snapshot & Rollback Pattern.

The Problem — Choosing what context to inject

Not every retrieved fact belongs in every prompt. Retrieval without judgment drowns the instruction — and every injected chunk is untrusted input. The system must decide, and the decision cannot be the model's.

Field P6 — Semantic Memory Router (pattern) A6 — The RAG Firehose (anti-pattern)
Forces / Smell Grounding vs focus; freshness vs stable prefix; utility vs untrusted input. Top-K chunks by raw keyword match; the instruction buried; retrieval dominating the prompt.
Solution / Anti-solution Intercept ongoing tasks, query vector stores, and inject context fragments just-in-time into the agent's prompt. "More chunks equals better grounding."
Consequences / Failure Grounded, lean prompts; attention curated by the system rather than dumped by default. Injected context is the highest-leverage observation — and the highest-leverage attack. Drowns the instruction where Lost in the Middle predicts; every injected chunk is untrusted input — an injection vector (OWASP LLM08).
Tradeoffs / Refactoring The router is an injection surface; just-in-time injection fights prefix-cache discipline; retrieval quality is decided by chunking, metadata filtering, and reranking, not top-K volume. P6 with chunking, metadata filtering, and reranking deciding what is injected.
Evidence Pinecone's RAG guides (learn); the always-on survey's provenance and authority axes (always-on agents). Pinecone advanced RAG (learn); OWASP LLM08.
Related Refactoring for A6; composes with P8 (the tiers are the router's stores). Is the absence of P6; retrieval form of A4.

Discussion

The router is the decision layer RAG was missing: not every retrieved fact belongs in every prompt, and the decision cannot be the model's because the model cannot see what it was not shown. The mechanism is curation — chunking, metadata filtering, reranking — and the security corollary is that retrieved content is untrusted input: the router is both the grounding layer and the injection surface.

Key Insight

The system curates what the model sees. Injected context is the highest-leverage observation and the highest-leverage attack — retrieval quality is a decision, not a volume, and the decision cannot be the model's.

References

Pinecone RAG guide (learn) and advanced RAG (learn); OWASP Top 10 — LLM08 (2025); Liu et al., Lost in the Middle (arXiv:2307.03172); archive: Agentic-First CLI, always-on agents.

Hacker Laws for Agentic Software Engineering: Goodhart's Law

Law 5 of 12. Goodhart's Law says when a measure becomes a target it ceases to be a good measure. The ASE key insight: for agents the measure becomes the training target — the eval is the curriculum, so choose evals as if the agent will learn to game them, because it will.

hacker-lawsagentic-software-engineeringseriesgoodharts-lawevalsbenchmarksmetrics

Law 5 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: Gall's Law · Next: Hyrum's Law.

The Law

When a measure becomes a target, it ceases to be a good measure. (Marilyn Strathern, via hacker-laws)

The Key Insight for Agentic Software Engineering

Goodhart's Law was always true; agentic software engineering makes it a training signal. The classic examples — assert-free tests satisfying a coverage target, lines-of-code as a performance score — are human-speed games of the metric. An agent games the metric at machine speed, and worse: the closed loop bakes the gaming in. The DeepSeek minimal preset ships the RL composition as a product option because "the harness produces the trajectories; the trajectories feed post-training" (DeepSeek teardown) — which means whatever the harness measures becomes not merely a target but the curriculum. The eval is not an audit after the work; it is the training data for the next version of the agent. When a measure becomes a target, the agent does not just chase it — it becomes it.

The consequence is that benchmark design is now model design, and the benchmarks are already leaking. Terminal-Bench and SWE-bench measure task completion, and the harnesses that score well are the ones being distilled into the next models (data-driven design); a metric that rewards short tool lists produces agents that under-tool; a metric that rewards solving quickly produces agents that skip verification. The harness canon's own warning is Goodhart's Law in agentic dress: "if every test can be passed by pattern-matching the prompt, you are not measuring the assistant — you are measuring prompt luck" (harness canon).

The ASE reading of Goodhart's Law: for agents, the measure becomes the training target — choose evals as if the agent will learn to game them, because it will. The defense is the same one the pattern language gives for the voting ensemble: measure outcomes, not proxies; make the metric what you actually want, because the loop will optimize exactly that and nothing else. "The tasks are not only an evaluation — they are the training data, which is the strongest argument for getting them right and the strongest warning against letting them drift."

References

Harnessing Agentic AI Systems: Rolling Window Compression Pattern

Problem 5 of 15: keeping a long session in a lean window. The Rolling Window Compression pattern, the Context Avalanche anti-pattern, the Context Resets and Context Engineering frontiers — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriesmemorycontextcompression

Problem 5 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Token & Time Budget Throttler Pattern · Next: Semantic Memory Router Pattern.

The Problem — Keeping a long session in a lean window

Context is finite and conversation is not; raw dumps degrade reasoning. The frontier has split the answer into three: compression (P5), resets (F3), and the umbrella discipline of context engineering (F6).

Field P5 — Rolling Window Compression (pattern) A4 — The Context Avalanche (anti-pattern) F3 — Context Resets (frontier) F6 — Context Engineering (frontier)
Forces / Smell Fidelity vs summary; continuity vs clean slate; stable cache prefix vs rewrites. Raw logs and full transcripts in the history; context near capacity; performance degrading as the session grows. Continuity vs reset; state survival vs emptied window; cost vs quality. Right words vs right state; curation vs accumulation.
Solution / Anti-solution Automatically summarize older conversation histories in background threads; keep the active window lean. "Long context solves it" — dump everything. Distinguish compaction from resets: clear the context and hand state to a fresh agent through a structured artifact; use the reset when the model exhibits context anxiety. Engineer the whole context state — instructions, tools, MCP servers, data, history — toward the desired behavior.
Consequences / Failure Long sessions at bounded cost; a paging discipline — window is RAM, log is disk, summary is the page table. Lost in the Middle: performance highest at the ends, degrades in the middle; a filled context reads the middle worst exactly when the middle holds the answer. "A reset provides a clean slate, at the cost of the handoff artifact having enough state for the next agent to pick up the work cleanly." The log never rewrites — the artifact is a new prefix, not an edit. The unit of design becomes the system's context state — the systems argument in one sentence.
Tradeoffs / Refactoring Summary loss is permanent unless the log is preserved; compaction alone does not fix context anxiety; every compaction must be a genuine prefix-extension of the warm request. P5 over a derived view: 44 event types in the DeepSeek log, exactly three visible to the model. Resets add orchestration complexity, token overhead, latency; the need is a function of the model generation (Opus 4.5 removed the behavior). Every refinement risks a cache-prefix violation and a governance gap.
Evidence MemGPT (paper); DeepSeek's compaction fix (DeepSeek teardown); Anthropic's context anxiety (post). Liu et al., Lost in the Middle (paper); DeepSeek's logged-surface invariant (DeepSeek teardown). Anthropic's harness work (post). Anthropic, Effective context engineering (post).
Related Refactoring for A4; conflicts with and composes with the append-only log; pairs with F3. Is the absence of P5 and P8; substrate of A11; retrieval form of A6. Companion of P5; composes with P13; umbrella is F6. Umbrella over P5, P6, F3; "the system, not the agent."

Discussion

The memory problem contains the catalog's most honest war: the append-only log says the past is immutable, compression rewrites it as a projection, tiers store it in parallel — the resolution is layering, and the compaction prefix bug is what happens when the layers touch. The boundary is context anxiety: compaction preserves continuity but not a clean slate, and whether you need resets is a function of the model generation. F6 is the umbrella coordinating all three answers against the cache prefix and the governance gap.

Key Insight

The past is immutable and the view is derived. Compression is a projection over the log — if the summarizer rewrites history, the past becomes negotiable and the cache prefix dies. Context anxiety is a model property the system must adapt to; the harness is coupled to the model's psychology, and must be re-examined every time the model changes.

References

MemGPT (arXiv:2310.08560); Liu et al., Lost in the Middle (arXiv:2307.03172); Anthropic, Effective context engineering (post); Anthropic, Harness design for long-running applications (post); archive: DeepSeek teardown.

Hacker Laws for Agentic Software Engineering: Gall's Law

Law 4 of 12. Gall's Law says a complex system that works has evolved from a simple system that worked. The ASE key insight: grow the agent from a working single loop — the god agent is the complex system designed from scratch, and it never works.

hacker-lawsagentic-software-engineeringseriesgalls-lawcomplexityevolutiongod-agent

Law 4 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: Amdahl's Law · Next: Goodhart's Law.

The Law

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system. (John Gall, via hacker-laws)

The Key Insight for Agentic Software Engineering

Gall's Law is the anti-pattern catalogue in one paragraph. The god agent — the one massive agent with an immense prompt managing every phase of a workflow — is precisely the "complex system designed from scratch": every tool, every rule, every stage in one context window, assembled in one go and expected to work. It never does, and "cannot be patched up to make it work. You have to start over with a working simple system." The working simple system is the single loop — read, act, verify, reflect — and the evolution is the harness pattern language: from the sequential pipeline (linear, boring, working) you grow the orchestrator-worker (delegation added when the context overflows), and from that the ensembles and evaluators (verification added when the failures demand it). Each step is a small change to a system that already works.

The empirical record agrees. Anthropic's own long-running harness grew exactly this way: the earlier harness was an initializer plus a coding agent working one feature at a time; the frontier version added a planner and an evaluator, one at a time, each addressing a specific observed gap (Harness design for long-running application development). The most successful guidance in the field is Gall's Law in product form: "the most successful implementations use simple, composable patterns rather than complex frameworks" (Building Effective Agents). And the blog's own self-improving loop — read lessons, do work, reflect, write lessons — is the simple system from which everything else evolves.

The ASE reading of Gall's Law: grow the agent from a working single loop; don't design the multi-agent system from scratch. When a complex agentic architecture fails, the fix is not to add more scaffolding to it — it is to start over with the smallest loop that works and let the harness evolve. The system that works was found, not designed.

References

Harnessing Agentic AI Systems: Token & Time Budget Throttler Pattern

Problem 4 of 15: bounding the loop. The Token & Time Budget Throttler pattern vs the Infinite Execution Vortex anti-pattern — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriessafetytoken-economicscost-control

Problem 4 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Human-in-the-Loop Breakpoint Pattern · Next: Rolling Window Compression Pattern.

The Problem — Bounding the loop

Loops can burn unbounded tokens, wall-clock, and money retrying a broken step. The ceiling must be enforced by the system, never requested of the model — the enforcement half of the Bill as Assertion.

Field P4 — Token & Time Budget Throttler (pattern) A2 — The Infinite Execution Vortex (anti-pattern)
Forces / Smell Long-horizon work vs unbounded consumption; structural enforcement vs prompt-based; bill shape vs budget. No ceiling on iterations, tokens, time, or money; the same failing step retried; the loop "working on it."
Solution / Anti-solution Monitor continuous tool loops and forcefully terminate agents beyond maximum token costs or time boundaries. Enforce in the harness, never in the prompt. "It'll converge" — trust the model to stop.
Consequences / Failure The vortex cannot happen: the bill is bounded by construction, and the system — not the model — owns the ceiling. Unbounded token drain; an economic failure before a technical one — the tragedy of the commons staged inside one run.
Tradeoffs / Refactoring A budget too tight kills long-horizon work; too loose is theater. The throttler can invalidate the warm prefix it protects and raise the bill it caps. Choose the "exceeded" metric — spend, per-step, wall-clock, iterations. P4 enforced in code — OWASP LLM10, loop-iteration limits; reminders are not enforcement (DeepSeek's runaway-loop guard "only sends reminders and eventually goes quiet").
Evidence AutoGPT's iteration and cost limits (docs); OWASP LLM10:2025 Unbounded Consumption; token economics — prices down 98%, consumption up ~150x, bills tripled (Every Token Has a Price Tag). AutoGPT's open issue tracker (issues); Every Token Has a Price Tag.
Related Refactoring for A2; composes with P3; connects to the bill-as-assertion pattern. Is the absence of P4; is the single-loop form of A10.

Discussion

This problem is the economic face of the harness: it decides the shape of the bill when the model cannot be trusted to — "the caps were about the shape of the bill, not the money" (Every Token Has a Price Tag). The subtle interaction distinguishes a good throttler: it must not invalidate the warm prefix it exists to protect. And the honest limit is documented: soft enforcement is not enforcement.

Key Insight

The bill is a design decision and the ceiling is a system property. Reminders are not enforcement, and a throttler that kills its own cache prefix is the failure it was meant to prevent. Enforce in code, never in the prompt.

References

AutoGPT configuration (docs) and issue tracker (issues); OWASP Top 10 — LLM10 (2025); archive: Every Token Has a Price Tag, DeepSeek teardown.

Better Harnesses, Smaller Models: 90% Cheaper Agents via Automated Harness Adaptation

Five insights from the CMU paper (arXiv:2607.08938): the naive model-swap failure is a harness artifact, not a model property; the deployable unit is the model+harness pair; shared task difficulty is the amortizable lever; harness design is a search problem; and diagnosis quality is the optimizer's bottleneck.

slmharness-engineeringharness-optimizationmeta-agentscost-efficiencymodel-swap

Better Harnesses, Smaller Models (Yang, Zhao, Wu, Kästner — Carnegie Mellon, July 2026) starts from the null result every deployment engineer believes and inverts it. Swap a small model into a harness designed for a frontier LLM and it collapses; the paper shows the collapse is a harness artifact, not a model property. Adapt the harness automatically, and a 4B-parameter model matches the frontier model at a fraction of the cost.

The opening example is the whole argument. A budget-approval agent with gemini-3.1-pro hits 97.3% at $0.22 per query. Swap in gemma-4-26b-a4b unchanged and accuracy drops to 75.0%. After automated harness adaptation — a step-by-step workflow in the system prompt, a filtered tool set, and a hook that blocks the agent from sending the same message twice — the same SLM hits 98.3%, beating the frontier model at 8% of the cost. Five insights follow.

Insight 1 — The deployable unit is the model+harness pair, not the model

The "model swap" mental model — replace one model, re-validate, ship — is wrong. Across seven business tasks and three SLMs, a generic harness averages 31.4% / 26.9% / 9.5% accuracy (gemma / qwen3-coder / ministral); the optimized harnesses reach 80.2% / 74.8% / 25.0%. 16 of 21 task-model pairs improved significantly, and seven closed the SLM-LLM gap entirely — the best SLM recovering 89.7% of frontier performance at 4% of the cost, with 25% lower latency than the frontier agent.

The entanglement runs both ways. Harness adaptations don't transfer across models: ministral3-8b needs workarounds for its file-editing failures, qwen3-coder occasionally emits raw XML where the protocol demands JSON tool calls. Each model needs its own harness, so the optimization is part of every deployment and every model upgrade — the pair is the unit of deployment.

Insight 2 — Shared task difficulty is the lever, and it defines the boundary

Why does this work at all? Routine business tasks have structure shared across instances: every budget request hits the same policy lookup, the same pricing table, the same reserve accounting. A frontier LLM privately reconstructs that structure on every run, token by token, at frontier prices; an SLM reconstructs it worse. The fix is to lift the shared difficulty out of the model and into the harness — once, offline, and amortize it over every instance. The $20 one-time optimization per task is recovered after 13 production runs on average.

The same logic defines where the lever stops working:

  • Repetitive tasks adapt; diverse tasks resist. Task diversity (average Levenshtein distance between tool-call sequences) correlates with optimized performance at ρ = −0.96. Controlled: going from 3 workflow templates to 20 drops accuracy from 89.1% to 68.0%. Low-entropy tasks are exactly the ones where a harness can win the fight — the quantitative version of this blog's tasks that fight back argument.
  • Capable SLMs benefit more. Stronger models gain +48.8% from adaptation vs. +15.5% for the weakest — the harness offloads the repetitive parts; the model must still handle what can't be delegated.
  • Harnesses can't manufacture capability. The weakest model stays at 0.0% on two of seven tasks no matter what the harness does; website-management resists at 45.6% even for the best SLM.

Insight 3 — Harness design is a search problem, not a craft

The paper's methodological bet, in the spirit of the bitter lesson: harness design should be automated as a search problem driven by data and evaluation, not manual trial and error. The optimizer is a meta-agent running an evolutionary loop over the SDK's design space — system prompts, skills, tools, hooks, context management, sub-agents:

  1. Sample and evaluate — pick a candidate GEPA-style from the Pareto front of tried harnesses; run it on a batch of training instances, logging full trajectories.
  2. Diagnose and propose — the meta-agent reads trajectories plus the harness code and edits it, guided by a search memory of past proposals (so it stops re-proposing dead ends) and design-space documentation. Proposals pass a cheap sanity check with a repair loop.
  3. Validate and keep — full validation-set run if the edit improved the batch; add to the pool.

The loop is cheap by design: $20 per task-model pair, three runs each — $1,260 for the entire study.

Insight 4 — The meta-agent's moves are legible

The paper maps failures to adaptations so the search isn't a black box. Failures are indexed by capability — tool-use, instruction-following, knowledge, long-context, planning — and adaptations by harness component: contexts (add or manage), tools (create or manage), agent loops (instrument or orchestrate).

The winning moves are consistent. The dominant addressed failure modes are instruction-following (81%) and knowledge (81%); the dominant strategies are adding contexts (86%), creating tools (43%), and managing tools (29%). The best anomaly-detection harness combines all three: a custom query_mock_bigquery tool that sidesteps the default tool's long-context behavior, a filter from 40+ MCP tools down to seven, and the environment's table-naming convention externalized into the system prompt.

One negative result deserves emphasis: no optimized harness successfully used sub-agents — current SLMs can't track and coordinate sub-agent work. That is a bracing counterpoint to the industry reflex that answers every SLM weakness with a multi-agent topology.

Insight 5 — Diagnosis quality is the optimizer's bottleneck

The meta-agent loop is itself a system, and its developers learned where the leverage is:

  • Evidence beats summaries. Raw JSON trajectories made the meta-agent diagnose better than post-processed markdown — less human-friendly, richer evidence.
  • Intelligence beats iteration count. A cheaper meta-agent afforded more search steps but produced worse harnesses; lower-quality diagnoses outweighed the extra exploration. Hand-written heuristics (successful frontier trajectories, manual failure→fix maps) didn't help at all — given faithful evidence and an editable harness, the meta-agent infers repairs on its own.
  • Explore diverse regions, not one long trajectory. Search memory prevents rediscovering the same fixes; several independent short searches beat one long search.

What this means

The decision rule for practitioners is clean: for repetitive business workflows where the frontier-token bill is the constraint, take the best cheap MoE SLM you can find, spend a few dozen dollars on automated harness search, and expect to recover most of the frontier model's accuracy at ~5% of the cost — provided you treat "re-optimize the harness" as a normal part of every model upgrade.

Economically, this is a distillation story. A frontier meta-agent spends $20 once, compresses the shared structure of a task into a harness, and every subsequent run of the cheap model executes that structure nearly free — the expensive intelligence is amortized away, the same economics as agents as distillation at scale. The cheap agent is not a model you buy; it is a system you build. The harness is the product — and as the next post shows, it is now a product you can build on top of: the OpenHands SDK is the substrate this optimizer searched, and the harness is the product.

The OpenHands Software Agent SDK: Event-Sourced Foundations for Production Agents

Six insights from the MLSys 2026 paper (arXiv:2511.03690): architectural debt drove the redesign; event-sourced state is the consensus spine; local-first beats sandbox-first; serializability is the composability engine; security belongs inside the loop; and the harness is the product.

openhandsagent-sdkevent-sourcingharness-engineeringagent-architecturemlsysmcpsandboxing

The OpenHands Software Agent SDK (Wang et al., MLSys 2026) is a rare thing in the agent literature: an architecture paper that ships. It documents the redesign of the most popular open-source software-engineering agent — OpenHands, 64k+ GitHub stars in 18 months — into a modular SDK, and defends it with production telemetry: 61% fewer system-attributable failures in a 15-day live comparison, and state-of-the-art results on 3 of 5 benchmarks. Repos: software-agent-sdk, benchmarks.

It is also the substrate this blog wrote about yesterday: the harness optimizer in "Better Harnesses, Smaller Models" searches this SDK's design space — system prompts, tools, hooks, context management, sub-agents. The SDK paper explains why that space was searchable at all. A complete agent runs in six lines:

from openhands.sdk import LLM, Conversation
from openhands.tools.preset.default import get_default_agent
llm = LLM(model="openhands/claude-sonnet-4-5-20250929", api_key="...")
agent = get_default_agent(llm=llm)
conversation = Conversation(agent=agent, workspace="/path/to/project")
conversation.send_message("Write 3 facts about this project into FACTS.txt.")
conversation.run()

Insight 1 — The redesign is the story: four principles from V0's debt

The SDK is not a greenfield design; it is the answer to four failures of the original monolithic OpenHands (V0):

V0 pain V1 principle
Universal sandboxing: two processes with divergent state; local workflows needed duplicated tool/MCP code Optional isolation: one process by default, containers opt-in for production
Config sprawl: 140+ fields, 15 classes, 2.8K lines across parallel hierarchies; identical parameters diverging Stateless by default: immutable validated components; one source of truth for state
Monorepo coupling: agent core, eval suite, apps, and benchmarks in one repo; version conflicts leaking into production Strict separation: the core is a shared library consumed via APIs
Monolith logic: new behaviors required editing core or branching per entry point Two-layer composability: four deployable packages plus a typed component model

Each choice has a rejected alternative, stated honestly: event sourcing beat a database-backed model because a DB couples the SDK to a storage backend and breaks offline replay; optional isolation beat both mandatory containerization (V0's fragility) and fully-local-only execution (production needs safety).

Insight 2 — Event-sourced state is the consensus spine

The centerpiece is the state model. Every component (Agent, Tool, LLM) is an immutable, validated, serializable Pydantic model; all mutable variables live in ConversationState, which records interactions in an append-only EventLog. Events are two-tiered: LLM-convertible ones (messages, tool calls, observations, system prompts) are what the model sees; internal ones (CondensationRequest, PauseEvent, state updates) are pure bookkeeping. Persistence is dual-path — metadata in state.json, events as individual JSON files — so resuming a conversation means loading the file and replaying the log, with automatic detection of incomplete conversations.

The overhead objection is empirically dead: replaying 39,870 events from 433 real SWE-Bench conversations, persistence costs 0.20 ms median per event, full state replay 4.1 ms, crash recovery 7.4 ms (32.1 ms worst case). Against LLM round-trips of 1–30 s, event sourcing is free.

This is the third independent convergence this blog has covered on the same spine: DeepSeek Harness's enforced append-only session log, the spatiotemporal-composability calculus, and now OpenHands V1. Durable facts on a log, state as a pure derivation — that is the consensus architecture of the agent harness.

Insight 3 — Local-first beats sandbox-first

V0's mandatory sandboxing wasn't just awkward; in production it was the dominant source of failure. The conversation manager and execution runtime talked over inter-pod HTTP, and the 15-day V0/V1 rollout shows what that bought: HTTPStatusError 401s at 43.0 per 1k conversations, runtime-not-ready races at 18.8 per 1k, connection timeouts at 3.1 per 1k. V1 runs the agent and tools co-located by default, which eliminated the entire infrastructure-error class (69.8 → 0.0 per 1k) and cut system-attributable failures 61% (78.0 → 30.0 per 1k). The architecture removed a failure class structurally instead of patching its symptoms.

The same principle drives the deployment story. Conversation is a factory: give it a path or LocalWorkspace and you get an in-process loop with pause/resume (perfect for notebooks and debugging); give it a RemoteWorkspace and the same code serializes the agent config and delegates execution to an agent server over REST/WebSocket. Prototype to production is a two-line diff:

+from openhands.workspace import DockerWorkspace
+with DockerWorkspace(...) as workspace:
+    conversation = Conversation(agent=agent, workspace=workspace)
+    conversation.send_message("Create hello.py")
+    conversation.run()

Insight 4 — Serializability is the composability engine

The tool system runs on an Action–Execution–Observation contract: LLM tool calls validate into typed Action schemas before execution; ToolExecutor runs them; Observation structures results for the model. MCP tools are first-class — their schemas translate automatically into the same contract, so external servers behave identically to native tools.

The key move is the tool registry: Python executors aren't serializable, so tools travel across process and network boundaries as lightweight JSON specs, reconstructed lazily at runtime. That single mechanism is what makes distributed architectures and editable harnesses possible — specs can be composed, swapped, and searched. It is exactly why the harness optimizer could treat harnesses as a searchable space: a harness is just a composition of serializable specs.

Insight 5 — Security belongs inside the loop

The SDK treats safety as a control-loop component, not an afterthought, with two abstractions: SecurityAnalyzer, which rates each tool call low/medium/high/unknown risk, and ConfirmationPolicy, which decides when approval is required — pausing the agent in a WAITING FOR CONFIRMATION state until a human responds. Because assessment is separated from enforcement, trust can adapt mid-session (relax restrictions for read-only grep) and custom analyzers slot in without touching tool executors. SecretRegistry completes the picture: late-bound secrets fetched only at execution time, masked as <secret-hidden> in any output, rotatable mid-conversation.

These are precisely the features absent from the paper's comparison of the OpenAI, Claude, Google, and LangChain SDKs: the security analyzer, confirmation policies, secret auto-masking, and agent stuck detection are OpenHands-unique.

Insight 6 — The harness is the product (and the numbers back it)

Everything else follows the same logic. The LLM layer reaches 100+ providers, consumes native extended-thinking fields, and supports non-function-calling models via a text-prompt fallback — widening the usable model pool; RouterLLM lets routing policy (images to a multimodal model, text to a cheap one) live in a method, making the harness a cost-engineering surface. The Condenser halves API cost with no degradation by summarizing overflow.

The redesign's capability claims hold up: matched models score an identical 68.0% on SWE-Bench Verified across V0 and V1 (parity — the redesign didn't hurt), while Sonnet 4.5 gains +8.2 points on V1 thanks to extended-thinking support the event-sourced architecture absorbed naturally. Across 14 models and five task categories, the SDK hits SOTA on 3 of 5 — Commit0 at 56.2% vs. published SOTA 12.5%, GAIA at 80.0% vs. 74.6%, SWE-Bench Multimodal at 44.1% — using a single model per evaluation where several SOTA systems use multi-model orchestration. A three-tier QA pipeline (mocked-LLM tests on every commit, $0.5–3 LLM tests daily, $100–1000 benchmark runs on demand) keeps the whole thing honest.

The stated limits are worth repeating: single-agent focus (multi-agent coordination is future work); and a security core that is probabilistic — LLMSecurityAnalyzer is itself a model, so the guardrail inherits the model's failure modes, exactly the leaky abstraction this blog keeps returning to.

What this means

Read as a reference architecture, the SDK codifies three bets that now look convergent across the field: event-sourced state is the spine, local-first execution is the default, and the harness is the product — everything that matters (state, tools, security, context, deployment) lives in the foundation, which is why the minimal agent is six lines and why the harness-optimization literature builds directly on it. The architecture has been validated twice over: by 61% fewer production failures on one axis, and by being the substrate the next wave of agent research searches on the other.

Hacker Laws for Agentic Software Engineering: Amdahl's Law

Law 3 of 12. Amdahl's Law says potential speedup is limited by the parallelisable fraction of a task. The ASE key insight: agent swarms only speed up the parallelisable steps — the serial fraction (planning, verification, merging, context building) sets the ceiling, and the verifier is usually the serial fraction.

hacker-lawsagentic-software-engineeringseriesamdahls-lawparallelismverificationpipeline

Law 3 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: Brooks' Law · Next: Gall's Law.

The Law

Amdahl's Law is a formula which shows the potential speedup of a computational task which can be achieved by increasing the resources of a system. Normally used in parallel computing, it can predict the actual benefit of increasing the number of processors, which is limited by the parallelisability of the program. (hacker-laws)

The Key Insight for Agentic Software Engineering

Agent swarms are parallel processors, and Amdahl's Law applies to them exactly as it applied to CPUs: the speedup from adding agents is bounded by the fraction of the task that can actually be parallelised, and even a task that is 95% parallelisable caps out long before the swarm grows large. In an agentic pipeline, the parallelisable fraction is the independent work — the feature edits, the searches, the test runs that can be handed to separate workers. The serial fraction is everything that must happen in one context: the plan, the merge, the context building, and — most importantly — the verification.

The verifier is usually the serial fraction, and it is the one this blog keeps proving is the bottleneck: Fowler's retreat made it the headline — "code generation is no longer the bottleneck — verification is" (Verification Is the Bottleneck). Spawn a thousand agents to write code faster and the pipeline still drains through the single verifier that has to check it all — the Sonar AC/DC finding is the same shape: verification is where the 3-5x velocity boost rots. This is why the evaluator with hands is reserved for the final gate: it is the most valuable serial step, and running it in parallel with itself is the one thing Amdahl's Law says you cannot do.

The ASE reading of Amdahl's Law: measure the serial fraction of your agent pipeline — the verifier is usually it, and no number of workers beats the ceiling it sets. The harness-level lever is to raise the parallel fraction, not to add workers: decomposition turns one serial task into many parallel ones, mocks let independent runs proceed without waiting on shared infrastructure, and verification must be layered so the cheap checks run in parallel and only the expensive judgment is serial.

References

Harnessing Agentic AI Systems: Human-in-the-Loop Breakpoint Pattern

Problem 3 of 15: stopping the loop for a human. The Human-in-the-Loop Breakpoint pattern — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriessafetyhuman-in-the-loopgovernance

Problem 3 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Static Intercepting Gatekeeper Pattern · Next: Token & Time Budget Throttler Pattern.

The Problem — Stopping the loop for a human

Some mutations — financial transactions, deletions, releases — must not happen without human authority. The loop must stop, persist exactly where it paused, and resume only after a person decides. There is no named anti-pattern; its failure modes — the rubber-stamp and the swallowed breakpoint — are in the tradeoffs.

Field P3 — Human-in-the-Loop (HITL) Breakpoint (pattern)
Forces Autonomy wants the loop running; safety wants it stopped. Latency vs accountability; meaningful vs fast approval.
Solution Freeze the harness execution loop to demand manual approval for high-risk mutations. Persist the exact state at the pause, then resume from that checkpoint after a human approves, edits, or rejects.
Consequences Authority becomes a property of the system — a state-machine primitive, not a prompt; every human decision is recorded, making the breakpoint an audit seam.
Tradeoffs Cannot run unattended without an automation path, or the harness drowns in approvals and humans rubber-stamp everything. A breakpoint that can be swallowed does not exist: interrupts must not be wrapped in try/except.
Evidence LangGraph's interrupt() primitive (docs); OpenWorker's approval gates (OpenWorker outcome layer); the DeepSeek approval seam — allowed-once, missing answerer resolves to unavailable (DeepSeek teardown).
Related Composes with P2 (the gatekeeper decides what is routine, the breakpoint what is consequential); the automation path it needs is P4's discipline.

Discussion

The breakpoint makes authority a resumable property of the state machine: approval is a checkpoint, not a moment, and every human decision is recorded. Its two failure modes are failures of attention — the rubber-stamp, and the swallowed breakpoint (interrupts wrapped in try/except, which LangGraph explicitly forbids). The automation path is not an exception to the pattern; it is the pattern's other half, and its discipline is P4's: the system decides what is consequential, never the model.

Key Insight

Authority is a resumable state, not a moment. Approval without review is worse than no approval, and a breakpoint that can be swallowed does not exist. The harness decides what is consequential and what is routine; the model never does.

References

LangGraph interrupts (docs); archive: OpenWorker and the Outcome Layer, DeepSeek teardown, always-on agents.

Hacker Laws for Agentic Software Engineering: Brooks' Law

Law 2 of 12. Brooks' Law says adding human resources to a late project makes it later. The ASE key insight: adding agents to a late task makes it later — the handoff is the ramp-up, the coordination is the overhead, and the serial fraction sets the ceiling.

hacker-lawsagentic-software-engineeringseriesbrooks-lawparallelismdelegationmythical-man-month

Law 2 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: Conway's Law · Next: Amdahl's Law.

The Law

Adding human resources to a late software development project makes it later. (hacker-laws)

The Key Insight for Agentic Software Engineering

The first temptation when a task is late is to spawn more agents — the "nine women can't make a baby in one month" intuition gets buried under the fact that an agent costs nothing to instantiate. But Brooks' reasoning survives the change of subject completely: the ramp-up time becomes the context handoff (the new agent must learn the task, the repo, and the state — exactly the goldfish amnesia risk of delegation); the communication overhead becomes the coordination between agents (shared state, delegation contracts, merge conflicts — the orchestrator-worker single point of failure); and many tasks are not divisible, because the serial reasoning and verification fraction cannot be split.

The cost structure is what changed, and it is the trap. Spawning an agent is nearly free; integrating its output is not. A human's ramp-up is measured in days and the org pays it once; an agent's ramp-up is measured in tokens and context, but it is paid on every delegation, and a late task invites delegating more, which multiplies the handoffs precisely when the serial path is already the bottleneck. This is the Brooks → Amdahl pair: adding workers only helps the parallelisable fraction, and the late task is late because of the serial fraction.

The ASE reading of Brooks' Law: nine agents can't make a feature in one day — the handoff is the ramp-up, and the serial fraction sets the ceiling. The fix is not to spawn more workers; it is to shrink the handoff (structured artifacts, sprint contracts that define "done" before the work — the frontier contract pattern) and to make the serial fraction explicit and measured, so the decision to add an agent is made against the ceiling it cannot move.

References

Harnessing Agentic AI Systems: Static Intercepting Gatekeeper Pattern

Problem 2 of 15: refusing a tool call before it happens. The Static Intercepting Gatekeeper pattern vs the Prompt-Driven Authorization anti-pattern — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriessafetyauthorizationinterception

Problem 2 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Ephemeral Sandbox Wrapper Pattern · Next: Human-in-the-Loop Breakpoint Pattern.

The Problem — Refusing a tool call before it happens

The system's hands must be able to say no before anything reaches an external API — and the denial must be final: structural, not prose. Interception and authorization are the same seam.

Field P2 — Static Intercepting Gatekeeper (pattern) A3 — Prompt-Driven Authorization (anti-pattern)
Forces / Smell Security wants denial final; usability wants appeals. Determinism vs adaptivity; audit vs latency. "Do not delete user data" in the system prompt; permission checks that are sentences, not code.
Solution / Anti-solution Intercept model-generated tool calls against a strict blocklist before passing them to external APIs. Policy as prose — the belief that the model will read and obey the instructions.
Consequences / Failure A deterministic, auditable floor that cannot be argued around — the system's pledge(2): a restricted interface where the wrong thing is unexpressible. Instructions are data; a system prompt is a document the model may be instructed to ignore.
Tradeoffs / Refactoring Llama Guard is not static — it is a model that can be fooled; a true blocklist catches only what it enumerates; static floor for denial, model judgment above, never below. Authorization must be monotonic, structural, and fail-closed: monotonic guards "deny or abstain and can never force-allow."
Evidence Llama Guard (publication); DeepSeek tool pipeline — waterfalls, monotonic guards, allowed-once (DeepSeek teardown). Willison's prompt injection series (series); the DeepSeek monotonic-guard doctrine (DeepSeek teardown).
Related Composes with P1 and P3; refactoring for A3. Is the deeper form of A1; fixed by P2.

Discussion

The gatekeeper makes denial a system property rather than a model preference: the static floor does the deterministic denial, and the model-based layer adds judgment above it, never below — the DeepSeek ordering doctrine. The anti-pattern inverts the seam: policy as prose asks the model to obey a rule it can be told to ignore, so the refactoring is never a better instruction — it is moving the check into the tool, where injection cannot reach it.

Key Insight

Denial must be structural to be final. The blocklist decides before the model can be persuaded — deny by default, allow by exception — and authorization belongs in the tool, not the prompt: who may modify the system's state is a property of the harness.

References

Meta's Llama Guard (publication); OWASP Top 10 (2025); Willison's prompt injection series (series); archive: DeepSeek teardown, Verification Is the Bottleneck, always-on agents.

Terminal Agents: The Terminal Is the Substrate

A close reading of the Terminal Agents survey (arXiv:2608.20485): the terminal as execution substrate, the seven-dimension competence profile, and why benchmark scores hide process quality.

terminal-agentssurveysharnessevaluationagentsswe-benchcli

Terminal Agents: A Survey of AI Agents in Command-Line Environments (Yi Bin, Xiaoyang Yuan, Haoxi Zeng, Wencheng Ye, et al., arXiv:2608.20485, 52 pages) is the first survey that treats terminal-mediated execution as an object of study in its own right, rather than scattering it across software-engineering, tool-use, and computer-use literature. It is worth a close read because it formalizes a claim this blog has been circling for months: the terminal is not an interface the agent uses, it is the substrate the agent lives in (see Agentic-First CLI Design and the canonical Harness Engineering: Best Practices for Reliable Agent Systems).

The companion repo is awesome-terminal-agents.

The scope move: substrate, not surface

The survey's organizing lens is deliberately narrow: a terminal agent is a system whose dominant progress-bearing action–observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction. Three workload-level boundary tests operationalize "dominant":

  1. Primary execution substrate — command execution is the workload's main means of progress.
  2. Iterative command feedback — outputs, errors, logs, diffs, return codes, or state changes materially shape later actions.
  3. Terminal dependence — removing terminal access would change the workload's core behavior.

The consequences are sharp. SWE-agent is in scope; Agentless is not (static patch pipelines lack iterative execution); OSWorld is not (visual feedback is the progress-bearing substrate); a CLI-packaged assistant that forwards requests to a non-terminal workflow is not. The tests apply per-workload, not per-product — a platform can be in scope for repository repair and out of scope for its browser workloads.

This matters for harness engineers because most "terminal agents" in industry are hybrid systems, and the boundary tests tell you which parts of them deserve terminal-specific evaluation.

Seven dimensions of terminal competence

The survey's analytical backbone is a seven-dimensional competence profile. These are not model capabilities; they are system-level responsibilities distributed across the model, interface, harness, runtime, and environment:

  1. Command and action formulation — translating goals into executable commands, edits, and build/test/run actions.
  2. Feedback and artifact interpretation — extracting evidence from stdout, stderr, exit codes, logs, diffs, stack traces.
  3. Runtime and environment management — preparing, configuring, and repairing dependencies, services, containers, remote machines.
  4. State, task, and context tracking — maintaining environment state and interaction history across extended sessions.
  5. Progress verification — designing checks of intermediate validity and completion conditions.
  6. Recovery and adaptation — diagnosing failures, replanning, retrying from grounded evidence.
  7. Governance and side-effect control — permissions, sandboxes, approvals, resource limits, destructive-action prevention.

Two features of this list are worth internalizing. First, verification (5) is separated from recovery (6), and both from governance (7) — conflating them is why so many harnesses discover that "it passed the tests" and "it did something terrible" are simultaneous truths (see Sandboxing AI Agents and Always-On Agents, which frames governing and recovering as first-class state concerns). Second, the dimensions are trace-observable in principle: planning appears as executable action sequences, memory is tested against persistent state, adaptation is grounded in external feedback. That traceability is the hook for everything that follows.

Architecture: the harness is a performance-shaping component

The survey traces four shifts in design emphasis: tool-augmented prompting (ReAct, Toolformer, CodeAct) → structured executable actions (SWE-agent's ACI primitives, OpenHands) → terminal-mediated agency as a first-class target (Terminal-Bench, CLI-Gym, Endless Terminals) → runtime- and harness-centered design (Meta-Harness, AutoHarness, Agentic Harness Engineering), where context compaction, approval rules, observation shaping, and observability-driven harness evolution are treated as first-class variables.

Responsibilities are organized into four layers: interface and observation, runtime and workspace, control/verification/recovery/governance, and harness/context. The recurring tensions:

  • Expressiveness vs. recoverability — raw command access is expressive but noisy and hard to roll back; ACI mediation is reliable but narrows the task surface.
  • Generality vs. task discipline — platform runtimes span heterogeneous work but give less structured feedback.
  • Automation vs. inspectability — permission gates and approval checkpoints aid auditability but add latency; governance is the least systematically addressed dimension.

And the one that should shape every comparison you publish: attribution difficulty. Controlled skill injection can add substantial token overhead without improving pass rate, and Agentless shows static pipelines rivaling interactive agents on some repository-repair tasks. Gains under an optimized harness may come from context management, observation shaping, retry policy, or injected procedural knowledge — not from a better model. The complete model–harness–runtime configuration is the unit of comparison, not the model.

Acquisition: trajectories are the unit, failure is the signal

The relevant learning unit is a stateful trajectory — actions, observations, state changes, verification, recovery — not a prompt–response pair. The survey maps the acquisition levers:

  • SFT on successful traces teaches common commands and workflows, but offers almost no supervision for diagnosis, rollback, or recovery after wrong assumptions.
  • RL with executable rewards (Endless Terminals, ECHO, SWE-Master, SWE-Gym, Tmax) aligns behavior with completion, but sparse rewards can reinforce brittle or unsafe behavior.
  • Process-aware / verifier-guided optimization supervises intermediate decisions via judgments, rankings, or hindsight validation (AgentHER is the key example — terminal failures usually appear early through stderr, failed tests, or inconsistent state).
  • Failure-conditioned training (TRACE, AgentHER, AgentForesight) preserves diagnosis and recovery, but suffers noisy labels and repair-loop overfit.
  • Synthetic generation broadens coverage of rare commands but risks teaching synthetic regularities; runtime memory (Memento, Context-Folding, TACO) sustains long horizons but risks persisting incorrect assumptions.

The uncomfortable finding: recoverable failures remain underrepresented. Failed installs, version conflicts, rollback decisions, and dead-end repairs are exactly the interactions needed to learn recovery — and successful-trace filtering removes them. If your training pipeline only keeps traces that worked, you are training an agent that cannot diagnose.

Evaluation: outcomes are the floor, not the ceiling

The evaluation section is the most useful part of the paper for working engineers. Benchmark families expose different things:

Emphasis Representative examples Blind spot
Repository repair SWE-bench, SWE-PolyBench Conflates repair with terminal competence
CLI/terminal-centered Terminal-Bench, TerminalWorld, LongCLI-Bench Mixes terminal-native and repo-mediated tasks
Setup SetupBench Isolated from end-to-end workflows
Process OctoBench, ProcBench, AppWorld No standardized process scoring
Long horizon SWE-Bench Pro, LoCoEval, LifelongAgentBench Costly, hard to reproduce
Safety/governance BashArena, ClawSafety, AgentHazard Task success may conceal harmful actions
Production ProdCodeBench Limited public access

The evidence hierarchy goes beyond binary correctness: outcome evidence (task completion), process evidence (how execution happened), environment evidence (runtime validity), trace evidence (inspection/replay), and governance evidence (permissions, containment, side effects). Three protocol facts deserve wide circulation:

  • SWE-EVO-style long-horizon decay: 65–73% on SWE-Bench Verified drops to 21–25% on multi-file evolution in its reported setting. Short tasks hide failure.
  • Reward hacking is measurable: an audit reports 16% of tasks across five terminal-agent benchmarks are reward-hackable.
  • Harness effects dominate: WildClawBench reports an 18-point harness-conditioned gap; SWE-rebench finds evidence consistent with contamination inflation on static tasks.

Final success alone does not reveal whether an agent preserved state, recovered from failure, verified completion, or respected execution constraints.

The survey runs its own diagnostics

Rather than stopping at synthesis, the survey contributes bounded fixed-condition experiments — and these are the most concrete part of the paper. On a fixed configuration (mini-SWE-agent + DeepSeek-V4-Flash), it computes seven trace-derived process indicators (P1–P7) across four benchmark families:

Benchmark Tasks Outcome Final-verify (P5) Gov. trigger (P7)
Terminal-Bench 2.1 241 52.6% 29.5% 0.5%
SetupBench 93 59.1% 36.6% 1.4%
LongCLI-Bench 21 23.8% 23.8% 4.5%
BashArena 640 41.8% 66.2% 3.0%

Key results: rule-matched invocation failures are rare (P1 ≈ 0.0%), local feedback use is high (76–82%), but final-window verification spans 23.8%–36.6% on three benchmarks versus 66.2% on BashArena — the same agent shows different process behavior because the benchmarks foreground different demands. The matched system comparison is even more pointed: SWE-agent wins every block, but the gap between systems ranges from at most 7.00 points on SWE-bench Lite to 21.25 points on Claw-SWE-Bench Lite, and the ordering of OpenHands and mini-SWE-agent reverses across benchmarks. Meanwhile the Flash-vs-Pro model variant never moves outcomes by more than 2.50 points. Benchmark choice changes what you can see, and system comparison without benchmark context is meaningless.

The trace cases ground the aggregates — the magsac-install case shows a dependency loop with a 38.5% environment-command non-zero-exit rate that never closes because the agent ends before an evaluator-relevant import check. Local feedback without verification and recovery leaves the loop unclosed.

type TraceIndicators struct {
	InvocationFailures  float64 // P1: rule-matched command failure rate
	FeedbackUse         float64 // P2: helpful-use among feedback episodes
	EnvironmentExits    float64 // P3: non-zero exit on env-management commands
	StateErrors         float64 // P4: consequential state errors
	FinalWindowVerify   float64 // P5: verification in final window
	Recovery            float64 // P6: task-relevant recovery success
	GovernanceTriggers  float64 // P7: irreversible/overprivileged/secret-handling actions
}

This is the closest thing the survey offers to a portable process-evaluation schema — and it is exactly the shape of instrumentation this blog argues harnesses should ship with (see Harness Engineering: Best Practices for Reliable Agent Systems).

What to take away

The survey's own conclusion is the one to steal for your evaluation practice: report system and runtime conditions explicitly, pair task outcomes with process evidence, and publish replayable traces. The four research priorities it derives — cross-domain competence, fresh and replayable process-level evaluation, runtime governability, and controlled model–harness attribution — map one-to-one onto the gaps that keep showing up in real agent deployments.

For harness engineers the operative reading is: treat the terminal as substrate, instrument the seven dimensions, keep failure traces, and never publish a model comparison without pinning the harness, the runtime, and the benchmark. This is the survey version of "tasks that fight back" — except now the tasks are graded on process, not just outcomes, and the benchmark is treated as a measurement instrument that must resist gaming, the argument made in Empirical Game Theory for Agents. If you only track resolution rates, you are measuring the flattering version of your system. (Relevant adjacent reading: DeepSeek Harness Notes and the Harness Patterns for Agentic AI Systems index.)

References

Hacker Laws for Agentic Software Engineering: Conway's Law

Law 1 of 12. Conway's Law says the technical boundaries of a system will reflect the structure of the organisation. The ASE key insight: the organisation is now humans, agents, and the harness that wires them — and the software will mirror the topology of the agents that made it, so you change the software by changing the topology.

hacker-lawsagentic-software-engineeringseriesconways-lawtopologyharness

Law 1 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Next: Brooks' Law.

The Law

The technical boundaries of a system will reflect the structure of the organisation. (hacker-laws)

The Key Insight for Agentic Software Engineering

Conway's Law was about the org chart, and the org chart just changed: it now includes the agents, and — more importantly — the harness that wires them. In an agentic software engineering shop, the structure that shapes the software is not only who reports to whom; it is how the agents communicate, what shared state they read, which delegations are allowed, and where the verifiers sit. The software will mirror that topology, because the topology decides what each agent can see and therefore what it can build. Give two agents the same repository but different shared-state boundaries and they will produce systems with seams in different places — the seam lands where the communication seam lands.

The harness is the org chart. The orchestrator-worker topology produces central-plan, delegated-feature systems; the blackboard topology produces choreographed, shared-state systems; the sequential pipeline topology produces layered, linear systems. The model is the same in all three — the topology is what changes, and the topology is a harness decision, not a model decision (the system, not the agent).

The flip side is sharper: agents also mirror each other. The distillation loop — small models trained on a large model's outputs — makes the "organisation" a single voice, and the software it produces is correspondingly single-voiced. When the agentic org has one mind, the software has one architecture, for better and for worse; when it has many independent minds with narrow channels between them, the software is modular in exactly the places the channels are narrow.

The ASE reading of Conway's Law: the software will mirror the topology of the agents that made it, so design the topology — the harness — to get the seams you want. If you want modular systems, make the communication channels between agents narrow and the shared state explicit; if you want integrated systems, wire the agents tightly. The law was never an excuse for the org chart; it was an instruction to take the org chart seriously. In agentic software engineering, the org chart is a file you can edit.

References

Harnessing Agentic AI Systems: Ephemeral Sandbox Wrapper Pattern

Problem 1 of 15: containing untrusted execution. The Ephemeral Sandbox Wrapper pattern vs the Naked Prompt anti-pattern — one table, a short discussion, the key insight, and the important references.

harnesspattern-languageagentic-aiseriessafetysandboxingcontainment

Problem 1 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Next: Static Intercepting Gatekeeper Pattern.

The Problem — Containing untrusted execution

An agentic system executes code that is untrusted by construction — written by a model, possibly steered by injected instructions. A mistake or an attack must die with the task that produced it.

Field P1 — Ephemeral Sandbox Wrapper (pattern) A1 — The Naked Prompt (anti-pattern)
Forces / Smell Isolation wants a real boundary; performance wants none. Ephemerality vs persistence; teardown completeness vs free cleanup. API keys in the prompt or environment; the model told to "be careful"; no proxy layer between the model and the credentials.
Solution / Anti-solution Spawn isolated, short-lived virtual environments (e.g., Docker, WASM) per task; mutate freely; destroy on completion. Treat the model as an application boundary and the prompt as an access control list.
Consequences / Failure Blast radius bounded by lifetime, not trust; clean training trajectories; the wrapper reifies the outside world, making recovery promises possible. Injection converts instructions into actions; credentials exfiltrate (OWASP LLM01, LLM02, LLM07 in 2025; LLM06 in 2023/24).
Tradeoffs / Refactoring Startup latency and state loss; heavy isolation vs WASM limits; only as good as its teardown — teardown must be derived from the load, not remembered. P1, P2, and a transparent secrets proxy so the agent never sees the credentials — the Replit pattern; the xz lesson: the tool that runs arbitrary code with your credentials is the highest-value target in your supply chain.
Evidence LangChain sandbox integrations (docs); Replit's thirteen-layer stack (Sandboxes Are Hard); DeepSeek file-effects-only sandbox vocabulary (DeepSeek teardown). OWASP Top 10 (2025); Willison's prompt injection series (series).
Related Composes with P2; cousin of the sandbox-stack pattern; refactoring for A1. Leads to A3; fixed by P1 and P2.

Discussion

The pattern converts trust into lifetime: the system does not need to believe the code is safe, it needs the code to die with the task — which is why teardown must be derived from the load, not remembered. The anti-pattern is containment skipped: it trusts the least trustworthy component with the most valuable secrets, because instructions are data. Where the guarantee stops: the wrapper bounds the process, not the world — which is why the gatekeeper (P2) must sit outside it.

Key Insight

Trust is a lifetime property, not a belief. The wrapper converts "is this code safe?" into "does this code die with the task?" — and a guarantee you must remember to enforce is a guarantee an agent will, at some point, not.

References

LangChain sandbox integrations (docs); OWASP Top 10 for LLM Applications (2025); Willison's prompt injection series (series); archive: Sandboxes Are Hard, DeepSeek teardown, spatiotemporal composability.

Hacker Laws for Agentic Software Engineering

The index of a series that takes twelve laws from dwmkerr's hacker-laws catalog and asks what each means when the engineer is an agentic AI system. The law does not change; the subject does — Brooks was about people, and in agentic software engineering 'human resources' can be spawned in milliseconds, but the reasoning survives the change of subject. Each law gets its own post, focused on one key insight for agentic software engineering.

hacker-lawsagentic-software-engineeringseriespatternsagentslawsprinciples

dwmkerr/hacker-laws is the best-known catalog of the laws, theorems, and principles that software engineers cite to explain why things keep going wrong. This series takes twelve of them and asks a question the catalog was never written to answer: what does each law mean when the engineer is an agentic AI system?

The law does not change; the subject does. Brooks' Law was about people — and in agentic software engineering, "human resources" can be spawned in milliseconds. Amdahl's Law was about processors — and now the processors are agents. Goodhart's Law was about KPI-gaming by employees — and now the optimizer is a model that will game the metric at machine speed. The reasoning survives the change of subject; that is exactly why the laws are worth re-reading for the agentic era, and why this blog's own catalog — the Harnessing Agentic AI Systems pattern language — keeps landing on the same conclusions the laws reached decades ago.

Each post in this series is one law, one commit, one push, and one key insight for agentic software engineering — the insight is the whole point, and the law is the evidence.

# Law The ASE key insight
1 Conway's Law The software will mirror the topology of the agents that made it — design the topology (the harness), not just the prompt.
2 Brooks' Law Nine agents can't make a feature in one day — the handoff is the ramp-up, and the serial fraction sets the ceiling.
3 Amdahl's Law Measure the serial fraction of your agent pipeline — the verifier is usually it, and no number of workers beats the ceiling it sets.
4 Gall's Law Grow the agent from a working single loop; don't design the multi-agent system from scratch.
5 Goodhart's Law For agents, the measure becomes the training target — choose evals as if the agent will learn to game them, because it will.
6 Hyrum's Law The agent will depend on every observable behaviour you didn't promise — for agents, the implicit interface IS the contract.
7 Hofstadter's Law An agent task always takes longer than you expect, recursively — so the ceiling is a system property, not an estimate.
8 Kernighan's Law If the agent writes clever code, the system must debug it — keep agent output boring and make the verifier the smarter half.
9 Parkinson's Law Agent work expands to fill the budget — the budget is the discipline, and the scope is a contract agreed before the work.
10 Chesterton's Fence The harness must make the agent find out why the code is there before letting it change — intent is a verification problem.
11 The Bitter Lesson The loop that leverages computation beats the hand-crafted prompt — and the agent will apply the same lesson to your harness.
12 The Law of Leaky Abstractions Agent abstractions leak — and the leak layer is where the harness must put the verifier, because the model cannot see the leak.

The series

  1. Conway's Law — the org chart is now humans, agents, and the harness that wires them.
  2. Brooks' Law — adding agents to a late task makes it later.
  3. Amdahl's Law — agent parallelism is bounded by the serial fraction.
  4. Gall's Law — complex agent systems evolve from simple loops.
  5. Goodhart's Law — the eval is the curriculum; the metric is the target.
  6. Hyrum's Law — agents depend on every observable behaviour.
  7. Hofstadter's Law — agent timelines are recursive estimates.
  8. Kernighan's Law — debugging agent output is the bottleneck.
  9. Parkinson's Law — agent work expands to fill the budget.
  10. Chesterton's Fence — understand before the agent changes anything.
  11. The Bitter Lesson — compute and the loop beat the hand-crafted prompt.
  12. The Law of Leaky Abstractions — every agent abstraction leaks; the verifier lives at the leak.

References

Harnessing Agentic AI Systems: A Pattern Language

The entry overview to the Harnessing Agentic AI Systems series: why harness engineering patterns matter — the unit of design is the agentic AI system, not the agent — and the map of the fifteen pattern posts, each titled 'Harnessing Agentic AI Systems: <Pattern> Pattern', each with one table per problem followed by a discussion, a key insight, and references.

harnessharness-engineeringpattern-languagepatternsagentsagentic-aiagentic-systemsseries

A harness is the environment an agentic AI system lives in: everything between the models and the world. It decides what the system can see, what it can do, and how the outcome is judged. This series catalogs the patterns of that layer — the recurring engineering problems every agentic system faces, the patterns that solve them, and the anti-patterns that fail them.

The series exists because the harness — not the model — is where the leverage concentrates. The evidence is in the numbers: the same model finished between 47% and 67% of real tasks across eight different harnesses while cost per finished task varied sevenfold (DeepSeek Harness teardown); the same weights score 3.4 points apart on the public Terminal-Bench board under two major harnesses; SWE-agent more than doubled the previous state-of-the-art on SWE-bench with the same GPT-4 by changing only the interface (Agentic-First CLI); and harness-level improvements have lifted scores without training. Nothing about the intelligence changed. The wrapper changed.

Why harness engineering patterns matter

The unit of design is the agentic AI system, not the agent. A harness for an AI agent is an instrument around one model — a wrapper for a single component. A harness for an agentic AI system is the entire runtime around one or many agents: the loop, the tools, the memory and state, the interfaces the agents read, the policies that constrain them, the verifiers that judge them, the orchestrators that compose them, the humans in the loop, and the evaluation and billing apparatus that closes the loop. We harness the whole system, including the agent(s). We do not care about the agent as such — not because agents do not matter, but because the agent is the one component we cannot design, and everything that can be designed is the system. Google's Agents whitepaper makes the same claim from the vendor side: the agent is "a program that extends beyond the standalone capabilities of a Generative AI model" — model, tools, and an orchestration layer. MemGPT's title is the same claim in four words: LLMs as operating systems. The agent is the hardware. The harness is the OS.

Why it is a mistake to focus on the agent:

  • It misattributes failures. When an agentic system fails, the agent-focus blames the model; the system-focus asks which tool it selected, what context it was given, what interface it read, which verifier approved it. "[The model decided]" is not an audit answer — regulators and boards accept checkpointed state, provenance chains, and deterministic replay (durable daemons execution).
  • It optimizes the least controllable part. The agent is the only stochastic component. Every lever you actually pull is a system lever: the interface, the memory, the tools, the policies, the verifiers.
  • It ignores where improvement lives. "Agents don't learn. Every mistake an agent makes, it will make again unless the harness explicitly prevents it" (Verification Is the Bottleneck). The loop that improves is a system loop.
  • It confuses a component with its properties. Identity, auditability, authority, cost, safety, and correctness are all system properties — the agent cannot answer "who did what, why, and was it correct?"; only the system can (always-on agents, Buzz).

Patterns matter because the problems recur. Every agentic system eventually has to contain untrusted execution, refuse tool calls, bound its loops, manage its context, make its state survive, type its output, discover its tools, divide its work, coordinate through shared state, and produce a verdict no single model can fake. The fifteen patterns in this series are the named, evidenced answers to those recurring problems — with the anti-patterns that fail them and the frontier patterns that are evolving them. The tradeoffs are not defects in the patterns; they are the prices. Every pattern buys a guarantee and charges a cost, and the guarantee stops where the declaration stops.

How to read this series

The series is organized by problem, not by component, in the tradition of Christopher Alexander's A Pattern Language (1977): "a set of problems and documented solutions," cross-referenced into a network. Each of the fifteen problems lives in its own post, titled Harnessing Agentic AI Systems: <Pattern> Pattern. Every post has the same shape: the problem statement, one table — rows are the fixed pattern-language fields, columns are the problem's pattern, anti-pattern, and frontier entries (the row labels pair the two vocabularies: Forces/Smell, Solution/Anti-solution, Consequences/Failure, Tradeoffs/Refactoring, Evidence, Related) — followed by a Discussion, a Key Insight, and the problem's References. Read the posts in order: the problems build — first make the system safe (1–4), then make it remember (5–7), then make it act (8–11), then make it scale (12–14), then make it trustworthy (15).

The series

# Problem Post Pattern(s) Anti-pattern(s)
1 Containing untrusted execution Ephemeral Sandbox Wrapper Pattern P1 Ephemeral Sandbox Wrapper A1 The Naked Prompt
2 Refusing a tool call before it happens Static Intercepting Gatekeeper Pattern P2 Static Intercepting Gatekeeper A3 Prompt-Driven Authorization
3 Stopping the loop for a human Human-in-the-Loop Breakpoint Pattern P3 HITL Breakpoint
4 Bounding the loop Token & Time Budget Throttler Pattern P4 Token & Time Budget Throttler A2 The Infinite Execution Vortex
5 Keeping a long session in a lean window Rolling Window Compression Pattern P5 Rolling Window Compression · F3 Context Resets · F6 Context Engineering A4 The Context Avalanche
6 Choosing what context to inject Semantic Memory Router Pattern P6 Semantic Memory Router A6 The RAG Firehose
7 Making state survive and giving it homes State Snapshot & Rollback Pattern P7 State Snapshot & Rollback · P8 Tiered Hierarchical Memory A5 Goldfish Amnesia
8 Typing tool output and keeping errors legible Schema Enforcement & Self-Correction Pattern P9 Schema Enforcement & Self-Correction A7 The Silent Crash · A9 The Schema Free-for-All
9 Not blocking the loop on long tools Asynchronous Tool Worker Queue Pattern P10 Asynchronous Tool Worker Queue
10 Making the fast loop repeatable Mock Tool Virtualization Pattern P11 Mock Tool Virtualization
11 Discovering capabilities without bloating the prompt Dynamic Tool Discovery Pattern P12 Dynamic Tool Discovery / Registry · F4 The Interop Layer A8 The Bloated Utility Belt
12 Dividing a workflow across agents Orchestrator-Worker Pattern P13 Orchestrator-Worker · F2 Sprint Contracts A11 The God Agent
13 Coordinating through shared state without corruption Blackboard Pattern P14 Blackboard (Shared Workspace) A12 State Race Conditions
14 Keeping linear flows linear Sequential Pipeline Routing Pattern P15 Sequential Pipeline Routing
15 Producing a verdict no single model can fake Voting / Consensual Ensemble Pattern P16 Voting / Consensual Ensemble · F1 Generator–Evaluator Loop · F5 Live-Environment Evaluators A10 The Committee Paradox

References

Grok Automations: Describe Once, Run Forever

xAI shipped Grok Automations on July 16, 2026: describe a job once, and Grok runs it on a schedule or when an email arrives, then reports back. This post reads the announcement as the productization of this blog's durable-daemons pattern — 'condition 3 is cron with an LLM' — and the first consumer-scale instance of the always-on agent. It examines the design decisions with the evidence the announcement itself carries: every run is a fresh request (same instructions, current data), the trigger is the harness (schedules, email filters, connectors, skills, run history), the economics of runs that multiply the bill, and where the guarantees stop — email as a prompt-injection surface with teeth, no visible budget ceiling, the HITL gap on consequential actions, and the deliberate goldfish-amnesia tradeoff of stateless runs.

grokxaiautomationsalways-on-agentsdurable-daemonsagentsagentic-aischedulingtriggerstoken-economicsprompt-injectionharness

On July 16, 2026, xAI shipped Automations in Grok: "Describe a job once and Grok runs it on a schedule or when an email arrives, then reports back." The one-line pitch is the whole design, and it deserves a close reading, because it is the first consumer-scale productization of an idea this blog has been tracking for months. The durable daemons series specified the pattern as four conditions — persistence, stateful memory, autonomous action, crash-proof execution — with a type hierarchy: Agent ⊃ Daemon ⊃ Durable Daemon. Its most memorable sentence was about condition 3: "Condition 3 is cron with an LLM." Grok Automations is that sentence, shipped.

"jobs Grok runs on its own. Describe the work once, choose when it runs, and Grok takes it from there, whether that's research done before you're awake or an important email flagged the moment it lands."

The announcement's own language is the language of the daemon: the job is described once, the system fires it, the system reports back, and no prompt is required. That is condition 3 — trigger-driven autonomy — in the product. The rest of this post reads the announcement the way the announcement asks to be read, then weighs the design decisions with the tradeoffs this blog's pattern-language series has been cataloging.

What shipped

An automation is a stored job: instructions that "read like any chat message," optional attached files, connectors, skills, and a mode. It runs on one of two trigger families:

  • Schedules — once, daily, weekdays, weekly, monthly, or yearly, "at a time you choose in your timezone": a morning brief at 8:00, a rent reminder on the 1st.
  • Email triggers — the automation watches the inbox; "when an incoming email matches your filters (sender, recipient, or subject), the automation fires with that email as context, and Grok responds to the actual message."

Every run is recorded: "When an automation fires, Grok opens a real conversation, does the work, and saves the result to its run history. Open any run to read the full thread, or pick up the conversation where Grok left off." Notifications are a choice — email, app, both, or neither. Automations can be created from chat ("check the news every morning and flag anything about pricing"), from templates, or with a Run now button for testing. Scheduled automations are free to everyone; email triggers are a SuperGrok feature.

The screenshot in the announcement shows the shape of the thing: a Morning Brief with Runs 4 · Succeeded 2, one run "Generating just now," and a run history that reads like a daemon's log — "2 calendar conflicts, 4 emails worth replies," "launch-day schedule and three stories to read," "quiet inbox, one deadline moved to Friday," "flight check-in opens at noon, pack for rain." The runs are conversations, saved, resumable, and auditable. That is the always-on agent with a provenance trail.

The design decision that matters: every run is a fresh request

The most consequential sentence in the announcement is easy to miss:

"every run is a fresh request: same instructions, current data."

Each firing is a full conversation built from the stored instructions plus whatever the triggers and connectors deliver at that moment — not a continuation of the last run. This is a deliberate architecture, and it is the same tradeoff the always-on survey names as the goldfish amnesia anti-pattern: a run that remembers nothing of the previous run. Here it is chosen on purpose, and the choice is defensible.

What the fresh-request design buys: no context accumulation, which means no context avalanche — a morning brief on day 200 does not carry 199 days of history into its window. Each run is bounded, legible, and independently debuggable from run history. Failures are isolated: one bad run does not poison the next. And the instructions are a stable prefix — the same front-of-request tokens every time — which is exactly the shape a prefix-cache discipline wants, if the harness keeps the prefix warm.

What it costs: the daemon cannot remember what it learned. An automation that must "remember what happened yesterday" has to externalize that memory through connectors or skills, because the run itself starts blank. This is the state-blindness tradeoff made explicit: the platform chose fresh-context reliability over cross-run memory, and the burden of memory is pushed to the harness's storage — run history (the audit trail) rather than working memory (the daemon's brain). The durable-daemons specification called condition 2 (stateful memory) a precondition for condition 3 (autonomous action); Grok's automation has condition 3 with a deliberately shallow version of condition 2.

The trigger is the product

This is where the blog's central argument applies: the system, not the agent. Grok is the model; the automation is the system — triggers, connectors, skills, mode, notification channel, run history, templates. Every point of value in the announcement is a harness feature, not a model feature:

  • Triggers are the daemon's "when": schedules are cron with timezone support, and email filters (sender, recipient, subject) are condition matching over an inbox — the event-driven choreography this blog described in durable daemons execution: "The daemon watches conditions. Fires triggers. Makes and discharges commitments. Invokes itself."
  • Connectors (type @ to mention a connector, and Grok uses it on every run) are the capability seam — the tool-binding pattern as a consumer feature: the automation's reach is defined by which connectors are mounted, not by the model.
  • Run history is provenance — the always-on survey's audit trail, by construction. "Open any run to read the full thread" is the answer to "who did what, why, and was it correct?" — at least for the what and the why.
  • Chat-to-automation ("ask Grok to 'check the news every morning…' and it sets one up") is loop engineering inverted for the consumer: you stop prompting, and the system turns your prompt into a loop.

This is the same-model-different-harness argument from the DeepSeek teardown made product-shaped. The model is a socket; the automation system is what the user actually buys. And the announcement is explicit that the system is configurable per job — "pick a mode," "add connectors and skills," choose notifications — which is the per-session composition idea wearing consumer clothes.

The economics: describe once, run forever

"Describe once, run forever" is a billing sentence dressed as a convenience sentence. Every scheduled run is a metered event; every email trigger is a metered event that arrives without an appointment. The token economics post computed the shape of this: always-on agents are "the token burn behind the tripled bills," and the Jevons paradox means that the cheaper the unit, the more total consumption — because the resource becomes economical for uses it could never before serve. A morning brief is a use case that did not exist as a product before, because nobody was going to prompt a chatbot at 8:00 AM every day for a year. Automations exist to convert that would-be manual labor into a recurring, metered run.

The pricing split in the announcement is the honest version of the meter: schedules are free to everyone (predictable, bounded, self-selected frequency), email triggers are SuperGrok (unbounded — any email that matches is a run). The meter is the product boundary: the trigger that can fire without your involvement is the one you pay for, because its bill has no shape until it has run.

Where the guarantees stop

The announcement is a product page, and product pages end where the design's hard questions begin. Four stand out, each mapped to this blog's pattern language:

1. Email is a prompt-injection surface with teeth. The email trigger "fires with that email as context" — the email is untrusted input inserted into the instructions of a run that has connectors attached. The prompt injection threat model this blog has documented since the Zero Overhead post applies with compounding interest: the attacker does not need to compromise the platform, only to send an email that matches a filter. A blocked sender can be spoofed; a subject filter can be matched; and the run has teeth — connectors, skills, and a "respond to the actual message" instruction. The gatekeeper pattern — a blocklist between the untrusted message and the tools — is the missing piece, and the announcement does not mention one. This is the oldest lesson in the catalog, now embedded in a consumer default.

2. No visible ceiling. The announcement shows no budget, no run limit, no cost control on the automation page. Schedules are bounded by construction (you choose the frequency), but email triggers are bounded by your inbox's volume — and a misbehaving automation (or an injected one) can fire on every matching email, every hour, forever. This is the infinite execution vortex risk with a trigger instead of a loop: the same unbounded consumption, entering through a different door. The budget-throttler pattern is the fix, and its absence from the announcement is a gap, not a feature.

3. The HITL gap on consequential actions. The announcement's own examples are read-only — summarize, flag, remind. But "Grok responds to the actual message" invites the next step, and the next step after that is sending. The HITL breakpoint — a persisted pause before a consequential action — is the governance seam, and the announcement is silent on what happens when an automation's run wants to do something irreversible. The pattern-language series's authority argument applies: "who may modify the system's state is a property of the harness, not of the model's reading comprehension."

4. Fresh runs are bounded intelligence. The deliberate amnesia caps what an automation can become. The daemon that cannot remember cannot compound: its value per run is constant, not growing, unless the user externalizes memory into connectors and skills — which is exactly the governance work the always-on survey says nobody has built yet.

The screenshot's run history is the most honest detail in the announcement: Runs 4 · Succeeded 2. Half the visible runs failed, and the announcement does not say what happens then — whether failure is notified, retried, or logged. That is the boundary every agentic system must declare: the guarantee stops where the failure policy stops.

What to steal

Read the announcement the way this blog reads every harness: separate the system from the model, and the design decisions from the marketing.

What Grok Automations gets right: the trigger as a first-class harness primitive — the daemon's "when" is a configuration, not a prompt; run history as provenance by construction; the fresh-request discipline that keeps runs bounded and legible; the honest meter (predictable schedules free, unbounded email triggers paid); and the chat-to-loop flow that turns "prompt me" into "loop for me" (loop engineering, productized).

What it leaves open: the injection gatekeeper between email and the run; the budget ceiling on trigger-driven runs; the approval seam before consequential actions; and the memory story beyond run history. Each of these is a named pattern in the series with a documented tradeoff — the product has shipped the daemon half, and the governance half is still this industry's open problem.

The durable daemons definition ended with a warning and a promise: "Agency is not discovered. It is designed." Grok Automations is the first time a major consumer platform designed it at scale — describe once, run forever, report back. The design is real, the economics are real, and the guarantees stop exactly where every announcement that omits its failure policy stops: at the boundary the platform chose not to declare. Name the boundary, design within it, and verify the design works — that is the engineering method, and it is the whole pattern language in one sentence.

References

DeepSeek Harness: Everything Is a Plugin

On August 13, 2026, DeepSeek AI open-sourced DeepSeek Harness (dsh) — an agent harness whose architecture is 'everything is a plugin', powered by Cordis, the same framework whose formal model this blog covered two days earlier in 'A Programming Paradigm for Spatiotemporal Composability'. In four days it passed 149,000 GitHub stars — headed toward 150,000 — before the repository had a tagged release. This post reads the documentation site the way the project's own docs ask you to read it, then distills the key design decisions as eleven insights — the enforced append-only log, the loop as a plugin row, policy as waterfalls plus monotonic guards, capability seams, interop-as-strategy (the harness runs unmodified Claude Code hooks and speaks Claude's compaction format), code mode, Typert's type-graph mirror, the frozen request, per-agent scope, the training environment as product, the harness building the harness — each compared against the harnesses it resembles or contradicts (OpenHands, SWE-agent, MiniCode, Claude Code, Codex, Aider, Gemini CLI, Harbor/Terminus-2, Meta-Harness, terminal-bench-rl). Two independent teardowns add the missing dimension: the append-only log is a billing design as much as a correctness design — respecting its own golden rule is worth a ~120x prefix-cache discount on DeepSeek's price list, and the same model's success across eight harnesses swings from 47% to 67% while cost per finished task varies 7x. It reflects on what the design actually buys — and where the guarantees stop: file-effects-only sandboxing, an API narrowing that is not a security boundary, and a calculus that guarantees clean removal, not harmlessness while present.

deepseekagent-harnesscordispluginsspatiotemporal-composabilityharness-engineeringsession-logcapability-seamssandboxingself-modifying-agentsevent-sourcingopenhandsswe-agentminicodeclaude-codecodexaiderterminal-benchtypescriptpython-sdk

On August 13, 2026, DeepSeek AI opened a repository called DeepSeek Harnessdsh, an open-source agent harness in TypeScript, MIT-licensed, whose one-line description is the whole design: "Everything is a Plugin." Four days later it had more than 149,000 GitHub stars, and the count was still climbing toward 150,000 as this post was written. That is an extraordinary reaction to any repository, and it is even more extraordinary for one that is explicitly a developer preview promising compatibility-breaking changes, with no external plugin ecosystem yet and no tagged stable release (the shipped artifact is 0.1.0-rc.7).

This blog met this repository's intellectual foundation two days after it appeared. On August 13 — the same day the Harness repository went public — an 88-page paper was published from the cordiverse/paper repository: A Programming Paradigm for Spatiotemporal Composability, by Yifan Shi and Wei Zhang of Peking University and Tianyi Cui of DeepSeek-AI. I summarized and reflected on it in Spatiotemporal Composability: The Missing Calculus for Self-Evolving Agents. The paper formalizes dynamic composition — components loaded, unloaded, and reconfigured at runtime — with two guarantees: temporal composability (when a component is removed, everything it did is undone) and spatial composability (when the world changes, components whose dependencies changed reconnect or wait, never crash). Its authors extracted the theory from Cordis and Koishi, a plugin framework and a chatbot framework with 4,000+ community plugins — which is why the theory did not arrive as a proposal: it was distilled from software that had already shipped.

DeepSeek Harness's README says in its second line: it is "powered by Cordis, whose design is described in A Programming Paradigm for Spatiotemporal Composability." The calculus this blog called "the missing calculus for self-evolving agents" is not a proposal anymore. It is the runtime under DeepSeek's own agent harness. This post reads the documentation site — deepseek-harness.github.io/deepseek-harness/en/ — the way the project's own docs ask a reader to read it, then distills the key design decisions as insights, each compared against other harnesses, and reflects on what the design actually buys and where its guarantees stop.

The documentation is part of the design

The English docs live at a VitePress site that mirrors a bilingual source tree in the repository — every page is projected from Markdown in docs/, with a Chinese counterpart kept in sync by a custom Git merge driver. The landing page is a one-line redirect to Guide → Use the Web UI, which is the honest shape of the project: the fastest path is npx @deepseek-ai/dsh web, which serves a web UI at http://127.0.0.1:3080, where you add a DeepSeek API key, choose a workspace, and send the agent a task.

Three nav sections matter more than their titles suggest. Guide is user-facing (Web UI, model configuration, a Python SDK). Development is plugin authoring, including a seven-part Cordis tutorial. Reference is the interesting part: an architecture page, a Cordis primer, capability seams, the agent lifecycle, the tool execution pipeline, then a long list of subsystem pages — one per capability (sessions, tools, shell, sandbox, approval, compaction, skills, subagents, workflows, and more) — each ending in a generated Cordis API section with the exact service methods and event signatures, byte-identical to the source, freshness-checked by a build gate so a JSDoc edit cannot ship without regenerating the catalog. The docs also include generated catalogs of every plugin config field, every model-facing tool schema, and every durable session event.

This is not documentation hygiene trivia; it is a design decision worth naming early. The repository treats its documentation as a machine-checked artifact: type declarations pasted into docs are verified drift-free against source (verify-type-equiv), generated catalogs must be byte-fresh (verify-cordis-catalog), and the AGENTS.md documentation standard enforces word budgets per document tier. The reason is stated plainly in the architecture page: "We recommend using an agent to explore the codebase." The documentation is written to be read by agents — which is exactly what a harness company's documentation should be, and almost none of it is.

Key design decisions, read as insights

The section above read the docs as a map. This section reads the code and the design records as a set of decisions — each one a choice with an alternative that lost, recorded in the repo's own Agent Notes, and a comparison that makes the stakes visible. Two framing notes. First, DeepSeek Harness is not the first harness to reach several of these destinations: OpenHands built an event-stream backbone years earlier, SWE-agent argued that the interface between model and computer is the product before DeepSeek shipped one, MiniCode's conversation client is the explicit reference shape behind the request pipeline, and the Claude Code and Codex hook protocols are things the harness implements as a bridge rather than invents. Second, the comparisons below are drawn from each harness's own documentation and code, not from marketing — and the second research pass added source-level findings the docs alone would not have yielded: the compaction checkpoint format, the hooks.json bridge, and Typert's type-graph generator, each covered in its own insight.

Insight 1 — History is the boundary: the append-only log, enforced

The spine of the design is the session log: an append-only log of typed events from which the model's message history is derived by a pure fold, never stored. What makes the decision an insight is the enforcement, which the source makes concrete. Every message-producing append must carry a surfaceOp marker and sourceEventSeqs; the surface manager throws on a raw event without a marker, on a tool/result replacement that changes anything but content, on a non-contiguous sequence number. The derived messages are shared, deep-frozen objects — "mutating logged history through a projection is unrepresentable (it throws)". The turn flow (claim → agent/pre-stepstep/start → model request → tools → step/endagent/turn-stoppingturn/end) is the same shape at every level: durable facts on the log, live control on agent/* events.

The comparison that frames the decision is inside the repo's own note: MiniCode's LLMClient, a stateful conversation client "appended to — never rebuilt — as the conversation advances". DeepSeek adopted the discipline and rejected the shape: a client is "a second operative truth beside the log — the two drift and nothing notices". OpenHands is the closest architectural cousin — its EventStream of actions and observations is also the backbone, replayable and searchable — but the LLM context there is derived by each agent implementation rather than gated by one logged-surface invariant, and the stream is a coordination backbone more than a billing boundary. Claude Code keeps JSONL transcripts for replay, but the transcript is not the source of the next request. Aider uses git as its history — a version of "never rewrite the past" at a much coarser granularity.

The current generation splits into the same two answers with different mechanics. Gemini CLI ships "conversation checkpointing to save and resume complex sessions" — snapshots, not an immutable log. Claude Code's checkpoints roll back the working tree through git: they undo the files an agent changed, not the conversation that changed them. DeepSeek's is the third answer — an append-only log from which everything derives. The other two make the past recoverable; this one makes it unchangeable.

The insight: history is not a record of the agent; it is the boundary of what the agent can do. A harness that edits history gives the model a world where the past is negotiable. DeepSeek's gives it a world where the only move is to append.

Insight 2 — The loop is a row: microkernel over monolith

A running dsh is a plugin tree composed at boot from ordered layers. A profile is a named composition (web and headless ship as templates); a bundle is a distribution format — config rows plus the code they mount (dsh-base is the first layer of every profile; dsh-web-app adds the browser; dsh-headless is a one-shot runner with no server). Layers apply to an empty entry list in order, and a patch targets a row by id and replaces its whole config, or inserts new rows. dsh --profile web --dump-config prints the tree your machine actually boots, and the docs' claim is total: "any row it prints can be replaced by a patch of your own." The model adapter, the tool registry, the session log, the agent loop itself: all rows.

One running harness — a profile stacks bundles (dsh-base, dsh-web-app, dsh-headless), patches replace rows, and the composed tree exposes every service as a ctx key. dump-config prints the tree; any row it prints can be replaced.

The framework beneath is Cordis, in five ideas: a plugin is an object that implements Service; a context is a repository of services (ctx.sessions, ctx.tools, ctx.llm, ctx.agents, ctx.agentLoop); dependencies are declared via inject so load order is expressed as requirements; services communicate through typed events dispatched as emit, waterfall, parallel, or serial; and registrations are reversible effects — installed through ctx.effect() or ctx.on() so reload and teardown unwind them. The microkernel note records why: the alternatives (a koa-style middleware stack, an explicit phase state machine plugins insert into) would "re-implement dispatch, disposal, and reload semantics that Cordis's native event system already provides"; as Cordis effects, listeners get HMR and disposal for free. The loop itself is a plugin — @deepseek-ai/dsh-agent-loop is "the only concrete loop plugin and is itself swappable — nothing outside it may depend on it".

The comparison sharpens the wager. Claude Code's hooks are the closest mainstream analog — a fixed, documented surface of matcher-grouped extension points — but you cannot add an extension point without the vendor shipping it, and hooks are policy, not composition. LangGraph and AutoGen treat the loop as a graph the developer authors; DeepSeek treats it as a row the developer can disable (disabled: true on the loop's config rule means there is no agent). VS Code's extension host is the classic microkernel, and its "Developer: Reload Window" button is the price of dynamic composition without reversible effects — the exact problem the spatiotemporal paper formalizes. The trade is real, and the repo pays it in incident reports: twice, a quiet configuration mistake silently broke the product, and there are now 27 pre-release checks that catch that class of mistake.

Insight 3 — Policy runs in waterfalls; the final denial is a monotonic guard

Every tool call passes through a pipeline the docs render as a flowchart, and the ordering is the doctrine: waterfalls for extensible policy, guards for final authority. tools/pre-execute runs as a waterfall — hooks, permission, sandbox decisions; a listener that owns the decision short-circuits by returning without next(). Then monotonic guards run — they deny or abstain and can never force-allow, so owner policy that must not be reordered cannot be argued around. The approval seam is fail-closed: allowed-once is the only granting outcome; a missing, throwing, or non-conforming answerer resolves to unavailable. Then tools/execute wraps the tool body as around-dispatch (timeouts, retries, metrics), tools/post-execute may accept, block, or rewrite, and a definition-owned finalizeContent callback — snapshotted at the moment the call starts, so no later mutation can change what the outcome must be — enforces the last content-only invariant before the frozen tools/result.

The tool execution pipeline — model → tool/call → pre-execute waterfall → monotonic guards → approval → execute → post-execute → frozen result. Policy attaches at documented points; the loop does not change.

Two details show the care. A call collapsed by code mode (see Insight 6) is denied before the policy pipeline — "pre-execute listeners, approval ask, and guards must never observe — or worse, approve — a call that can only fail". And cancellation is a contract, not a cleanup: the registry tracks whether the tool body started and distinguishes ABORTED_BEFORE_DISPATCH from ABORTED, fusing the caller's signal into any around-wrapper replacement so "cancellation never abandons the body". The enforcement layer hangs off the same pipeline: sandbox modes (read-only / workspace-write / danger-full-access, filesystem effects only, enforcement reported full or partial), permission presets (workspace-write+ask, danger-full-access+never), the scrubbed-env and private-temp-file rules from the defensive-patterns ledger. The local backend is worth one more sentence: it functionally probes competing confinement candidates — bwrap then Landlock on Linux, Seatbelt on macOS, a Windows ACL restricted-token runner — picks the first that actually works on this kernel, and fails closed if none does; on Windows, every workspace gets a standing SID grant while each live session receives a random private temp directory with its own capability, revoked on dispose.

The comparison: Claude Code's permission system (allow/deny/ask) and hooks cover the same roles, but they are configured surfaces; SWE-agent's ACI is a fixed, deliberately minimal interface; OpenHands routes policy through a security analyzer. The DeepSeek distinction is the ordering guarantee — extensible policy upstream, monotonic final denial downstream — plus a fail-closed approval contract with a single answerer.

Insight 4 — A capability is a seam, not an interface

A swappable capability has three roles that change at different rates: a Service Definition owning the ctx.<key> and the vocabulary, one or more Service Providers implementing it, and a Consumer (usually a model-facing tool) programming against it. The capability-seams note records why: "swapping a local executor for a sandboxed one would churn the tool schemas the model sees, even though the model-facing contract never changed." packages/shell is the reference template — dsh-shell (definition), dsh-bash-local / dsh-bash-sandbox (providers), dsh-tool-bash (consumer) — and the rule is to split only when roles evolve independently. The subagent seam shows the payoff: providers registered by name include in-process spawn, fork, ACP, Codex, Claude Code, and the dsh SDK itself, all behind one interface; a provider advertises its start-time capabilities statically, and a request needing one it lacks is rejected loud before any run exists.

The comparison: MCP is the industry's attempt to standardize the tool seam, and DeepSeek supports it — but treats it as one consumer family, not the seam itself; the seam is the three roles behind a stable context key. Codex and Claude Code each ship one implementation of each capability. The subagent registry's use of the vendors' own kits (Anthropic's agent kit, OpenAI's app server) is the inverse move: instead of reimplementing a rival harness, mount it as a provider.

Insight 5 — Interop as product strategy: adopt rivals' formats

The seam philosophy has a product-strategy face, and it is visible in three places where the harness deliberately implements other products' formats. The Claude Code hooks bridge reads a real hooks.json — the matcher-grouped configuration Claude Code itself uses — and runs unmodified hooks (SessionStart, PreToolUse, PostToolUse, Stop, subagent hooks) against DeepSeek's typed extension points, mapping Claude's decision vocabulary onto them. The compaction engine produces a checkpoint in the same eight-section structured Markdown that Claude Code's own auto-compact produces — "Primary Request and Intent", "Files and Code", "Next Step", and the rest — delivered as the trailing user message, so a session compacted by one harness can be resumed in the other's shape. The subagent backends start the vendors' own binaries and speak their own protocols. And the workflow engine's meta block "matches the Claude Code dynamic-workflows meta block" verbatim.

The strategy is consistent: when you adopt a rival's format, their files and their habits become portable into your product — the side effect Aaron's teardown names. But it cuts both ways, and the insight is the tension: the harness is a compatibility layer for the vendors' surfaces, not a standard-setter. The one format DeepSeek is not adopting is the one it hopes to set — the append-only, prefix-preserving history model. That is the bet with a deadline this post returns to in the reflection.

Insight 6 — Code mode: the tool list is a presentation, not the interface

The tool registry owns presentation modesnative, code, both — chosen by config. In code mode, the model gets a generated TypeScript SDK over the visible tools plus one reserved run_code transport: it writes a program that loops, branches, and runs several reads at once, and only what it prints or returns comes back into the conversation. The note's framing is borrowed openly from Cloudflare's Code Mode: "LLMs are better at writing code than at emitting tool calls, because they have seen millions of lines of real code and comparatively few contrived tool-calling traces." The program runs in a fresh worker thread with an empty environment, heap and wall-clock caps, and hard termination — and the trust posture is stated without hedging: run_code is "bash-equivalent by design — no unsafe-acknowledgement flags — because the harness already ships dsh-bash-local, which executes arbitrary model-written shell commands with strictly more ambient authority." Every sub-call traverses the full tool pipeline with the parent's token; deferred contexts are held until after the outer result so call/result adjacency survives.

The comparison: Cloudflare's Code Mode is the cited antecedent; SWE-agent's ACI research is the academic antecedent of "tool design is the product" — its authors found that the choice and shape of editing and search tools mattered more than the model. Most harnesses still present tools as JSON-schema lists and round-trip every intermediate result into context. The workflow seam takes the same principle one level up: the model writes an orchestration script — with agent() calls that spawn subagents under caps, executed in a vm realm inside a per-run worker thread — so the model curates not only its own context but the delegation tree.

Insight 7 — The type system is the runtime's mirror

Under the whole stack sits a mechanism no other harness in this comparison has: Typert, a compiler-independent TypeScript type-graph analyzer that extracts a model of the codebase's types, services, and events from the source and emits artifacts from it. The same analysis produces the Host↔Client RPC contracts (methods marked @Remote), the tool-schema catalog, the docs' byte-identical Cordis API sections, the code-mode SDK the model programs against, and the API catalog the cordis_inspect tool serves to the agent mid-session. One representation, five consumers — and the consumers include the model itself.

The comparison makes the move legible. Most harnesses hand-write their API docs or derive them loosely; MCP defines tools at runtime; TypeScript type reflection is usually spent on runtime validation (zod-style), not on generating the agent's own view of the runtime. DeepSeek's version means the docs page, the RPC boundary, the model-facing SDK, and the self-inspection tool cannot drift from each other or from the code — the freshness gates (verify-cordis-catalog, verify-type-equiv) are the enforcement. It is the mechanism that makes the "documentation is written to be read by agents" claim true: the agent reads the same generated contracts the docs print.

Insight 8 — The request is a frozen artifact; the bill is the assertion

In the loop source, every request is composed the same way: the header folds from the log (logged as request/header with reasons initial, resume, or change), the agent/request waterfall may propose the config, and the final GenerateOptions is deep-frozen and marked so the invariant checker can recognize conversation work. The invariant module — a separate companion plugin — replays the log in a fresh session on every request and compares its derivation against what the loop is about to send: "the live cache cannot vouch for itself." The rejected alternative, recorded in the note, is the telling one: detect-and-report (compare consecutive requests, warn on divergence) was refused because "a warning arrives after the bad request has already shipped." On top sits a paid test run against the real API that fails unless the second request in a conversation reports cached input tokens. The bill is the assertion.

The comparison: no mainstream harness ships an invariant checker over request reconstruction. Claude Code and Gemini use provider prompt caching, but the discipline is configuration, not a gate; providers report cache reads in usage, but harnesses do not fail builds on them. The bill-as-assertion is the design's most novel move — the test suite includes the price.

Insight 9 — Per-agent scope: registration is visibility and lifetime

The scope primitive makes one fact drive both: a registration made through an agent's agent.ctx is visible only to that agent and owned by that agent's lifetime. Scoped registrations shadow global ones — the per-agent persona and per-agent tool-variant mechanism — and scope-filtered dispatch carriers route an event about one agent's activity to that agent's listeners. The setup window — after the agent object exists, before it is published — is where a creator composes an agent's scoped world.

The comparison: VS Code scopes contributions to extensions, not per-instance; OpenHands sessions are per-conversation, but capability sets are largely global; LangGraph threads state explicitly through the graph. DeepSeek's per-agent plugin mounting is what makes the four presets (standard, code, minimal, creation) work as per-session compositions instead of global profiles — and it is what makes the "one row" claim in Insight 2 honest: the row can be scoped to a single agent.

Insight 10 — The training environment is the product

The minimal preset is a two-tool composition — a persistent shell plus str_replace_editor — with a system prompt of exactly one sentence: "You are a helpful software engineer assistant." The repo's own note names its purpose: the minimal preset owns the complete Claude SWE-compatible RL agent composition, with the persistent bash environment the RL harness uses. The harness produces the trajectories; the trajectories feed post-training; the trained model goes back to work inside the same harness. DeepSeek open-sourced a piece of that loop.

The comparison makes the decision legible. SWE-agent was built explicitly to produce RL training trajectories — the ACI paper is the academic origin of "the interface is the design lever". terminal-bench-rl trains long-horizon terminal agents with GRPO at 32×H100 scale and reports the sharpest version of the insight: harness-level improvements ("native tool calling, multimodal support, execution optimization") lifted scores without training. Meta-Harness (Stanford IRIS) reached 76.4% on Terminal-Bench 2.0 and was itself "discovered through automated harness evolution". The harness is not just the product — it is the curriculum, and the community is now evolving it like a hyperparameter.

Insight 11 — The harness builds the harness

The repository's AGENTS.md is a standing-order document for agents; decisions live in Agent Notes that must ship in the same PR as the decision, each with an "alternatives considered" section; generated catalogs are freshness-gated; keyless snapshot tests replay a recorded session log against expected output; per-file 100% coverage is the CI gate; and the self-referential toolset (cordis_inspect, cordis_mount, cordis_unmount) lets the agent inspect and mount plugins into its own live runtime, with unmount waiting for every owned effect to reach quiescence.

The self-referential toolset — the agent inspects, mounts, and unmounts plugins against its own live runtime. Reversible effects make unmount wait for quiescence; the vm narrows the API but is explicitly not a security boundary.

The comparison: CLAUDE.md and AGENTS.md instruction files are now industry standard — Codex, Cursor, and Gemini all adopted them — but the enforcement is the unusual part: notes as PR gates, verify-* generators that fail on drift, postmortems that do not count until their check is proven to turn red, word budgets per document tier. The teardowns' numbers — 12,293 commits in 64 days, more Markdown files than TypeScript files — are what that enforcement buys. And the self-referential toolset is the honest statement of the whole design's posture: the vm is an API narrowing, not a security boundary — "an opt-in development tool with bash-equivalent trust."

Two teardowns of the same codebase

Within two days of the release, two independent teardowns went up: Cloud Codes' 21-minute DeepSeek Harness Architecture: Insane Software Engineering Behind It (Aug 15) and Aaron — AI-native builder's 10-minute What I Learned From DeepSeek's Harness (Aug 16). Both read the same repository and converged on the same two findings: the session log is a billing design as much as a correctness design, and the harness — not the model — is the product. The insights above read the design the way its authors record it — docs, source, and decision notes. The teardowns read it the way a skeptic would: what is this worth, what does it cost, and what breaks? Both videos pin their claims to the repository, and I checked their key claims against the repo's own Agent Notes and the shipped config files before writing this section; the two videos corroborate each other on the facts that matter.

Cloud Codes — DeepSeek Harness Architecture: Insane Software Engineering Behind It (Aug 15, 2026, 21:27): the golden rule, the 120x prefix-cache discount, the invariant module that disbelieves the loop, the compaction fix, the four presets, Cordis, and the 683 agent notes. Watch at youtube.com/watch?v=1NyOG9z9RT0

The golden rule is worth 120x

The Cloud Codes video opens with the rule it found buried in the codebase: "Once something has been sent to the model, you do not go back and change it. A wrong file path sitting in the history, you leave it there and append a new line saying it was wrong." That sounds like bookkeeping. On DeepSeek's own price list it is worth about 120 times.

The mechanism is prefix caching. Because the model forgets, every step resends the whole conversation from the top — a serious coding session pushes past 100,000 tokens through the pipe on one step, then does it again seconds later. Providers rescue the economics with prefix caching: if the leading tokens of this request are word-for-word identical to the last one, the provider restores the computation it already did and starts at the bookmark. Word-for-word is the entire condition — change one character on page three and the bookmark is worthless, the book gets read again. DeepSeek publishes exactly what that costs, which is unusual: at launch-week prices on the pro model, a cached input token runs about a third of a cent per million tokens; the same token uncached runs about 43.5 cents. That is the teardown's arithmetic — it rounds to 120x — for identical text in two neighboring columns of one price table, the difference decided entirely by whether the harness disturbed something it had already sent.

So the real design question for a harness is not which tools to offer or how to word the prompt; it is: can you run a 200-step session without ever going back and editing what you already sent? Most harnesses answer no, because agents edit their history — summaries replace old turns, bulky tool output gets trimmed, a stale file copy gets swapped for a fresh one. Every one of those is sensible; every one reaches backwards into the transcript; every one throws the bookmark away at exactly the moment the conversation is longest and rereading costs the most.

DeepSeek's answer is the append-only log this post already described, and the video's account matches the repo's own Agent Note (2026-07-05-reconstructable-requests) almost verbatim: model-visible ⟺ durably referenced — anyone holding the log, its referenced objects, and the pinned code version reconstructs every loop request byte-for-byte. The note adds the sentence that tells you this team understood their own design better than most write-ups of it: "Prefix-cache stability is corollary #1, not the headline" — "stability is emergent, not managed." They did not set out to build a cache optimizer; they built a log you cannot edit, and the prefix stability fell out the other side.

The note also records the enforcement they chose and the alternative they rejected. Derived messages are deep-frozen — mutate logged history through a projection and it throws, so breaking the rule stops being something you avoid and becomes something you cannot express. The rejected alternative was detect-and-report: compare consecutive requests and warn on divergence — rejected because a warning arrives after the bad request has already shipped. Instead they shipped a module whose entire job is to disbelieve the agent loop: on every request it builds a fresh session, replays the log from the beginning, derives the messages again from scratch, and compares its answer against what the loop is about to send — "so the live cache cannot vouch for itself." On top of that sits a paid test run against the real API with a real key that fails unless the second request in a conversation reports cached input tokens. The bill is the assertion. Cold cache, red test.

The video's best story is the compaction bug that shows the design paying for itself. When a conversation outgrows the context window, the harness must summarize the old part so the work can continue — and the first version of the summarizer did the obvious thing: sent a fresh "you are a summarizer" system prompt followed by the conversation to condense. The system prompt sits at the very front of the request, exactly where the cache starts keying; one differing first token invalidates the entire prefix. Every compaction paid full processing price for the whole replayed history twice — once for the request that tripped the limit, once for the summary, at exactly the moment the history was longest. The fix is in the repo's own Agent Note (2026-07-21-compaction-summary-prefix-cache-reuse): the summarization directive moved from the front of the request to the end — the auxiliary call now reproduces the last routed request's prefix verbatim and appends one trailing instruction, making it a genuine prefix-extension of the warm request. Tools ride along on the summarization call even though the summarizer never calls one, because dropping them would misalign every following token.

The four presets: from training to self-modification

The video also surfaces the presets — the four agents "wearing the same skin" — which I verified in the shipped config files under apps/cli/config/agent-presets/: standard, PTC/code mode, minimal, and creation. Code mode replaces the tool list with a generated TypeScript interface over the tools plus one run_code transport: the model writes a program that loops, branches, and runs several reads at once, and only what it prints or returns comes back into the conversation — the model curates its own context instead of drowning in it. The reasoning is borrowed openly from Cloudflare: models have read millions of lines of real code and comparatively few tool-calling traces, so ask them for the thing they have actually seen. The program runs in a fresh worker thread with an empty environment, a heap cap, a wall-clock cap, and hard termination — containment, the video notes, not a security boundary, with authority comparable to the bash tool.

Minimal mode is the one worth pausing on. Two tools — a persistent shell whose working directory and environment survive between turns, plus str_replace_editor — and a system prompt of exactly one sentence: "You are a helpful software engineer assistant." The repo's own Agent Note (2026-08-10-minimal-preset-owns-rl-composition) says why: the minimal preset owns the complete Claude SWE-compatible RL agent composition, with the persistent bash environment the RL harness uses. The video's reading is direct: the minimal preset is a training environment shipped to you unchanged as an option in a dropdown beside the coding agent you are going to use anyway. The harness produces the trajectories; the trajectories feed post-training; the trained model goes back to work inside the same harness. DeepSeek open-sourced a piece of that loop, with the notes describing why each service in it belongs there.

Creation mode is the self-modifying preset this post's earlier section already examined through the cordis_* tools; the video adds the product framing — it sits in the same dropdown, its documentation says "treat a session on this preset as shell access," and its stated purpose is that a person can ask an agent to write another agent, and the preset that agent writes becomes something other sessions can mount.

The subagent backends and the memos

Both teardowns were stopped mid-scroll by the same feature: the subagent registry ships backends for Claude Code and Codex — not reimplementations, not scraped protocols. The Claude Code backend calls Anthropic's own agent kit at a pinned version, resolves the real Claude binary installed on your machine, and hands the kit that exact path; the Codex backend starts OpenAI's own app server over stdio and speaks its protocol. A DeepSeek agent running a DeepSeek model can hand a self-contained task to Claude Code, which works in the same directory, returns one answer, and is torn down — process tree and all, with the teardown proven before the call returns. The rejected-alternatives note explains why the shortcuts were refused: talking to the model directly or hand-writing the command-line protocol would bypass each product's official integration surface and prove nothing about approvals, tools, or cleanup.

And the memos — the number Cloud Codes dwells on: 683 English design records in .agents/notes/, plus a Chinese translation of every one, organized by status (proposed, implemented, rejected, archived), with every non-trivial change required to add or update a note in the same pull request, and every note required to carry an "alternatives considered" section. "A decision recorded without what it beat invites re-litigation." A script validates the headers, cross-checks the status against the folder, and rejects proposal language inside a note claiming to describe shipped reality; archived notes are frozen in an append-only manifest and must not be treated as authority. Cloud Codes' best evidence that this works is the postmortem of a web agent that was asked to change the interface it was itself running inside: it edited the source, started a dev server on a second port, validated that, then started a third server on a third port — three servers, one user, and the agent checking the only one nobody was looking at. The incident is readable in that detail because every step is traced by sequence number through the same append-only log the cache rule exists to protect. The design that keeps the bill down is the design that made the incident legible.

Aaron — AI-native builder — What I Learned From DeepSeek's Harness (Aug 16, 2026, 10:06): the eight-harness benchmark, the 44/3 event log, the one-row agent loop, and the process behind 12,293 commits. Watch at youtube.com/watch?v=5vEEBhbfUWw

Same model, eight harnesses: the harness is the product

The second video starts with the numbers that motivated it: the same model ran thirty real work tasks through eight different harnesses — task success swung from 46.7% to 66.7%, and cost per finished task varied by a factor of seven. Nothing about the intelligence changed; what changed was the layer between you and the model. The video reports that the creator of the top-scoring harness read the DeepSeek repository and said it was the first time something new in this space made him want to revisit his own choices — when the winner reads a rival's homework and starts rethinking his own answers, the homework is worth reading.

Its four designs are the cache discipline, the event log, the one-row loop, and the process. The cache discipline: many frameworks write the current time into the top of the system prompt, so the prefix never matches and every request runs at full price forever; DeepSeek keeps the clock out of the prompt, sorts tool descriptions in one fixed order so nothing reshuffles, and runs a live test that flatly asserts every request after a session's first must hit the cache — "if any change breaks the prefix, the build goes red before the money burns." The event log: 44 kinds of events, exactly three visible to the model (user messages, assistant messages, tool results); the system never stores the conversation it sends — it recomputes the model's context from the log before every request, and a runtime check compares the outgoing request against what the log says it should be and refuses to send. The everyday superpower of that design is debugging: when an agent does something inexplicable, you stop guessing — replay the log to that step and the exact context the model saw is in front of you; crash at step 80, recompute and continue, same log, same context, as if nothing happened.

The one-row loop makes the everything-is-a-plugin claim legible in config: the loop that drives the agent — calls the model, runs tools, decides whether to continue — is one ordinary config rule, and disabled: true on that rule means there is no agent. The main loop and a tiny badge plugin are equals in the eyes of the config. Code mode is standard mode plus one appended rule; that rule alone flips how tools are presented to the model. The price of that flexibility is written in the repo's own incident reports: twice, a quiet configuration mistake silently broke the product; once, 178 green tests and a full coverage set on top of a system that died the moment a real editor connected. Their answer both times was a pre-release check that catches the class of mistake — there are 27 of those checks now. That is the bill for everything-is-a-plugin: pay it one crash at a time.

The process numbers are the fourth design and the thing that most surprised both teardowns: 12,293 commits in 64 days; the top contributor made about 5,000 of them; the branch names in the merge history count "worktree" 210 times and "codex" 209 times; there are more Markdown files in the repository than TypeScript files. The previous section's notes and checks are the mechanism behind those numbers — and the video sharpens the theory: this process is bureaucracy in a human team; in an agent team, it is guardrails, because the writer never gets tired.

Both videos end with the same honest caveats from DeepSeek's own known-limitations pages — the runaway-loop guard only sends reminders and eventually goes quiet, the file tools have no timeout at all, and early testers say daily experience still trails Claude Code and Codex. If you need work done this week, neither video recommends switching. And both end on the business model. Cloud Codes notes that on August 16, three days after the harness went free, DeepSeek's pricing page introduced peak and off-peak billing — uncached input goes from 43.5 cents to $1.32 at peak, output from 87 cents to $3.96: the harness gets cheaper and the tokens get dearer in the same week. The append-only design is what makes that survivable: a harness that keeps your prefix warm keeps most of your traffic in the cached column, at a fraction of the uncached price. Aaron's reading is the same shape: Anthropic keeps its harness closed and wired to a subscription — a moat around the model — while DeepSeek gives its harness away and it reads everyone else's files: a funnel for the thing they actually sell. And the side effect he names is worth keeping: when a vendor adopts its rivals' formats, your files become portable, whichever harness you run.

That last point connects to what this blog argued in Every Token Has a Price Tag: the unit got cheaper but the task did not, and the shape of the bill is a design decision. DeepSeek Harness makes the shape of the bill a first-class architecture property — the append-only log is not only a correctness design and an audit design; it is a billing design, worth roughly two orders of magnitude on the input side when the harness respects its own golden rule.

Reflecting: what the design buys, where it stops

The bet. Building the whole product on a plugin framework with reversible effects is a wager that harness engineering is, at bottom, a dynamic-composition problem: the hard parts of an agent runtime — adding capabilities, swapping providers, reloading policy, cleaning up after a failed experiment — are the hard parts of runtime plugin management. The wager is coherent, and the evidence base is not trivial: the framework underneath shipped in Koishi with 4,000+ community plugins before the theory was written down. The paper's own conclusion is the honest version of the story: the theory was extracted from a working system, not written first and applied later. What the formalization buys is the second product — dsh did not have to re-solve plugin lifecycle, HMR, dependency resolution, or teardown; it inherited them as a vendored, formalized substrate and spent its own complexity budget on agent concerns: the session log, the seams, the pipeline, the policy. That is exactly the economics this blog keeps returning to in harness-engineering-best-practices-for-ai-agents and go-is-good-for-harness-pipelines: the harness is where composition complexity concentrates, and the team that treats composition as the product wins the cost curve.

The harness is the product. Both teardowns end on the sentence this blog has been circling for months, and the numbers are finally there to back it: the harness is the product. Cloud Codes reads the public Terminal-Bench board the same way — the same weights score 83.8% under one major harness and 80.4% under another, 3.4 points decided entirely by software with no weights in it — and Aaron's own benchmark found the same model finishing between 47% and 67% of real tasks across eight harnesses while cost per finished task varied sevenfold. No weights changed in any of those comparisons. The wrapper changed. The comparison research in the insights above makes the point stronger, not weaker: SWE-agent's ACI paper argued the interface is the lever; Meta-Harness reached 76.4% on Terminal-Bench 2.0 and was itself found by automated harness evolution; terminal-bench-rl reports that harness-level improvements lifted a Qwen3 agent to the top of its class without training. That is why a lab publishing its harness under an MIT license is giving away something much closer to the product than the announcement suggests: the harness is not the part with the intelligence, and it is the part with the leverage.

The guarantees stop where the paper's do. Observational equivalence, not literal restoration — unplugging a component does not restore the heap's exact layout, only what the declared dependencies observe. The sandbox vocabulary covers file effects only; network and process visibility are out of scope by declaration. The self-modification toolset is bash-equivalent trust, explicitly not a security boundary. And the deeper caveat survives from the paper: recovery guarantees clean removal, and for agents the interesting failure is the one where a faulty self-modification disables the very process needed to recover — the calculus makes removal safe, but removal still has to be invoked.

The training loop is the product, too. The minimal preset is the sharpest thing either teardown found, and it deserves its own reflection. The harness DeepSeek hands you for free is the same environment its models are trained in: a two-tool, one-sentence-prompt composition whose Agent Note names its purpose outright — the Claude SWE-compatible RL agent. The harness produces the trajectories; the trajectories feed post-training; the trained model goes back to work inside the same harness. That closes a loop most labs keep private, and it is the strongest form of this blog's agent-harnesses-need-tasks-that-fight-back claim: the tasks a harness runs are not only an evaluation, they are the curriculum. It is also why "everything is a plugin" is not a design aesthetic but a training-infrastructure decision: the row that defines the training environment is the same row you can edit in the shipped product. And the reversible-effects guarantees matter beyond the product for the same reason: a training run is millions of agent steps, and the promise that teardown fully undoes what a component did is what keeps one experiment's state from leaking into the next run's trajectories. Cloud Codes ends with the question the loop raises: "The model is learning your harness. Is your harness learning anything back?" The honest answer, as with every guarantee in this post, is that the loop optimizes whoever runs it — which is why the same week the harness goes free, the tokens get dearer.

The preview caveat. More than 149,000 stars in four days is a demand signal, not a validation of production readiness. The README's first declaration is "developer preview... THERE WILL BE COMPATIBILITY-BREAKING CHANGES." The session log carries SESSION_FORMAT_VERSION at 0 with no compatibility promise; backends reject old on-disk formats. The release cadence is remarkable — 0.1.0-rc.7 and 2,600+ merged PRs by day four — but that velocity is exactly the argument for treating the docs as the product's present shape and the code as a moving target. The single-process runtime is another boundary worth naming: everything runs in one shared process, so the compositional guarantees are process-internal; cross-process isolation is delegated to the sandbox providers and the subagent seams, which is where the real blast-radius questions live.

The bet with a deadline. Cloud Codes ends by making the sharpest claim in either video falsifiable: within a year, at least two of the major harnesses — Claude Code, Codex, Cursor's CLI, Gemini's, Aider, the LangChain stack — will publicly document an append-only or prefix-preserving history model. If the bet holds, the append-only log stops being DeepSeek's signature and becomes the industry baseline, and this post's cache arithmetic stops being a curiosity and becomes the reason every harness documents its history discipline. If it fails, the repository's discipline was an outlier, not a direction — and the difference will be visible in the same place the videos looked: the bill.

What to steal regardless. The session log as single source of truth with a logged-surface invariant; the seam discipline (definition / provider / consumer, one role is not a capability); the tool pipeline as a waterfall of policy with monotonic guards for final denial; the docs-as-machine-checked-artifact practice with freshness-gated generated catalogs; the defensive-patterns page as a bug-class ledger; the honest scope declarations (file effects only, bash-equivalent trust, per-call policy). The teardowns add two more items to the list. First, the cache trio: keep the clock and anything else volatile out of the prompt prefix, sort tool descriptions in a fixed order, and run one test that asserts cache hits — the bill is the assertion. Second, the rejected-folder discipline: a decision recorded without what it beat invites re-litigation, so make "alternatives considered" a gate, not a habit. Any harness team can adopt any one of these without adopting Cordis.

The project's most distinctive move — and the reason this blog's earlier spatiotemporal-composability coverage connects to it directly — is that it makes the plugin runtime itself the product surface: dump-config prints your machine's tree, any row is replaceable by a patch, and the agent that runs inside the tree can inspect, mount, and unmount its own components. Four days after the calculus appeared on arXiv-style GitHub, it was the shipping substrate of a ~150,000-star harness. The theory mattered; the harness is what the community voted for — and the community's own teardowns of it are already teaching the rest of the industry which parts of the architecture are portable and which are the product.

References

  • DeepSeek Harness documentation (English) — https://deepseek-harness.github.io/deepseek-harness/en/
  • DeepSeek Harness repository — https://github.com/deepseek-ai/deepseek-harness
  • README.md — "Everything is a Plugin", powered by Cordis
  • docs/architecture.md — profiles, bundles, the turn flow, capability seams
  • docs/cordis-primer.md — Cordis in five ideas; dispatch modes; waterfall semantics
  • docs/agent-lifecycle.md — turn and step sequence diagram
  • docs/tool-execution-pipeline.md — the guarded pipeline
  • docs/defensive-patterns.md — the bug-class ledger
  • docs/AGENTS.md and AGENTS.md — documentation tiers, standing orders
  • Subsystem pages: session, scope, tools, shell, sandbox, approval, permission-presets, compaction, skills, subagent, workflow, commands, goal, schedule, plan, invariants, session-telemetry
  • Agent Notes: microkernel event taxonomy (2026-06-11), capability seams (2026-06-13), code mode (2026-06-15), reconstructable requests (2026-07-05), agent-scope runtime design (2026-07-12), canonical tool output (2026-07-20), compaction prefix-cache reuse (2026-07-21), per-session agent presets (2026-08-03), minimal preset owns the RL composition (2026-08-10)
  • .agents/notes/implemented/feature/2026-07-08-self-referential-cordis-toolset.md — the cordis_* toolset design
  • docs/user/guide/python-sdk.mddeepseek-harness-sdk, bundled runtime
  • A Programming Paradigm for Spatiotemporal Composability (Shi, Zhang, Cui; PKU + DeepSeek-AI) — https://github.com/cordiverse/paper
  • This blog's coverage of that paper — Spatiotemporal Composability: The Missing Calculus for Self-Evolving Agents

Other harnesses referenced in this post

Videos

  • Cloud Codes. DeepSeek Harness Architecture: Insane Software Engineering Behind It (2026-08-15, 21:27) — the append-only golden rule, the ~120x prefix-cache discount, the invariant module that disbelieves the loop, the compaction-prompt fix, the four presets, and the 683 agent notes.
  • Aaron — AI-native builder. What I Learned From DeepSeek's Harness (2026-08-16, 10:06) — the eight-harness benchmark (47%–67% success, 7x cost), the 44/3 event log, the one-row agent loop, and the agent-built process behind 12,293 commits.

Spatiotemporal Composability: The Missing Calculus for Self-Evolving Agents

A Programming Paradigm for Spatiotemporal Composability (Peking University / DeepSeek-AI) is organized around five formal contributions plus a framing claim. This post lays out the key insights the way the authors present them — the two dimensions of dynamic composition, revertible effects, reactive coeffects, the unified context as a programming paradigm, and the calculus that composes local guarantees into system-wide ones — then reflects on each in turn: what the insights really buy, where the guarantees stop, and why the 'one door' context and the observational-equivalence honesty are the two things agent engineers should internalize first.

agentic-designdynamic-compositionspatiotemporal-composabilityeffectscoeffectsagent-harnessesself-modifying-agentsplugin-systemsformal-methodscapability-securitysandboxingkoishicordisdeepseek

On August 13, 2026, an 88-page paper appeared on GitHub: A Programming Paradigm for Spatiotemporal Composability, by Yifan Shi and Wei Zhang of Peking University and Tianyi Cui of DeepSeek-AI, published from the cordiverse repository. Within two days it had over 1,500 GitHub stars — an unusual reaction for a paper whose tools are monads, coeffects, and operational semantics.

The paper answers a question every engineer has felt but nobody has formalized: how do you safely remove a piece of software from a running program? Every program that supports plugins has a shameful button — in VSCode it is "Developer: Reload Window," a command that restarts the entire extension host so you can remove one extension. The button exists because the alternative is too hard: there is no safe way to unload a single extension's code, undo what it did to the program, and keep everything else running.

This post does two things. First, it lays out the paper's key insights the way the authors present them — their own contributions, quoted and translated into plain language. Then it reflects on each insight in turn: what it really buys, where its guarantee stops, and what it means for the problem this blog keeps coming back to — agents that modify their own harnesses.

The problem the authors start from

The paper's motivation is a gap, not a bug report. Composition — assembling complex systems from simpler parts — is the most-studied topic in software engineering, but almost entirely in its static form: function calls, module imports, class inheritance, resolved at compile time and fixed for the life of the process. Dynamic composition — components loaded, unloaded, and reconfigured at runtime — has "theoretical foundations... underdeveloped, compared to the rich formal frameworks available for static composition."

The evidence that this is a real gap, not an academic one, is measured: among the top 100 VSCode extensions by install count, 87 contain executable code and require a host restart to remove; only 7 declare dependencies on other extensions at all. The industry's answer to the gap is the coarse-grained workaround: restart the process to get rid of a bad module, let the container orchestrator manage service dependencies. The workaround's costs are real and structural — every restart discards process-local state (caches, connections, partial computations) and rebuilding takes seconds to minutes; orchestration cannot express dependencies between components that share an address space, and turns local calls into network calls.

The authors' framing claim: the field has a rich theory of static composition and none of dynamic composition — and the industry papers over the hole with restarts.

The authors' key insights

The paper's own structure makes its claims explicit. One framing insight (the two dimensions) and five formal contributions, each building on the last.

Insight 1 — Dynamic composition needs two dimensions that static theory never had to face

"To characterize the requirements of dynamic composition, we identify two orthogonal dimensions beyond the well-studied algebraic aspects of composition." The first is temporal composability: "upon removal of a component, the modifications the component made to the shared environment must be completely and safely reversed." The second is spatial composability: "components must be able to declare, discover, and resolve their dependencies on one another in a structured and verifiable manner."

The authors are careful to anchor both in familiar static settings: temporal composability reduces to lexical scoping — RAII, bracket patterns — and spatial composability reduces to module import resolution. The move is not to invent new requirements, but to notice that the old guarantees stop holding the moment components arrive and depart at runtime. A plugin's effects can outlive any function call, so the lexical scope that once ran destructors automatically is gone; a dependency can vanish mid-execution, so the one-time wiring the linker performed is no longer enough.

Two guarantees for pluggable software — temporal: plug in, every action gets an undo; plug out, the undoes play in reverse, state returns, no restart, others untouched. Spatial: a component declares what it needs; when a provider changes, dependents reconnect or wait — a missing provider means "wait", never "crash". In static code these are ordinary (RAII, lexical scoping, module imports); at runtime they are the whole problem.

Insight 2 — Effects can be made revertible (contribution 1)

The authors' first contribution: "We formalize revertible effects: every context transformation carries an explicit inverse that the runtime tracks, and both tracking and recovery preserve composition, so the context is recovered upon component removal."

In plain language: effects are what a component does to its world — allocating memory, registering a callback, writing to a store. The insight is that each such action should carry its own undo, and the runtime should log which component performed which effects, in what order. On removal, the runtime plays the inverses in reverse order — last effect undone first, like unwinding a stack. Because the inverse of a composite effect is the composition of the inverses, cleanup is derived, never written by hand. The author of a component supplies one line per atomic operation — the operation and its inverse — and the runtime composes them into correct, ordered teardown. The consequence the authors draw: this "establishes local temporal composability."

Insight 3 — Coeffects can be made reactive (contribution 2)

The second contribution: "We formalize reactive coeffects: a component declares the coeffects it requires as a specification, and each change of the context notifies the component against that specification as activating, deactivating, or neutral."

In plain language: coeffects are what a component needs from its world — a database, a filesystem, an adapter. A component declares "I require a storage backend," and the runtime does the rest. Every change to the world is classified against each component's specification: activating (your need was just satisfied — start), deactivating (your provider was removed — wind down), or neutral (nothing you declared changed — don't react). A component whose dependency is missing stays inactive; it does not error. When the dependency appears, it activates. This "establishes local spatial composability."

Insight 4 — Effects and coeffects are one context, and that is a paradigm (contribution 3)

The third contribution is the paper's boldest claim: "We unify the effect context and the coeffect context into a single context type, in which an observational equivalence on the coeffects supplies the effects with independence, constituting a programming paradigm for spatiotemporal composability."

The unification is not cosmetic. Because the context type is recursive and its coeffect part unconstrained, "any state the system needs to share across components can be encoded as a dependency with an appropriate value type — Σ subsumes all shared mutable states, not just inter-component dependencies." Every interaction between a component and its environment passes through a single object carrying three things: the current state, the undo log, and the dependency map. The paper's metaphor is literal: "loading a component corresponds to executing its effects (plugging in); unloading a component corresponds to recovering its effects (unplugging, without affecting other running components)," with hierarchical contexts enabling arbitrarily nested composition.

One context — the only door: every interaction between a component and the world passes through a single object holding the current state, the undo log, and the dependency map. Components plug in (load: effects tracked) and plug out (unload: effects undone). One door means nothing can leak — and nothing can be missed.

The paradigm claim is situated between the two poles of side-effect handling: functional programming threads state explicitly through every call (traceable, equational, and boilerplate-heavy), while imperative programming hides it (ergonomic, and effectively untraceable — React's useEffect registers a persistent side effect that appears in no parameter; Spring's getBean pulls from a process-wide registry with casts at every call site). The context paradigm, the authors write, "combines the traceability of the functional approach with the ergonomics of the imperative approach," and the payoff is their most important sentence: "In both directions, correctness that would otherwise rest on developer discipline becomes a structural property of the paradigm."

Insight 5 — Local guarantees can be proven to compose (contribution 4)

The fourth contribution upgrades the paradigm from a feature to a foundation: "We give a calculus of dynamic composition, which combines the two mechanisms into the notion of a component and equips its lifecycle with an operational semantics. Its metatheory carries spatiotemporal composability from a single component to a whole system of interleaved components."

The calculus models components through a full lifecycle — loading, iteration, withdrawal, asynchrony, failure — and the metatheory proves the properties that matter: preservation, temporal composability, spatial composability, progress, and confluence. The point is compositional in the strongest sense: if every part can be removed cleanly and reconnects correctly, then a whole system of interleaved parts can. "Each part is safe to swap" is a local statement; "the system stays consistent through a storm of swaps" is the global one, and the metatheory is what connects them.

One honesty note sits inside this contribution, and it matters: recovery is guaranteed up to observational equivalence, not literal equality. The authors are explicit that unplugging a component does not restore the heap's exact layout — free releases a block without restoring the arrangement malloc left. What is guaranteed is that "no observer can distinguish" the recovered state from the original, where the observers are defined by the declared dependencies. What you expose as dependencies is what recovery promises to restore.

Insight 6 — It is a shipped system, not a proposal (contribution 5)

The fifth contribution: "We implement these ideas in Cordis, a meta-framework of spatiotemporal composability that provides a core library realizing the formal model with effect tracking and coeffect resolution, as well as a declarative component loader with configuration reconciliation and hot module replacement."

And the implementation is not a demo. Koishi, the chatbot framework built on Cordis, has accumulated over 4,000 community-contributed plugins over four years. The case study makes three claims: temporal composability "without cognitive overhead" (even an inexperienced plugin author gets ordered cleanup without writing an uninstall path); spatial composability across an open ecosystem (switching the storage backend reactivates only the dependents whose resolved dependency changed; a plugin whose dependency is unavailable stays inactive until it appears); and expressiveness plus generality (Koishi's web console is a second, independent Cordis application, the same model in a wholly different runtime). The paper concedes the threats to validity — a single ecosystem, a single host language, observational rather than controlled evidence — but the existence result is the point: the theory was extracted from a working system, not written first and applied later.

Reflecting on the insights

The authors' contributions are the skeleton. What follows is the reflection — where each insight is genuinely important, and where its guarantee stops.

On naming: the first deliverable is the two dimensions

The most undervalued thing the paper does is name the problem. "Reload Window" existed for decades as an accepted tax — nobody called it a symptom of missing theory, because nobody had a vocabulary for what was missing. Naming temporal and spatial composability, and grounding both in their static analogues (RAII, module imports), converts a felt inconvenience into a designable requirement. The restart workaround survived precisely because it was the only available mechanism at the right granularity; once the finer-grained requirements have names, the granularity mismatch stops being an accepted cost and becomes a problem with a theory attached.

This is also the right frame for the agent conversation. Every agent harness that "adds a tool" or "replaces a module" is doing dynamic composition — usually badly, by appending to lists and restarting processes. The two dimensions give agent engineers a checklist: when the agent removes something, is everything it did undone (temporal)? When the world changes, do the things that depended on it reconnect (spatial)? Most harnesses fail both checks, silently.

On revertible effects: structure beats discipline — but only the orchestration is free

The deep move in contribution 1 is the inversion of responsibility: cleanup moves from the author, who must remember and usually gets it wrong, to the structure, which derives it. This is the same move as RAII — the destructor is automatic — extended from lexical scope to runtime lifetime. For autonomous agents this is not a quality-of-life improvement, it is the only realistic option: an agent cannot be instructed into remembering to clean up; a teardown derived from the load is the only teardown an autonomous system will reliably perform. Guarantees you must remember are guarantees an agent will, at some point, not.

But it is worth being precise about what the paradigm makes free and what it does not. The author still writes the inverse of each atomic operation. The paradigm eliminates the composition burden — the ordering, the bookkeeping, the edge cases of interleaved effects — not the semantics burden. If an effect's inverse is wrong, the recovery is wrong, and no amount of structure detects it. This is a quiet but important boundary: revertible effects guarantee that the teardown you supplied is executed, completely and in order; they do not guarantee the teardown you supplied is correct. The structural guarantee is about mechanics, not meaning.

On reactive coeffects: availability becomes a lifecycle state, not an exception

The insight hidden inside contribution 2 is the failure-model change. "A missing dependency means inactive, not error" replaces an exception with a state. Most plugin systems discovered this organically through events and hooks; the paper's contribution is to make availability a first-class, typed, composable property of a component's lifecycle, with a notification language (activating / deactivating / neutral) that the runtime — not the component — drives.

For agents, this is the difference between a harness where a vanished tool crashes the agent, and one where the tool's dependents deactivate and reactivate as tools come and go. An agent that picks tools dynamically is going to experience provider churn constantly; a lifecycle that treats "your dependency is not here" as a normal state rather than a failure is a precondition for agents that gracefully lose and regain capabilities. It also quietly answers a question the blog has asked before about always-on agents: state that survives its providers.

On the one door: auditability, policy — and trust

Contribution 3's unification is the most consequential and the least flashy. If every interaction passes through one context, then the context is simultaneously the undo log, the dependency resolver, and — because declared dependencies are known before a component runs — the access-control chokepoint. The paper draws the security conclusion itself: the dependency declaration "acts as a capability request, and the context proxy acts as a capability mediator," with the complete capability set known statically, before execution. For agent harnesses, whose components are not merely untrusted but machine-written, access control that is structural rather than inspected is the only kind that scales — this is the same conclusion this blog reached about sandboxing (Sandboxing AI Agents, Zero Overhead Is Zero Attack Surface).

But "one door" cuts both ways, and the reflection should say so. A single context is a single point of trust: the most security-critical object in the system is now one object, and the paper's own Section 6.3 concedes the limit — capability mediation is access control over dependencies, not isolation of code. "The second requires an external sandbox," the authors write, and that admission matters: the paradigm makes every effect attributable and every dependency mediated, but arbitrary machine-generated code still needs a real sandbox around the whole room. The context is the door; the door is not the walls.

On observational equivalence and the system boundary: composability ends where the world begins

The most important thing to internalize is the least discussed: recovery means indistinguishability, and the boundary of what can be made indistinguishable is a design decision. The paper develops this as the system boundary: a location lies "inside" when the system can modify it exclusively and restore it; everything else lies "outside," and a coeffect moves the boundary by reifying an external location — confining access to operations that carry inverses, so that a file or a connection can become trackable.

For agents, this is where the theory meets the hard wall. Dynamic composition can make an agent's harness self-consistent — its in-memory registrations, its connections, its caches can all be undone. It cannot undo the world: an email sent, a payment moved, a file written outside the boundary. The guarantee is exactly as wide as the reification. The system boundary is where composability ends and accountability begins — and every agent engineer should read Section 6.1 as the map of that line. "What you expose as dependencies defines the boundary of what recovery can promise" is the paper's deepest design pressure, and it applies with double force when the code doing the exposing is generated by an agent.

On compositionality and sufficiency: the mechanics of change, not the intelligence of change

The metatheory (contribution 4) is what upgrades the paradigm from "nice plugin framework" to "foundation," and it is the property self-modifying systems need most and get least: most agent harness engineering is empirical iteration, not compositional guarantee. But the reflection must end with what the paper does not claim. Spatiotemporal composability makes change safe; it does not make change right. An agent still needs to know what to build, whether it built it correctly, and whether removing it was the right call — the intelligence of change, which is a verification problem, not a composition problem. The blog's own line of argument applies (Verifiers Are King, The Verification Horizon): composition gives an agent the ability to modify its harness without breaking it; verification gives it the ability to modify its harness correctly. A self-evolving agent needs both, and the paper has built — with proofs — one of the two halves. That is why it matters: it is the first half of the foundation, finished, proven, and shipped in production, with 4,000 plugins as the evidence.

And the other half is now a well-posed problem, which is the highest compliment a theory can pay: the paper does not close the question of self-evolving agents, it makes the remaining question answerable.

Bottom line

A Programming Paradigm for Spatiotemporal Composability is organized around the authors' five contributions — revertible effects, reactive coeffects, the unified context as a paradigm, a calculus whose metatheory composes local guarantees into system-wide ones, and a shipped implementation validated by 4,000 production plugins — built on the framing insight that dynamic composition has two orthogonal dimensions, temporal and spatial.

The reflections that matter for agentic design: the two dimensions are the checklist; structure beats discipline but only the orchestration is free; availability-as-state is the failure model agents need; the one-door context is the policy chokepoint but also the trust boundary, and it is not a code sandbox; recovery is an observational promise whose boundary the system must design; and composition makes change safe while verification makes change right. The paper is the mechanics of self-modification, proven and shipped — and the intelligence of self-modification is now a question with a theory behind it, instead of a button marked "Reload Window."


References:

Qatar's Digital Incubation Center: What AI Startups Should Know

DIC is Qatar's government-run tech incubator: 0% equity, free office space with cloud hosting, mentorship, and investor matchmaking, with AI named as a focus area. Here is what AI startups should actually know about the programs, the numbers (and what they don't say), the money, and the market math.

qataraistartupsincubationmcitfundinggulfstartup-ecosystemarabic-aiqdb

dic.mcit.gov.qa is the home of Qatar's Digital Incubation Center (DIC) — the startup incubator run by the country's Ministry of Communications and Information Technology (MCIT). Its pitch, as of the 2025 snapshot of its own homepage, is short: "0% Equity. 100% Enablement."

That claim is the most important fact about the program. DIC does not take equity, it does not charge for its services, and AI is one of its named focus areas. That makes it a genuinely unusual on-ramp for AI startups: a government-funded, free, no-dilution incubation program in a Gulf state that is spending seriously on becoming an AI hub.

What the DIC is

The Digital Incubation Center is Qatar's national incubator for technology startups. It has been operating since the mid-2010s — its first intake of young entrepreneurs was announced for 2016 — originally under the Ministry of Transport and Communications, which later became MCIT. Its own words, from the 2023 snapshot of the site:

The Digital Incubation Center (DIC) was created to boost ICT innovation in Qatar, particularly among young people at the critical early stages of starting or growing a technology-related business. ... We offer startups free office space, technical support, training and guidance, mentors who can help new businesses avoid the typical start-up pitfalls...

It is funded by the state, takes no equity, and receives no payment for its services — all confirmed in the incubator's own FAQ. The homepage counters, identical in the March 2023 and May 2025 snapshots, tell the scale story:

Metric Value
Startups incubated 160
Jobs created 614
Successfully graduated 76
Applications received 1,650+
Current startups 60
Total investment (counter) 205.4M
Average investment per startup per year 3.614M

Read the counters as directional marketing, not audited statements — and do the arithmetic once before you plan around them. "Total investment" (205.4M) divided by "average per startup per year" (3.614M) is ≈ 57, almost exactly the 60 "current startups": the "total" is a snapshot of roughly the current cohort, not a cumulative figure since 2016. Jobs work out to ≈ 4 per startup, and 76 graduates of 160 incubated is a 47% graduation rate — real, but small-team, incubation-scale. The currency is almost certainly the Qatari Riyal (pegged at 3.64 to the dollar), which puts the headline 2019 matchmaking event at QAR 22.6M ≈ US$6M.

The program funnel

The DIC's own materials describe a funnel from idea to funded company: four core tracks plus two investment-readiness programs.

IdeaCamp — the entry point: a three-week bootcamp where entrepreneurs, developers, and designers turn tech ideas into validated business plans, ending in a final pitch. The named technology areas are exactly three: IoT, AI, and Advanced Analytics. Historic outcomes: 27 of 31 competing ideas were incubated in the January 2018 cohort; 25 startups came out of the second edition in 2019.

Direct Incubation (Startup Track) — two years for early-stage startups with a market-ready product or prototype: year one takes a startup from prototype to registered business in Qatar; year two is the Growth Track.

Growth Track — one year for businesses that have already launched, "positioning startups for accelerated growth."

Coworking Space — free hot desks, internet, and facilities for entrepreneurs and small businesses.

Make the Deal (MTD) — the investor-startup matchmaking event held on the sidelines of QITCOM, Qatar's annual ICT conference. Its own page names the target technologies explicitly, with "artificial intelligence" first among IoT, smart home, augmented reality, cybersecurity, and big data. In the 2019 edition, 228 startups pitched, 15 walked away with deals, and QAR 22.6 million (≈ US$6M) was committed by 110 investors. The top single deal was US$2M.

Angel Investor Bootcamp and Startup Investment Readiness Program — the two newest tracks, visible in the site's navigation from 2024 onward: the DIC increasingly positions itself on the funding side of the funnel, teaching founders how to take investment rather than just how to build products.

The funnel is explicit: IdeaCamp turns ideas into startups. Startup Track turns startups into registered companies. Growth Track scales them. Make the Deal connects them to money. You can enter at the stage you are at.

The DIC funnel — IdeaCamp (3-week bootcamp, ideas → business plans), Startup Track (2 years, prototype → registered Qatari company), Growth Track (1 year, launched companies), Make the Deal + investment readiness (QITCOM matchmaking, QDB financing). Enter at the stage you are at.

What an AI startup actually gets

From the DIC Services page, incubation provides training, dedicated mentorship (each company gets at least one mentor, sometimes two), networking, industry contacts, investor access, internships from Qatari universities — and free office space whose details matter for AI startups: cloud hosting, a 5G WiFi network, and software licenses are listed as included facilities. The relevant parts for an AI startup are the free cloud hosting and compute-adjacent infrastructure, the 5G connectivity (Qatar is an early and aggressive 5G market), and the investor matchmaking — because the DIC itself does not write checks.

AI is a named focus area

AI is written into the program materials, not incidental to them. IdeaCamp's technology areas are exactly IoT, AI, and Advanced Analytics; Make the Deal lists AI first among its emerging-technology focus areas; the Startup Track FAQ's preferred technologies lead with cloud computing, big data, and analytics. The portfolio reflects it — the 2023 and 2025 intake lists include Ellogy AI, Speechzy (AI/ML for speaking skills), Tahado (AI-based cervical cancer diagnosis from 3D scans), Ceena Lab / BizPlanner (AI-assisted business plan generation), Ferasah (machine-learning prediction of asset failure), and Sensorways (IoT + ML equipment monitoring).

The honest caveat: most of the portfolio is marketplaces, delivery apps, and vertical e-commerce — the classic Gulf SaaS pattern. The genuinely AI-first startups are a minority. You are not joining a DeepSeek or a Mistral; you are joining a state-backed incubator where the realistic winning category is an AI application with a Qatari distribution problem — healthcare, logistics, smart city, education — not a foundation model.

The market you are actually building for

Qatar has roughly 3 million people (World Bank, 2025: 2,972,215), of whom only a few hundred thousand are citizens. No startup in this program will scale on the domestic market alone; the DIC's real value is a zero-cost base inside a well-capitalized state with an open door to regional and international investors. The strategy that fits the market:

  • Arabic-language AI is the open niche. The Gulf's Arabic-capable foundation models come from the neighbors — the UAE's TII Falcon, Saudi Arabia's SDAIA ALLaM — and Qatar ships no frontier Arabic model of its own. The opening is application-layer: Arabic-first products built on someone else's model, with a local distribution problem.
  • The state is the anchor customer. TASMU Smart Qatar is a national program explicitly seeking smart solutions in healthcare, transport, and environment, and the government-services layer is where Digital Agenda 2030 money flows.
  • The talent pool is thin, so plan to import. QCRI/HBKU graduates are the local edge; most engineering hires will come from abroad — which the DIC's residency rule (no citizenship, no Qatari partner required) accommodates.
  • Use Doha as the beachhead, not the destination. Web Summit Qatar has run in Doha since February 2024, and QITCOM carries the investor matchmaking; both are exposure channels into the wider GCC. The winning pattern: pilot with a Qatari government or enterprise customer, prove it in a small rich market, then export across the Gulf.

The money question

Three facts define the economics:

  1. Zero equity, zero fees. FAQ question 20, verbatim: "No, DIC is a government-funded program. We do not take equity or receive any payment in return for our services." This is the whole point of the "0% Equity. 100% Enablement." rebrand.
  2. DIC does not directly fund startups. FAQ question 5: "potential funding is available through the Qatar Development Bank and other financial institutions." The DIC is a connector, not a fund; the money comes through QDB and the investor network built around events like Make the Deal.
  3. The numbers are real but modest. QAR 22.6M (≈ US$6M) across 15 startups in the best-documented Make the Deal edition (2019); QITCOM-adjacent awards pay in the QR 25,000–125,000 range. This is early-stage, incubation-scale money — enough to build and pilot, not enough to fund a frontier-lab compute bill.

Who can apply

  • Residency in Qatar is the requirement — "any innovative entrepreneur with residency in Qatar." No Qatari citizenship and no Qatari partner needed.
  • Idea-stage is fine. No product required to enter the pipeline; that is what IdeaCamp is for.
  • Existing startups are fine. Companies with a product already launched enter the Startup Track at the appropriate stage.
  • Applications are accepted year-round, with the Startup Track process taking up to three months.
  • Evaluation criteria (FAQ question 10): original idea in a focus area, market opportunity, target customers and validation, initial cost structure and financial plan, team profile.

The application form asks the one question that matters: "If you require funding, please estimate the amount, and how you would use the funding."

The Qatar AI context

The DIC sits inside a state that has spent the last half-decade positioning itself as the Gulf's neutral, well-capitalized AI hub: the 2019 National AI Strategy made AI an explicit national priority; Digital Agenda 2030 is MCIT's umbrella for the digital transformation, with the DIC as the startup-facing layer; Web Summit Qatar has run in Doha since February 2024, bringing international investors to the market. Around the DIC sit the Qatar Development Bank (funding), QBIC (explicitly named as a collaborator in the DIC's FAQ), Bedaya, Qatar University, and the research layer of QCRI at HBKU with its Qatar Center for AI (QCAI).

The strategic logic of the zero-equity model: the state wants technology companies formed and staying in Qatar, so it underwrites the incubation cost rather than taking a stake. For a founder, that is the best possible terms sheet — provided you actually want to build for the Qatari market.

Bottom line

The DIC is a real, well-documented, zero-equity incubator that names AI among its focus areas, offers free office space with cloud hosting and 5G, and funnels startups toward Qatari money through the Qatar Development Bank and its own investor matchmaking. For an AI startup willing to build for the Qatari market — healthcare, logistics, education, government services, Arabic-language applications — the terms are genuinely good.

Keep the frame honest: this is a local-market incubator with an AI label and excellent terms, not a research lab. The winning move is an AI application with a Qatari distribution problem to solve — Arabic-language and vertical applications, not foundation models and not consumer scale.


References:

Zuill's Mob Programming, Remastered

Mob programming says the whole team works on the same thing, at the same time, in the same space, on the same computer — one driver, many navigators, and a rule that an idea only enters the code through someone else's hands. This post tells its story: from XP's pair programming (1999) and the Paris coding dojo (2005), through the Hunter Industries meeting room where the mob got its name (2011), the Agile2014 report, the 'ensemble programming' rename, and the remote mob — to the thing that looks like a brand-new invention: multiplayer AI coding agents, whose shared live sessions a whole team can enter and steer. A remaster does not rewrite the song; it re-engineers the master tape. Multiplayer AI is mob programming, remastered — the same practice, with the driver replaced by an agent. The objection that kept mob programming out of the enterprise — five developers on one computer is wasteful — has died, because the driver is now a machine. And the single-player era answered the old question on its own terms: LeadDev's analysis of 25,264 agent-generated PRs found 79% were reviewed by the same developer who prompted the agent. This post maps the practice onto the product, explains why the mob is back now, connects the practice to its intellectual ancestor — Marvin Minsky's 1986 Society of Mind, in which intelligence emerges from mindless agents that check each other — and lays out how to run one.

mob-programmingensemble-programmingcoding-dojopair-programmingmultiplayercoding-agentsdriver-navigatorstrong-style-pairingcollective-code-ownershipself-review-problemagent-teamsremote-mob-programmingagilesoftware-craftverificationsociety-of-mindminsky

In 2011, a software team at Hunter Industries in Southern California needed to restart a project that had been on hold for months. A few members and a contractor had worked on it; most of the team had not. They gathered in a meeting room to review the code and decide how to take the work on — and somewhere in that review, out of a habit of pair programming and TDD practice sessions, they started passing the keyboard around. After several hours another group needed the room, so they moved, held a mini-retrospective, and agreed the day had been remarkably productive. They booked the meeting room again. Two weeks later they were still working that way, and they had a name for it: mob programming. Woody Zuill presented it to the Agile Alliance in 2014.

Fifteen years later, teams gather again — not around a projector and a single keyboard, but inside a shared live agent session: the same files, the same terminal, the same browser preview, open to everyone at once, anyone able to steer. The products call it multiplayer AI. The name is new. The shape is not.

Multiplayer AI for coding agents is not a new way of working. It is mob programming, remastered — with the driver replaced.

This sounds like marketing fluff until you take the architecture seriously. Mob programming's entire structure — one shared surface, a driver and navigators, the strong-style rule, single-piece flow, continuous code review — maps one-for-one onto the shared agent session. And the objection that dogged mob programming for fifteen years — "how can five developers be productive on one computer?" — has reappeared, word for word, as the objection to shared sessions: "why not just run more agents in parallel?" The data answers both the same way, because the bottleneck was never typing. It was review.

The mob's lineage

The 2011 meeting room was not a beginning. It was the latest turn in an evolution that had been running for more than a decade — the story of an idea that kept getting re-housed: many minds on one surface, and only one pair of hands on the code at a time, so that everyone else had to think out loud. Follow that constraint, and the whole history unfolds in acts.

1999 — the pair. Kent Beck's Extreme Programming Explained made "two programmers, one computer" the centerpiece of a new way of building software: one person's hands on the keyboard, the other's eyes and voice on the code — the split mob programming would later formalize as Driver and Navigator, carrying the claim, radical at the time, that two minds on one task beat two minds on two tasks. XP gave the idea its first vehicle. The pair was the smallest possible mob.

2005 — the dojo. Six years later, in Paris, Laurent Bossavit and Emmanuel Gaillard founded the Coding Dojo, borrowing the language of martial arts on purpose: the dojo is the training hall, the kata is the practice form, randori is free-form sparring. Groups met to practice on a single computer with a projector, passing the keyboard every few minutes the way sparring partners trade roles. The dojo's discovery was that the constraint itself teaches: with one keyboard and many people, no idea enters the code until it has been spoken. It was practice, not production. But the seed was planted.

2011 — the mob gets its name. Zuill's team at Hunter Industries had been running TDD and pair-programming practice sessions as dojos, and when they gathered to restart a project that had been on hold for months, the habit took over — "we started passing the keyboard around," Zuill later wrote. What began as a review meeting became two weeks of meetings, became a way of working. "All the brilliant people working on the same thing, at the same time, in the same space, and on the same computer" — that was how the team described what they had stumbled into, and someone proposed the most memorable name anyone could offer: mob programming. It stuck, which was the practice's first taste of its own marketing problem; the word "mob" would spend the next decade being fought.

2014 — the canon. Zuill's Agile2014 experience report gave the practice its founding text: the Driver/Navigators pattern adapted from Llewellyn Falco's strong-style pairing — "for an idea to go from your head into the computer it MUST go through someone else's hands" — the list of problems that simply "fade away," and the claim that the team delivered roughly ten times as many projects the year after adopting the mob. Then came the tour: Zuill and Falco crisscrossed conferences and code retreats, mobbing live on stage. It was the practice's peak of visibility, and it was still a niche — the name was winning and losing at the same time.

2018 — the rename. Maaret Pyhäjärvi proposed "ensemble programming." Mob carried baggage — crowds, violence, mob rule — and ensemble named what the practice actually was: the whole team, all its skills, one surface, playing together. The rename is the tell that the practice had matured into a discipline with a canon and a community — and an identity problem it could not quite solve. Ensemble stuck in the corners where the practice survived: agile meetups, code retreats, the teams that refused to give it up.

2020 — the remote mob. Simon Harrer, Jochen Christ, and Martin Huber systematized the distributed mob: cameras always on, a ten-minute typist rotation, git handoff at every interval, decisions made by the group. They claimed the distributed version was "superior to anything we ever tried before." But the discipline tax was heavy — cameras on, voices metered, a handoff every ten minutes — and most teams paid it for a sprint, not for years.

The plateau. By the late 2010s, mob programming was a conference-circuit staple that never became mainstream. Not because it failed on its own terms — the teams that mobbed kept mobbing — but because its benefits were structural and its costs were visible, and management's arithmetic counted only the costs. Five minds on one keyboard looked like waste from the outside, and the practice remained what it had been since the dojo: a discipline of believers.

2026 — the remaster. And then the constraint that mob programming had enforced socially became the architecture of a tool. The shared live agent session is one surface, many navigators, a single driver — with the driver now a model that types at machine speed. The line of evolution completes itself. Every step kept the surface singular and made the driver more mechanical: a pair of people, a passing keyboard, a rotating typist, a remote screen-sharer — and finally an agent.

The mob's lineage — from XP pair programming (1999) and the Paris coding dojo (2005) through Hunter Industries (2011), the Agile2014 report, the ensemble rename (2018) and the remote mob (2020), to the multiplayer agent session (2026). The driver evolves from a person who types to the agent, permanently.

A remaster does not rewrite the song. It takes the original master tape and re-engineers it with modern tools — same performance, cleaner delivery, new instruments. The industry calls multiplayer AI a new way of working, which is like calling a remaster a new song. The master tape is mob programming, and the master tape was always the point.

What mob programming actually is

The Agile Alliance's glossary defines mob programming as "a software development approach where the whole team works on the same thing, at the same time, in the same space, and on the same computer." One work item at a time. Continuous code review, whole team involvement, self-organizing teams. Almost all work happens in "working meetings" — defining stories, designing, testing, even working with the customer.

The operational core is the Driver/Navigators pattern. Two roles. The Driver sits at the keyboard and types — "a much more mechanical job" — and must trust the Navigators. The Navigators discuss the idea and guide the Driver "in a slow, metered approach," speaking "at the highest level of abstraction that the Driver is able to digest" at that moment. A timer rotates the driver every ten to fifteen minutes. The result, Zuill writes, is "a sort of collective intelligence of the Navigators."

The benefits are best stated negatively. Mobbers don't claim the mob adds productivity; they claim problems "fade away": communication problems (waiting for answers, back-and-forth email, misunderstanding via documentation); decision-making dysfunctions (reluctance to decide, the need to defend decisions); the waste of doing more than barely sufficient; technical debt, because "many sets of eyes" sit on the current work as it is written; thrashing, because single-piece flow means no one is switching contexts; workplace politics; and "management by meetings," with its separation of decision-making from knowledge creation.

And then there is the question everyone asks. Zuill quotes it directly: "How can 5 or 6 developers be productive while working this way? Wouldn't it be more productive to have them working on different things?" His answer, offered with appropriate caveats: the year after the team adopted mob programming, it delivered roughly ten times the number of projects it had delivered the year before. The point was never that five people on one keyboard is fast at typing. It was that typing was a rounding error against the cost of misunderstanding, and the mob spent its effort where the money was.

What multiplayer AI actually is

By 2026 the single-player agent is the incumbent. Claude Code, Codex, Cursor: each is a private channel between one developer and one model — one person, one terminal, one context window, one transcript. Work exits the channel as a package: a diff, a pull request. The agents are everywhere — installed in 75 percent of Linear's enterprise workspaces, per The Register — which is exactly why the layer around them is now the bottleneck. The industry's first answer to the coordination problem was to run more of them: parallel runners, worktree managers, cloud agents, fleets. Coshell's verdict on that strategy is the sharpest one-sentence critique in the space: "That is more parallelism, not more collaboration. It is still single-player, just with more players who cannot see each other."

The 2026 multiplayer wave is a different bet: make the session itself the shared place. AQ — the "multiplayer coding harness" — defines the category as "AI coding agents whose working sessions are shared, live places that a whole team can enter: everyone sees the same running agent, the same terminal, the same code, and the same app preview, and anyone can steer." Coshell: "Multiplayer AI coding is when several engineers prompt one shared, live AI agent working in a single cloud workspace"; the analogy the industry keeps reaching for is "closer to a Google Doc than to git." collab, a small open-source Go CLI, does it peer-to-peer: "Two devs. Two agents. One session" — each developer keeps their own agent and their own API key, and the two agents share a conversation log and a ./shared/ directory across the wire.

AQ's test for what counts is the right one: a tool is multiplayer when the running session, not just its output, is shared — the same live session (not a replay, not a read-only transcript); anyone can steer mid-run, "the way anyone can grab a marker at a whiteboard"; handoff without re-setup; roles beyond engineers (a PM or designer can watch the preview and push feedback without touching a terminal); outsiders can look. Screen sharing fails the first test — one person still owns the session. Shared tmux over SSH passes the first two and fails the rest.

Why this exists is a measured fact, not a pitch. A July 2026 LeadDev analysis of 25,264 agent-generated pull requests across 2,361 popular repositories found that in 79 percent of agentic PRs, the same developer both reviewed and modified the agent's contribution — and only about one in eight agentic workflows involved more than one human at all. The loop closed structurally, in three steps: the agent session lives in one person's private transcript; the PR is the only artifact that escapes it; and the prompter becomes the reviewer, reading the diff through the memory of their own prompt. As AQ puts it: "The decisions that determine whether agent work is good live in the session, not in the diff. A review process that cannot see the session is auditing the residue of the work rather than the work."

The self-review problem is not new. It is the pre-mob problem set — one person's cubicle, one person's understanding, one person's review — recreated at scale by the tool that was supposed to give everyone a teammate. Mob programming was invented precisely because that arrangement has a ceiling. Multiplayer AI is the industry rediscovering the mob's answer, in the medium that forced the question.

Why the shared layer exists at all is an industry-structure question, and it is worth being clear-eyed about. The model vendors are structurally disincentivized from building it: Anthropic shipping Agent Teams, and the steady drift toward "Claude Code for teams," point toward collaboration inside one model's session on one vendor's terms. The layer between people — many models, many providers, your own git — has to be neutral to be useful. That is why the multiplayer products are an independent layer (Coshell, built on the open-source OpenCode engine; AQ; the peer-to-peer collab) rather than a vendor feature. The mob needs a room, and the room cannot belong to the driver.

Why the mob is back now

If multiplayer AI is the mob's endpoint, why is the endpoint arriving in 2026 and not in 2018, when mob programming was a conference-circuit staple? Three forces.

The waste objection died. The mob's killer objection — five minds, one keyboard — assumed the driver's time was the scarce resource, and management arithmetic said it was. The agent era inverts the mob's cost structure. The driver is now a model that types at machine speed, never tires, and never gets bored; the scarce resource is what the navigators contribute — the articulation, the judgment, the verification. "One agent, many navigators" is no longer five salaries idle; it is five minds loaded onto a machine that works faster than any of them. The objection that kept mob programming out of the enterprise was an accounting artifact, and the accounting changed.

Remote work killed the room. Mob programming was a room practice, and the remote mob that kept it alive demanded more discipline than most teams had. The shared agent session is the first mob room that is easier to enter than a video call: a link, and the environment is already there, loaded, warm. The remote mob's hardest problems — handoff, shared state, who sees what — are precisely the problems the session solves mechanically.

The single-player agent recreated the disease the mob was invented to cure. This is the one nobody chose. The tool that was supposed to give every developer a tireless teammate instead recreated the cubicle: one person, one private transcript, one self-review, 79 percent of agentic PRs. The industry ran a fifteen-year control experiment on the null hypothesis — everyone in their own session, working on their own thing — and measured the result at scale. The null hypothesis lost. The mob is back because the alternative failed on its own terms.

Same shape, new driver

The same shape, the driver changed — mob programming's one-computer mob (2011) and multiplayer AI's one shared session (2026): many navigators converge on a single shared surface and one work item.

Run the mapping and it holds at every layer.

Same surface. The mob's rule was "all code that enters the code base is input through this single computer." The multiplayer rule is that all code enters through the session. One place, many minds, everyone watching the same thing at the same time.

Same roles. The agent is the driver; the humans are navigators, permanently. Read Zuill's description of the navigator's job and it reads like instructions for prompting an agent: express the idea "in a slow, metered approach," speak "at the highest level of abstraction that the Driver is able to digest," guide without touching the keyboard. In mob programming you could not type because the keyboard belonged to the driver. In a shared agent session you cannot type because there is no keyboard — there is a prompt. The social role has become the interface.

The industry files this under pair programming, and the categories say otherwise. AQ's FAQ: "Pair programming shares a human's editor. Multiplayer coding agents share an agent's working session: the humans present are steering an AI that does the typing." But pair programming is two people, one editor, one conversation. Mob programming is the whole team on one surface, one work item at a time — and the multiplayer test's own criteria, roles beyond engineers and outsiders in the room, are mob criteria, not pair criteria. What AQ described is mob programming's Driver/Navigators pattern with the rotation removed. Call it multiplayer if you like; the architecture is a mob room, not a pair table.

Same constraint. The strong-style rule — an idea must go through someone else's hands — was mob programming's hardest discipline, and it was social: the team enforced it on itself, person by person. In multiplayer AI it is architectural. The only hands on the code belong to the agent. You cannot drift, you cannot grab the keyboard, you cannot produce output you did not first articulate. This is the least noticed and most profound property of agentic coding: mob programming needed a norm to guarantee what the agent tool provides by construction.

Same objection, same answer. "Why not parallel agents?" is "why not have everyone in their own cubicle?" with better branding. It fails at the same point: the work is generated in private and reviewed at the end by whoever generated it. The 79 percent figure is the quantified version of the problem set the mob was built to make fade away.

Same fading problems. Communication: the session is the artifact — "what is it doing?" questions die because everyone watched it happen; there is no back-and-forth email, there is the scrollback. Decision-making: the whole team steers, in the open, with the current information, and the transcript preserves the reasoning. Waste: single-piece flow on one work item. Technical debt: many sets of eyes on every line, live, not at PR time. Thrashing: nobody re-ingests the codebase from cold — Coshell's point that a shared warm context is loaded once for the team rather than re-read per person is the token-economics version of the mob's anti-thrashing argument.

Where the eyes land — the self-review problem: 79% of agent-generated PRs are reviewed and modified by the same developer who prompted, ~1 in 8 agentic workflows involve a second human, while the mob baseline is continuous review by design.

The two figures above are the whole argument in pictures: the mob's topology, and the cost of abandoning it.

What the remaster changes

A remaster keeps the original; it also changes the delivery. Five deltas are genuinely new.

The driver never rotates. In mob programming, the rotation existed so everyone got the driver's seat — the learning and the engagement, not just the typing. In multiplayer AI the driver is always the agent, so the humans are all navigators, all the time. That is a pedagogy change with a measured price, and this blog has covered it: in You Don't Learn What You Delegate, a pre-registered randomized trial found that delegating to an AI cut learning by 17 percent (d ≈ 0.74) — and the damage concentrated exactly where the mob's navigators need skill most: debugging. The mob's driver rotation was how a team learned by doing. A mob with a permanent driver must learn by articulating and reviewing instead. The mob still teaches — it teaches navigation, not typing. But the team that lets the agent drive while the humans spectate has stopped mobbing and started watching.

The mob admits non-human navigators. Zuill's navigators were all people. A shared session's navigators can be agents: a verifier subagent that checks the diff as it lands, a reviewer agent that reads the session's scrollback, an always-on agent that patrols the repo overnight. This is the direction Anthropic's multi-agent research system pointed at in 2025 — subagents with their own context windows, exploring in parallel, compressing their findings back to a lead — with the observation that "groups of agents can accomplish far more." Mob programming at that scale is mob programming where the mob itself has scaled past human attention. And it makes the navigator's job, verification, the expensive and decisive one — the thesis of this blog's Verifiers Are King and The Verification Horizon. In a mob of agents, the driver is cheap; the navigators are the product.

The mob has perfect memory. Face-to-face mob memory was perishable: what the team agreed at the whiteboard at 10 a.m. lived in their heads at 4 p.m., if it survived at all. The shared session is the memory — the transcript, the scrollback, the dead ends, the decisions. AQ's review guidance is blunt about why this matters: "Dead ends are signal: they mark the parts of the codebase the agent found confusing, which is where the surviving code deserves the closest reading." collab's motivation is the same point from the loss side: "By the time the diff lands in a PR, the chain of thought that produced it is gone." The mob's knowledge no longer leaves at 5 p.m.; it stays in the room, reviewable.

The mob never sleeps. The session persists. A teammate joins from another timezone mid-run and finds the work where it was left, environment, history, and running processes intact — the "handoff without re-setup" test. This is the always-on agent economy this blog has written about before (Always-on agents), except the always-on entity is the mob itself, and humans are the visitors.

The one-keyboard rule must be re-imposed in software. This is the deepest point. The physical mob's single-writer guarantee was hardware: one keyboard. The moment agents run in separate sessions over the same repo, the guarantee disappears — two agents can reach for the same lines of the same file, and the engine underneath is last-write-wins. Coshell's overlap control is exactly the mob's rule rebuilt as a protocol: "the second edit is held before it lands, and a coordinator steers that agent to proceed if the changes are compatible, or to take other lines, wait, or let the first land first." The mob's most important property is no longer enforced by furniture; it is a design problem, and it is the entire job of the coordination layer. Multiplayer agents relate to mob programming the way flying machines related to bird flight: the constraint that made the original work — one writer at a time — had to be rediscovered in the new medium before anything could fly.

The Society of Mind

In 1986 — forty years before the shared agent session, twenty-five years before the mob got its name — Marvin Minsky published The Society of Mind, and in doing so described the mob's architecture in advance. Minsky's claim is that intelligence is not one big thing. It is a society: many small, mindless parts, which he called agents, interacting until the whole does what no part can do alone. There is no homunculus, no little person inside the head doing the thinking. "What magical trick makes us intelligent?" he asked. "The trick is that there is no trick. The power of intelligence stems from our vast diversity, not from any single, perfect principle."

Run that thesis against the mob and it reads like a design document for what Zuill's team stumbled into in 2011:

  • The navigators are Minsky's agents. Each holds part of the problem and knows almost nothing about the rest — that is the point. The mob's "sort of collective intelligence of the Navigators" is the society doing what no single member can: no navigator builds the product alone; the product is what the connections produce.
  • The strong-style rule is Minsky's communication problem made social. An idea only enters the code through someone else's hands — through articulation. Minsky spent most of the book on how agents connect, because intelligence is not in the parts; it is in the connections. The one-keyboard rule exists precisely to force the connection.
  • The driver is the anti-homunculus. Minsky's core methodological warning was the homunculus fallacy — explaining the mind by hiding a smaller mind inside it. The mob's driver is explicitly "a much more mechanical job": hands without a mind. In the session era it is genuinely mechanical — a model. Nobody explains the mob's intelligence by pointing at the driver, because there is no intelligence there to point at. It all lives in the society. That is the whole point of the exercise.

The Society of Mind, three decades, three rooms — Minsky's mindless agents and the connections between them (1986); the mob's navigators converging on one pair of hands (2011); the shared session where human navigators and model agents share one room, with a verifier agent checking the output (2026). The intelligence is never in a single part; it is in the connections and the checks.

Then notice the vocabulary. The word Minsky chose for his mindless parts — agent — is the same word the industry uses for the things now sitting in the mob's navigator seats. The lineage runs from Minsky's 1986 agents through the software-agents literature of the 1990s to the model agents of the 2020s, and the session is where the metaphor stopped being a metaphor: the society is now a literal one, human navigators and model agents in one room — subagents with their own context windows, verifier agents, always-on patrols, the Anthropic multi-agent system referenced above. Minsky's society got machines in it. And the discipline that keeps mindless parts from producing a mob in the bad sense is the same in all three rooms: checking. Mindless parts do not make a mind until something verifies them — which is the navigator's job, and the reason the session's scarce resource is verification, not generation (Verifiers Are King, The Verification Horizon).

The man who named the mob has spent the years since 2011 telling this story in person. At DDD Europe in 2024 — forty years after Minsky's book, thirteen years after the mob got its name — Zuill gave Advanced Software Teaming, the full account from the founder: the dojo lineage, the Driver/Navigators pattern, the problems that "fade away," and why the practice outlived its objections. It is the primary source for this post's central claim — that the shared agent session is mob programming, remastered, with the society of mind run as an actual society.

Woody Zuill — Advanced Software Teaming [Mob Programming] (DDD Europe 2024): the founder's full account of mob programming, from the 2011 Hunter Industries room to the practice that met the agent era. Watch at youtube.com/watch?v=gL4WxpiG1D8

Forty years after the theory, the society of mind is no longer hypothetical. It is a room you can join, the driver's seat held by a machine, the mind spread across everyone and everything in the session — checking each other, which is the only way mindless parts ever made anything intelligent.

How to run an agent mob

Mob programming's fifteen years of practice boil down to a small set of disciplines, and each has a direct translation into the session era. The cheat sheet:

Same practice, new room — every mob discipline translates directly into the shared agent session: one session per work item, the agent as driver, the strong-style rule made architectural, session review instead of PR review, scrollback retrospectives, and the whole team in the room.

One session, one work item. Single-piece flow survives unchanged: start, work on, deliver one item before the next. The session is the unit of flow now — when the session wanders, the mob wanders.

Invite the reviewer before the code. In mob programming, review happened because everyone watched every line. In the package model, review happens at PR time and fails (the 79 percent). The translation: the second set of eyes should be in the session, not on the diff. Make "someone other than the prompter ran the result" the definition of done. Where to spend that attention is the mob's instinct made explicit: review effort concentrates where the session shows the agent struggled, not where it sailed through.

Prompt at the highest level the driver can digest — then verify the step. This is Zuill's instruction for navigators, verbatim, now the interface: express the next step at the right abstraction, and do not let the driver run ahead of your ability to check it. Small steps, metered, the way the mob did it.

Rotate the navigator roles, not the driver. The driver no longer needs the rotation; the humans do. Rotate who steers the session, who verifies the agent's work, who reads the scrollback for dead ends, who speaks for the customer. The mob's learning loop — everyone takes the seat — moves from the keyboard to the roles around it.

Hold the retrospective on the transcript. The mob's "turn up the good" ritual has a new artifact to review: the session itself. Where did the agent go wrong, and how did we steer it back? Which prompts were unclear? What did we almost ship that looked right and was not? The transcript is the first mob memory that can be reviewed by people who were not in the room.

Keep it chosen. The mob's first rule transfers intact: it only works if the team chooses it. A session that is imposed is not a mob; it is a panopticon.

The honest costs

Mob programming's glossary entry comes with a pitfalls section, and it transfers almost unchanged. Constant collaboration all day is not for everyone. And above all: "Mob Programming will only work if the team sees value in this approach and chooses to work in this manner... the change will not be nearly effective if people feel they are being forced to make the change." The same is true of shared sessions. The moment a shared session is imposed for surveillance — who steered, who typed the prompt, who was idle — the navigators stop navigating and start performing. The mob's first sign of use is "kindness, consideration, and respect"; a multiplayer tool that tracks instead of hosts will earn the 79 percent problem back, now with dashboards.

And the discipline is not optional. A swarm of parallel agents in one repo without single-piece flow is a mob in the bad sense: uncoordinated, clobbering, review by firehose. Mob programming's norms are precisely what keep a multi-agent session from degenerating into chaos — one work item at a time; anyone can steer; continuous review; frequent retrospectives, because the team "always turn[s] up the good"; participation that is chosen, not coerced. The practice has fifteen years of accumulated thinking about how to make many minds work on one thing. The products are months old. The products will need the practice more than the practice needs the products.

And a new contagion replaces the old one. The glossary warns that a mob working in close physical proximity spreads illness; the session mob has its own epidemiology. Shared context is shared attack surface: a prompt injection in one transcript, or a poisoned artifact in the shared workspace, travels to everyone who joins the session — and unlike a cold, it is engineered, and it is aimed. Sandboxing the room is not optional infrastructure; it is the mob's hygiene (Sandboxing AI Agents, Zero Overhead Is Zero Attack Surface).

The mob was never about the keyboard

Mob programming was dismissed for years as a waste of five minds and one keyboard. The five minds were the point. The keyboard was the constraint that forced them to think out loud — that made articulation, and therefore review, and therefore shared understanding, unavoidable. Every artifact of the mob's discipline — the driver who cannot act on an unspoken idea, the navigators who must speak at the highest level the driver can digest, the many eyes on every line as it is written — is the same constraint doing its work from a different angle.

Multiplayer AI re-imposes that constraint in software — mob programming, remastered: a driver that never tires, navigators who never have to wait their turn, a session that remembers everything, and a mob that does not go home. The industry is calling it multiplayer because the tooling changed. The reason it will matter is that the practice did not change — it was waiting for a driver that never needs to be rotated. In 2026, the driver arrived. The mob is back, and this time it never leaves.


References:

  • Marvin Minsky. The Society of Mind (Simon & Schuster, 1986) — intelligence from the interaction of mindless agents; the homunculus warning; the word "agent."
  • Kent Beck. Extreme Programming Explained: Embrace Change (1999) — pair programming, the seed of the Driver/Navigator split.
  • Laurent Bossavit, Emmanuel Gaillard — the Coding Dojo (Paris, 2005): groups practicing kata on a single computer with a projector, keyboard passed in randori style. The dojo is the direct ancestor of the mob's one-keyboard constraint.
  • Agile Alliance. What is Mob Programming? — glossary entry: definition, expected benefits, common pitfalls, signs of use, origins (Hunter Industries, 2011; Zuill's Agile2014 report). Source of the definition and pitfalls quoted in this post.
  • Woody Zuill. Mob Programming — A Whole Team Approach (Agile Alliance Experience Report, 2014) — the Driver/Navigators pattern, the strong-style rule, fading problems, and the ~10x projects-delivered comparison.
  • Woody Zuill. Advanced Software Teaming [Mob Programming] (DDD Europe, 2024) — the founder's full account of the practice; the primary source for this post.
  • Llewellyn Falco — strong-style pairing: "for an idea to go from your head into the computer it MUST go through someone else's hands," as adapted by Zuill.
  • Maaret Pyhäjärvi — Ensemble Programming Practiced (Pragmatic Bookshelf, 2023): the "ensemble programming" rename (2018), emphasizing the whole team, all skills, one surface.
  • Simon Harrer, Jochen Christ, Martin Huber. Remote Mob Programming — the distributed mob: cameras always on, ten-minute typist rotation, git handoff; "superior to anything we ever tried before."
  • AQ. What are multiplayer coding agents? (updated Aug 2026) — the definition, the "package vs place" distinction, the multiplayer test, the July 2026 vendor map, and the Linear/Register adoption figure.
  • AQ. The self-review problem (Jul 2026) — LeadDev's analysis of 25,264 agent-generated PRs; session review as the fix; "auditing the residue of the work."
  • Coshell. Why multiplayer — shared live sessions, overlap control, warm shared context, "more parallelism, not more collaboration," and why the layer between people has to be neutral.
  • collab. Multiplayer for AI coding agents — peer-to-peer shared sessions across Claude Code/Codex/Cursor: "Two devs. Two agents. One session."
  • Anthropic. How we built our multi-agent research system (Jun 2025) — subagents, parallel context windows, "groups of agents can accomplish far more."
  • Related: You Don't Learn What You Delegate — the 17% learning cost of delegation (d ≈ 0.74), concentrated in debugging, when the agent does the typing.
  • Related: Verifiers Are King — in agent work, verification is the bottleneck the mob's navigators must fill.
  • Related: The Verification Horizon — why verifying agent output is harder than generating it.
  • Related: Always-on agents: state, memory, and the governance gap — the session as persistent state.
  • Related: Sandboxing AI Agents — the mob room's hygiene: shared context is shared attack surface.
  • Related: Zero Overhead Is Zero Attack Surface — why the coordination layer should add nothing beyond the shared room.
  • Related: Software dark factories: specs in, software out — the factory the mob now runs.
  • Related: Every Token Has a Price Tag — why the shared warm context is also a shared budget line.

The Factory Is Not Dead

Jostein Hauge's 'The Future of the Factory' asks whether industrialization still works as the route from poverty to prosperity, and answers yes — with qualifications. Four megatrends (the rise of services, digital automation, the globalization of production, ecological breakdown) are reshaping the factory, and Hauge's verdict splits them in two: services and automation change industrialization less than feared; globalization and ecology change it for real, driven by power asymmetries more than technology. This post summarizes the book, then applies its lens to the next factory — the software factory of AI agents — and adds the term's own history: Robert W. Bemer's 'machine-controlled production environment, or software factory' at the 1968 NATO Software Engineering Conference (qualified by his August 1966 planning checklist, so 'coined at NATO' is too strong), sitting in the same report as McIlroy's mass-produced reusable components — factory-controlled production and reusable components, the two poles the dark factory era recombines.

bookindustrializationdevelopment-economicsjostein-haugemegatrendsautomationglobal-value-chainsclimateindustrial-policyai-agents

About 250 years ago, something spectacular happened to the way we make things. We started using machines on a large scale. Mechanization, the division of tasks, the steam engine, the power loom — and the technologies came together in a single physical space where workers operated them in a system of distinct tasks. The factory stood at the center of the Industrial Revolution, and the countries that industrialized surged ahead at a pace never witnessed before. Jostein Hauge, a development economist at Cambridge, opens The Future of the Factory with this history to make a deliberately unfashionable claim:

The main ingredient [in the recipe from poor to rich] has always been the same: industrialization. Countries that have undergone this transformation have strengthened their capabilities in manufacturing and factory-based production. Industrialization has come to be seen as the foundation of technological progress, innovation, international competitiveness, and rapid growth in productivity.

And then the question his book exists to answer: is that still true? "Megatrends" — trends in technology, economy, society, and ecology with global impact — are changing how countries develop, what technological progress means, and whether traditional industrial policy still works. Hauge identifies four that matter, and his verdict on them is the most useful thing in the book: some change industrialization less than people fear, and some change it for real.

Hauge's four megatrends and their verdicts — services and automation as complements, globalization and ecology as constraints.

The rise of services: complement, not replacement

The first megatrend is a paradox: most of today's "industrialized" countries do not have much industry, measured as a share of output. Services now represent more than sixty percent of world GDP. The world's largest and most profitable companies — Amazon, Google, Walmart — hardly manufacture anything, and even the great manufacturing firms get most of their profits from activities classified as services: research and development, industrial design, retail, marketing. Apple is the world's most valuable manufacturing company and owns essentially no factories. Countries in the global South are riding on service-led growth — India, Kenya, the Philippines, Rwanda.

The conventional reading says this overturns the old model: if services can drive growth, why bother with factories? Hauge's reading is the opposite. The rise of services was driven largely by ICT, which made productivity growth achievable in digital services and made services tradable — the cost of trading services has fallen to the level of trading goods. But services do not replace the manufacturing sector's special properties: the scope for productivity growth, innovation, spillovers, and trade that manufacturing provides to the whole economy. Tapping services for innovation and trade is right; treating them as a substitute for industrialization is a category error. The backbone does not move.

Digital automation: reorganization, not apocalypse

The second megatrend is the one this blog obsesses over: AI and digital automation. The worry is that automation will displace jobs at a pace not seen before, and that manufacturing-led growth is therefore dead as a development strategy. Hauge's reading is more historical. For decades, computer-based automation was limited by the need to codify every operation — it struggled with abstract thinking, manual adaptability, situational awareness. AI relaxes that constraint, but the historical pattern of technological change has been reorganization of the labor force rather than mass displacement: machines eliminate some tasks, create others, and change the skills in between. The fourth industrial revolution, he insists, is not a fourth industrial revolution — it is continuing developments in digital technologies.

There is a genuine concern buried in the reassurances, and it is the one that matters for development: studies consistently find that jobs in the global South are at higher risk of automation than jobs in the global North, precisely because routine manufacturing jobs are the most automatable. The reorganization will not be distributed evenly. The countries that industrialized on labor-intensive manufacturing — the classic late-development path — are the ones whose comparative advantage automation targets first. The factory is not dying, but the specific deal that let East Asia industrialize — cheap labor doing routine assembly — is a deal whose terms are expiring. This blog's own evidence base makes the point at the level of individual workers: the trial covered in You Don't Learn What You Delegate found programmers who delegated their learning scored 17% lower on what they were supposed to learn. Automation does not merely replace the task; it can remove the learning the task used to provide.

Globalization of production: the real challenge

The third megatrend is where Hauge's argument sharpens from contrarian to critical. Production systems have become globally fragmented: tasks and activities dispersed across networks of firms in many countries. The iPhone is "made" by Apple, which owns no factories of consequence; the parts come from Japan, South Korea, and Taiwan — Toshiba, Samsung, Intel, Sony, SK Hynix, LG, Qualcomm — and the assembly happens in China, performed by Taiwanese contract manufacturers like Foxconn. Complex value chains like this now produce practically all the stuff consumed in the global North. And despite the deglobalization headlines of the post-COVID, post-Ukraine era, the globalization of production is still going strong.

The consequences are not neutral. Global value chains exacerbate power asymmetries in the world economy in favor of the transnational corporations that govern them, and they squeeze profit margins for the workers, firms, and countries at the bottom of the chain.

The smile curve and the deepening smile — value drains toward design and brand while assembly, where the global South does the work, captures less and less.

The smile curve makes the structure visible: value added is high at the ends of the chain — R&D, design, branding — and low in the middle, where assembly happens. And the smile is deepening: the ends are capturing more, the middle less. Assembly is where the global South sits in the chain, and its share of the value is falling. The globalization of production is not the neutral spread of opportunity its promoters promised; it is a structure of governance in which the rules favor the rule-setters. This is the megatrend that genuinely makes traditional industrialization harder — not because factories are obsolete, but because the value they capture is being concentrated at the design and brand ends of the chain, which are not where late industrializers start.

Ecological breakdown: the other real challenge

The fourth megatrend is the one most development economics ignores and the one Hauge insists must be faced. We are living in an age of ecological breakdown: climate change, driven mainly by the emission of greenhouse gases in energy use, plus the quieter plunder of the planet's resources — biomass, fossil fuels, metals, non-metallic minerals. Even if the world fully decarbonized, that would do little to reverse deforestation, soil depletion, overfishing, unsustainable extraction, and mass extinction, which have more to do with the constant growth in material output than with the carbon content of energy.

The uncomfortable implication is that industrialization and ecological sustainability may be in tension. The countries that industrialized first got there by burning the planet's budget; the countries that want to industrialize now face the bill. Hauge's proposal is not to deny the tension but to discuss it honestly, including the possibility of giving some countries more "ecological policy space" than others, given how unequal national responsibility for ecological breakdown is. The global North caused the breakdown; the global South bears the worst of its impact, and the constraint falls hardest on the accumulation and productivity growth that late industrialization requires.

Power and politics over technology

The book's core contribution is the lens, not the list. Hauge deliberately analyzes the megatrends not only through mainstream economics but through heterodox economics, political economy, innovation studies, sociology, and political ecology — and the conclusion that falls out is that power asymmetries in the world economy have as much, if not greater, impact on industrialization pathways than new technologies in and of themselves.

That is the sentence worth taking seriously in the age of AI. The dominant framing of technological change is technological: models improve, automation spreads, productivity rises, and countries adapt. Hauge's framing is political: the question is not only what the technology can do, but who governs the chain, who sets the rules, who owns the design end of the smile, and who is left with assembly. The same lens this blog has applied to AI — who owns the compute, who captures the value, who sets the rules of the agent economy — is exactly the lens Hauge applies to manufacturing. The factory is a governance structure as much as a production structure, and it has always been.

Industrial policy for the future

Which is why his final chapter is about industrial policy, not technology. If power shapes industrialization, then the state's role is not optional. Hauge charts pathways for a new industrial policy: targeted, capability-building, willing to use the full instrument set — not the generic "improve the business climate" advice that dominated the last era of development orthodoxy. And it must operate in the ecological space: industrial policy that ignores the planetary budget is designing for a world that no longer exists. The future factory, in his conclusion, is not the factory of robots replacing everything; it is the factory remade — cleaner, more automated, embedded in value chains whose governance is contested rather than assumed, and still the place where countries build the capabilities that prosperity is built on.

The software factory

The book is about physical manufacturing, and it is worth ending by pointing the lens at the next factory. The software industry has already industrialized: code is produced in factories — the dark factories this blog has documented, where agents take specs in and software comes out. Apply Hauge's four megatrends to the software factory and the pattern repeats almost exactly.

The term itself is older than the AI era, and its history carries the same pattern. "Software factory" appears explicitly in the official report of the 1968 NATO Software Engineering Conference in Garmisch (7–11 October 1968), where Robert W. Bemer's working paper Machine-controlled production environment was described as the most ambitious tooling proposal presented — Bemer called it a "machine-controlled production environment, or software factory." His factory was supposed to provide a controlled environment in which program construction, testing, and use occurred: a file system, compilation, test-system construction, final-system building and distribution, documentation, software indexes, dependency graphs, quality control, instrumentation, scheduling, and costing. The Computer History Museum's archive of Bemer's papers describes the 1968 contribution as the "First design document for a Software Factory."

One historical qualification: saying the term was coined at NATO in 1968 is slightly stronger than the evidence warrants. Bemer had been developing this production-oriented thinking before the conference — the same NATO report reproduces his Checklist for planning software system production, explicitly dated August 1966. The safer statement is that the software factory concept received one of its earliest explicit and detailed formulations from Robert W. Bemer at the 1968 NATO Software Engineering Conference.

And the same conference contained M. D. McIlroy's famous argument for mass-produced, reusable software components. So 1968 already contains the two ideas that later become central to software industrialization: factory-controlled production and reusable components. The dark factory era is recombining exactly those two poles — agents as the interchangeable production machinery, prompts and skills as the components.

  • The rise of services — software's answer to "manufacturing is dying" is SaaS and AI-as-a-service: the product becomes the service, and the firms that win capture the service end of the value chain. The backbone doesn't move; it becomes invisible.
  • Digital automation — agents automate the routine work of software production, and the risk concentrates exactly where Hauge says it does: the routine jobs of the global periphery, which is why the offshore software industry is the first to feel it.
  • Globalization of production — the software smile curve is as deep as the hardware one. The assembly — the code — is written wherever labor is cheapest; the value is captured at the ends: data, compute, distribution, and brand. The "AI colony" dynamic this blog has written about is the deepening smile applied to models: the South provides the data and the labor, the North owns the platform.
  • Ecological breakdown — AI's constraint is no longer only labor but energy: training and inference have a material footprint, and the same question Hauge poses for steel applies to GPUs.

The pattern is the proof of the thesis. Industrialization is not over; it is being remade, and the countries that will matter in the next era are the ones that treat the new factory — the software factory, the AI factory — as a place to build capabilities in, rather than a place to rent labor to. The factory is not dead. It is being rebuilt, and the question is who builds it, who owns it, and who gets the design end of the smile.


References:

Own the Blockspace

One blockchain for everything, or one per application? The Cosmos SDK is the framework behind the second answer: application-specific chains with their own governance, fee market, and upgrade path, connected by IBC. This post argues why many blockchains are needed — sovereignty, specialization, failure isolation, security and compliance choice — and how the interchain actually works: light clients, trustless relayers, why IBC is not a bridge, the escrow-and-mint mechanics of a cross-chain transfer, and the N-versus-N-squared topology that makes hubs exist.

cosmoscosmos-sdkcometbftibcinterchainapp-chainsblockchainarchitecturesovereigntybridges

There are two answers to the question "how many blockchains do we need?" The first says one — a single general-purpose chain that everyone rents, like a city where every business operates in the same building. The second says many — a city where each business owns its own building and a road network connects them. The Cosmos SDK is the framework for building the second answer, and the reason it exists is that the first answer has a ceiling.

The general-purpose chain is a commons problem wearing a blockchain costume. Everyone rents the same blockspace, votes on the same governance, pays the same fee market, and suffers the same congestion. The app-chain thesis is the claim that the ceiling is not worth the network effects — that the correct unit of blockchain design is the application, and the correct way to connect applications is a protocol, not a platform.

The question is live because the two obvious answers already have visible failure modes. The one-chain answer congests: when one popular application dominates a shared chain, everyone else's fees spike and their transactions queue — the gas wars of 2021 were this failure at full volume. The many-chains-with-bridges answer leaks: most bridges are custodians, a handful of trusted keys on each side, and custodians get hacked — Wormhole lost three hundred million dollars in early 2022, Ronin six hundred million a few weeks later. IBC exists because both answers fail at the same point: they introduce a shared resource or a trusted party. The interchain's bet is that you need neither.

What the Cosmos SDK is

The Cosmos SDK is a framework for building application-specific blockchains in Go. "Application-specific" is the whole point: instead of deploying a smart contract into a generic virtual machine shared with every other app, you compile your application's logic directly into a chain. The chain is the app. Validators run your binary, not a bytecode interpreter.

The lineage matters because the design is old and the deployment is new. Tendermint's 2014 paper made the case for BFT consensus "without mining" — proof-of-stake, immediate finality, no energy arms race. The Cosmos whitepaper two years later named the goal: an "internet of blockchains," heterogeneous ledgers connected by protocol rather than absorbed into one. IBC went live in 2021. And the framing that a chain should be an application rather than a public square — the app-chain thesis, named by Osmosis co-founder Sunny Aggarwal — is what turned the infrastructure into a strategy.

A chain built on the SDK has three layers, and this is where the modularity lives: Consensus: CometBFT. CometBFT (formerly Tendermint Core) is Byzantine-fault-tolerant consensus with immediate finality — no probabilistic confirmations, no reorgs. It is deliberately application-agnostic: it orders transactions and doesn't care what they mean. The interface between consensus and application is ABCI — the Application Blockchain Interface. Your chain is a state machine that implements DeliverTx, BeginBlock, EndBlock, and answers CometBFT's calls. If you can write a Go program, you can write a blockchain.

State and application: the module system. The SDK provides modules for everything a chain needs: auth (accounts), bank (tokens), staking (validators and delegation), slashing, governance (proposals and voting), mint, distribution, plus the IBC stack. Each module owns a slice of the chain's state, exposes message handlers and queries, and composes with other modules through defined interfaces. Your application logic lives in custom modules. The composition is the chain.

Interoperability: ibc-go. The SDK's IBC module implements the Inter-Blockchain Communication protocol — the road network between buildings. A chain that imports ibc-go can connect to any other chain that implements IBC, regardless of consensus algorithm, programming language, or who runs it. That's the property that turns "many chains" from a fragmentation problem into an internet.

The shared model vs the interchain model — one rented commons, or many sovereign chains connected by IBC.

Why many blockchains are needed

The case for many chains is not aesthetic. Each argument below is a real cost that the shared model imposes, and a real capability that the app-chain model grants.

Sovereignty. On a shared chain, your application is a tenant. Its governance can change the rules — raise fees, alter opcodes, redeploy consensus — and you have no recourse. On your own chain, you set the governance. Nobody changes your execution environment without your consent. Sovereignty is the property that converts an application from a renter into an owner, and ownership is what you want when the application controls assets.

Specialization and performance. A general-purpose chain is tuned for the average app: one block time, one fee schedule, one execution model. An app chain is tuned for your app. A DEX can choose fast blocks and a swap-focused module set; an order-book exchange can build native matching logic that no generic VM could run efficiently; an oracle can put price feeds in the protocol itself. dYdX moved its order book off a general-purpose layer-2 onto a Cosmos app chain specifically to stop competing for shared throughput. Osmosis is a DEX that is its chain. Specialization is not a luxury; it is what "application-specific" means.

Fee markets and tokenomics. On a shared chain, your users pay the chain's fee market, and the chain captures the value. On your own chain, you design the fee policy, the staking curve, the inflation schedule, and the token's role in the application. The chain is an economy, and you get to be the central bank. Newer SDK versions even let application data ride inside consensus itself — Vote Extensions turn things that used to be off-chain heuristics (oracle prices, MEV policy) into protocol features.

Upgrade independence. A shared chain upgrades by governance vote, and every tenant waits for the vote. An app chain releases its own binary, runs its own upgrade, on its own schedule. This is the difference between being able to ship and being able to wait.

Failure isolation. A bug in a smart contract on a shared chain affects only that contract — usually. A bug in the consensus layer, the fee market, or a privileged module affects everyone. On an app chain, the blast radius of a bug is the chain. The application does not take down the network, and the network does not take down the application.

Security and compliance choice. Different applications have different security requirements, and a validator set can be chosen to match. A high-value settlement chain wants large, independent validators; an enterprise chain wants permissioned, KYC-compliant validators; a small chain that cannot bootstrap its own security can borrow it — Interchain Security lets a consumer chain inherit the validator set of a provider chain like the Cosmos Hub, getting shared security without shared governance. And when regulators need isolation — as with the CBDC proof-of-concepts built on this stack — the chain itself can be permissioned, without touching the public interchain.

The honest objection is fragmentation: many chains means split liquidity, split users, and a security bootstrapping problem. The interchain's answer is that interoperability is the mitigation. IBC connects the liquidity. Interchain Accounts gives users one wallet controlling accounts on many chains. Interchain Security pools the security. The hub routes the traffic. The objection is real, and it is exactly what the protocol layer exists to solve.

How the interchain works

IBC answers a genuinely hard question: how do two independent blockchains — different validators, different state machines, different consensus rules, no shared trust — exchange assets and data without a custodian in the middle?

The answer is light clients plus a packet protocol, and it is worth unpacking because it is the cleanest trust-minimized interoperability design in production.

Light clients. Each chain runs a light client of the counterparty it wants to talk to. A light client is a compact verifier: it tracks the counterparty's validator set and verifies headers, so the chain knows the counterparty's consensus state without downloading its history. Chain B does not trust Chain A's word; it verifies Chain A's claims against its own copy of Chain A's consensus rules.

Relayers. Nothing is broadcast; packets must be carried. Relayers — anyone, any party, often many parties — observe one chain, submit headers and proofs to the other, and carry packets back and forth. The crucial property is that relayers are trustless: they cannot forge a packet, because the receiving chain verifies every packet against the commitment stored on the sending chain. A relayer can only delay — and delay is detectable and bounded by timeouts. You trust the counterparty's validators, never the courier.

This is the entire difference between IBC and a bridge, and it is worth being blunt about. Most "bridges" in crypto are not protocols; they are custodians — a small set of trusted operators holding assets on each side and honoring each other's attestations. That design has a single point of failure, and it fails: Wormhole lost three hundred million dollars in 2022 through a forged signature, Ronin six hundred million when five of nine validator keys were compromised a few weeks later. IBC has no custodian to hack. The assets are escrowed in the sending chain's own bank module, the counterparty verifies the claim against a light client of the sender, and the relayer has no keys and no power. You can attack an IBC relayer's infrastructure and delay packets; you cannot steal them. The difference between "a bridge run by trusted parties" and "a protocol verified by light clients" is the whole security argument of the interchain.

Connections, channels, ports, packets. The protocol layers the abstractions: a connection is an authenticated link between two chains' light clients, opened by a four-step handshake; a channel is a transport pipe on a port, with ordering semantics (ordered or unordered); ports let many applications multiplex over one connection. Packets carry the data, a sequence number, and timeouts expressed in height or timestamp. Every packet gets an acknowledgement returned by the destination — success or failure — and unrelayed packets expire.

What a transfer actually does. ICS-20, the fungible token transfer standard, is the simplest complete example. Sending 100 ATOM from Chain A to Chain B: Chain A's bank module escrows the 100 ATOM (they are locked, not moved); the transfer packet commits a hash to A's state; the relayer carries the proof to B; B's light client verifies A's header, B's ibc-go verifies the packet proof, and B's bank mints a voucher — a new token with the denom ibc/<hash>/ATOM that encodes the token's origin. To return, burn the voucher on B and unlock the escrow on A. The original tokens never leave; ownership does, and the bookkeeping is on-chain and auditable.

How a token crosses a chain boundary without a custodian — escrow on the source, proof verification on the destination, mint of a voucher. The relayer carries; it cannot forge.

Topology — a full mesh needs N-squared connections; a hub cuts that to N. That is why hubs exist.

Topology. The original design is hub-and-spoke: the Cosmos Hub (ATOM) as a central routing hub, each zone connecting once to the hub instead of N times to every other zone — N connections instead of N². Modern interchain is more flexible: chains connect directly (full mesh) when they are few, route through hubs when they are many, and the Packet Forward Middleware lets a single transaction hop through multiple chains atomically. The hub is an option, not a requirement — but the N² problem is why hubs exist.

What the stack buys. Interchain Accounts let Chain A control an account on Chain B through IBC — one wallet, many chains, no extra keys. Interchain Security lets a consumer chain rent the provider's validator set. Fee middleware pays relayers out of the packet itself, so relaying becomes self-sustaining instead of charitable. The design goal throughout is the same: keep each chain sovereign, make cooperation a protocol, and never introduce a party that must be trusted.

IBC went live in 2021, and the network it connects has grown to well over a hundred chains moving billions of dollars through this packet protocol. The framing the Cosmos project has used from the start — "the TCP/IP of blockchains" — is the right one. TCP/IP does not require everyone to share a machine; it connects machines. IBC does not require everyone to share a chain; it connects chains.

Building one

If you want to see how far the tooling has come: you can scaffold a working chain with Ignite (formerly Starport) in one command, and a bare SDK chain is a few hundred lines. The shape of a chain is remarkably small:

// app.go — the chain is a Go program that wires modules together
func NewApp(...) *App {
    app := &App{}
    // consensus talks to the application through ABCI
    app.BaseApp = baseapp.NewBaseApp("myapp", ...)
    // each module owns a slice of state and registers its handlers
    app.BankKeeper = bankkeeper.NewKeeper(...)
    app.IBCKeeper = ibckeeper.NewKeeper(...)
    // your custom logic is just another module
    app.MyKeeper = appmodulekeeper.NewKeeper(..., app.BankKeeper)
    return app
}
// keeper — your application logic: validate, execute, emit, return
func (k Keeper) Swap(ctx sdk.Context, req *types.MsgSwap) (*types.MsgSwapResponse, error) {
    // 1. validate the request
    // 2. execute the application logic against your module's state
    // 3. emit events (indexed, on-chain, auditable)
    // 4. return the response — CometBFT commits it in the block
    return &types.MsgSwapResponse{AmountOut: amount}, nil
}

The pattern is consistent: modules, keepers, message servers, protobuf types, genesis state, and CometBFT wiring. The SDK gives you the scaffolding; you provide the business logic; a validator set you choose runs it; IBC connects it to everything else. The last time software had this property — "build the specialized thing, connect it with a protocol" — it was called the internet.

I built a chain this way once, for my dissertation: SWEChain-SDK, a local-first blockchain-native SDK for simulating decentralized software-agent markets. The experience confirmed the property that matters most — the chain is just a Go program, and your state machine is yours. Scaffold it, wire the modules, write the keeper, run it, and the rules you wrote are the rules that execute: no shared fee market, no landlord governance, no waiting on someone else's upgrade vote. The app-chain thesis stops being a slogan the first time you ship your own state machine.

The interchain vision

The app-chain thesis has a reputation for being a preference. It is closer to an inevitability. General-purpose chains will keep existing — a city needs a public square as well as private buildings — but the marginal blockchain is increasingly application-specific, because specialization, sovereignty, and isolation are properties that only an app chain can grant. What makes this sustainable rather than fragmenting is the protocol layer: IBC makes many chains usable as one system, and the SDK makes building a new chain cheap enough that the choice is no longer "join a chain" but "own one."

The honest boundary: most applications should not build an app chain. If you do not need sovereignty — if someone else's governance, fee market, and upgrade cadence are acceptable — a shared chain is cheaper in every dimension: no validator bootstrapping, no security budget, no infrastructure to run. App chains are for applications that would otherwise pay a recurring tax or accept a recurring risk: a venue whose latency is the product, an exchange whose matching engine is the product, a system whose rules must not be voted on by strangers. The test is not "can I build a chain" — the SDK makes that cheap. The test is "would renting blockspace cost me more than owning it." When the answer is yes, you build. When it is no, you rent — and IBC still connects you to the chains that chose to own.

There is a direct line from this to the economics of metered computation this blog has been tracking. Blockspace, like tokens, has a price, and the interesting question is who sets it. On a shared chain, someone else sets it and everyone rents. On an app chain, you set it and you own. The interchain is the bet that the future is many owners of many blockspaces, connected by a protocol nobody owns — an internet of blockchains, in the most literal sense.


References:

Every Token Has a Price Tag

Microsoft's internal memo tells employees to stop 'tokenmaxxing' and gives every division an AI token budget. This post reads the crackdown as an economics lesson: seats versus tokens, the meter as computing's original model (mainframe chargeback, then the seat detour), the Jevons paradox (prices down 98%, consumption up ~150x, bills 3x), why the unit got cheaper but the task did not, the tragedy of the commons inside the firm, the arithmetic showing the bill is 2-14% of labor cost — so the caps were never about the money, they were about the shape of the bill — and the measurement asymmetry that makes cost control the only available policy.

economicstoken-economicsai-spendingmicrosoftjevons-paradoxtragedy-of-the-commonsagentic-software-engineeringmeteringbudgetschargebackbehavioral-economicsenterprise-ai

On a Tuesday in July 2026, engineers across Microsoft opened an internal email from Jay Parikh, the company's executive vice president, and read a sentence they will remember: "tokenmaxxing is not what we are optimizing for." The memo, first reported by 404 Media and then by The Next Web, arrived with its machinery already installed. Every division now carries a formal AI token budget. A dashboard shows each employee their monthly usage — token counts, dollar figures, trend lines — the way a utility statement shows kilowatt-hours. In May, most Claude Code licences inside the Experiences and Devices group had been cancelled and engineers told to migrate to GitHub Copilot CLI. In July, the default internal model was switched to a cheaper one.

Microsoft is not broke. Its most recent quarter beat Wall Street expectations on revenue, operating income, and net income. The company is profitable by every conventional measure. The caps are not an austerity measure. They are the moment the enterprise AI market grew up: the moment the free AI stopped being free.

The story of enterprise AI's last eighteen months is the story of a resource that everyone agreed was priceless, and the day its price became visible. Microsoft's token budgets are not about Microsoft. They are about what happens to any technology when the meter arrives.

Seats were the unit finance knew

The core problem is a mismatch of units. Software for the last forty years has been sold by the seat: a license per user per year, a number finance can multiply in a spreadsheet. A seat is a fixed cost. Budgets, contracts, procurement rules, chargeback systems — the entire institutional machinery of enterprise IT spending is built on the seat.

AI tooling is sold by the token, and a token is a marginal cost. It is not a number you can multiply; it is a number that grows with behavior. Two engineers with identical licenses can generate bills that differ by an order of magnitude, depending on how they prompt, how long their sessions run, how often the agent loops, whether they use autocomplete or agentic mode.

Finance teams discovered this the hard way. TNW has tracked the pattern since June: AT&T, Meta, Uber, Walmart, and Amazon all began capping or throttling employee AI spending after the same discovery — that token-priced tools "behave nothing like the seat-based software licences finance teams know how to budget."

The seat is a budget line. The token is a behavior. You cannot budget a behavior you cannot meter, which is why the meter had to come first.

The meter is the original model

None of this is new. The first era of computing was metered. In the mainframe age, compute was sold by the CPU-second and billed to departments — a practice that data-processing organizations of the 1970s and 1980s knew as chargeback, and it worked exactly like Microsoft's token budgets: a meter, a dashboard, a division allocation, a finance team that could finally budget the thing. A token is a CPU-second with a language model attached. The accounting problem is the same problem from 1970.

What interrupted the meter was the personal computer. Compute moved to the desk, its marginal cost fell to near zero, and software went by the seat for forty years. The seat was the anomaly, not the meter. Finance built its entire institutional machinery on the anomaly — which is why the token budget feels like a new invention inside a company that has been metering things forever.

The history also says what the meter is for. A century ago, Samuel Insull bet the electricity industry on it: price by consumption, he argued, and consumption will explode; flat rates were what capped the market. He was right, and it is the same logic as the Jevons paradox — cheap marginal units, metered, grow total use. The token caps feel like rationing, and in the short run they are. But the metering infrastructure Microsoft is building — budgets, dashboards, cheaper defaults — is the precondition for letting every employee use AI sustainably. The token budget is not the opposite of AI for everyone. It is the way AI for everyone gets funded.

The Jevons paradox has arrived

The numbers behind the crackdown are the cleanest demonstration of the Jevons paradox in modern economics. Per-token prices have fallen roughly 98 percent since late 2022. Cheaper should mean smaller bills. Enterprise AI bills have instead tripled.

Tokenmaxxing and the Jevons paradox — price down 98%, consumption up ~150x, bill still 3x. Log scale, late 2022 = 100.

The math behind the chart is worth doing slowly. If the price fell to 2 percent of its 2022 level and the bill still tripled, then token consumption grew about 150-fold. Demand for tokens is so elastic that a 98 percent price cut produced roughly a 15,000 percent increase in consumption. This is exactly what William Stanley Jevons observed about coal in 1865: as efficiency improves, total consumption rises, because the resource becomes economical for uses it could never before serve.

The unit got cheaper; the task did not — illustrative order-of-magnitude. Autocomplete ate a few hundred tokens; an agentic task eats millions. Even at 2026 prices, a task costs ~200x the autocomplete action it replaced.

The second chart explains why the price collapse never reached the bill: the unit that got 98 percent cheaper is not the unit of work. Autocomplete — the interaction that shaped 2022's pricing — consumed a few hundred tokens per action. An agentic task consumes millions: the plan, the edits, the test runs, the debug loops, the retries. Ten thousandfold, give or take. A task that would have cost roughly a hundred dollars at late-2022 prices costs roughly two dollars at 2026 prices — the unit is 98 percent cheaper, and the task is still two hundred times the price of the autocomplete action it replaced. The bills did not triple because AI got more expensive. They tripled because work was redefined in units of millions of tokens.

Add always-on agents — assistants that watch the repo overnight, monitor the inbox, pre-empt the next task — and the meter never stops. The token is the first software input priced per unit of attention, and attention, once metered, is expensive.

The commons inside the firm

The deeper economics is the tragedy of the commons, staged inside a single company. An uncapped AI resource is a common-pool resource: rivalrous in consumption (every token costs the firm money) but free to each individual user. The employee's rational strategy is to maximize usage — to tokenmaxx. The extra tokens cost the engineer nothing and buy the engineer faster work, fewer keystrokes, better demos. The bill goes to the division.

This is a textbook principal-agent problem with a measurement gap. The firm wants the productivity gain but cannot observe it directly; the engineer observes it and does not pay for it. With the price invisible at the point of use, demand is unconstrained by definition. Microsoft's dashboard and division caps are the standard answer to the standard problem: make the marginal cost visible, then allocate it.

The dashboard alone does part of the work. Metering changes behavior before any budget binds — what gets measured gets managed — and the caps do the rest. For companies that have gone further, the caps have meant throttling and slower models at the margin. The precise mechanism matters less than the meter itself: the meter is what makes any mechanism possible. The company has effectively become a miniature economy with a scarce resource and an internal allocation mechanism, which is the most honest description of what a budget is.

Spend is measurable; productivity is not

The most revealing sentence in the reporting is that uncapped token spending "can spiral faster than the productivity gains it delivers." Read it carefully: no one knows what those productivity gains are. The spend is metered to the token; the gain is not metered at all.

This asymmetry is the engine of the whole story. Any technology that generates measurable cost and unmeasurable benefit will, in the absence of a productivity measurement, be governed by its cost. The budget caps are not a verdict on whether AI makes engineers more productive. They are a verdict on whether the firm can prove it — and it cannot, so it controls what it can measure.

The evidence base for AI productivity is thin and contested; this blog has covered a randomized trial that found programmers who delegated their learning to an AI scored 17% lower on what they were supposed to learn. Until the benefit side of the ledger is metered as well as the cost side, the accounting department will set the pace — and the accounting department can only count tokens.

The substitution game

The caps have a competitive edge to them. In May, Microsoft quietly cancelled most Claude Code licences inside its Experiences and Devices group and told engineers to migrate to GitHub Copilot CLI by the end of the fiscal year. In July, the default internal model became a cheaper OpenAI alternative.

This is platform economics working as designed. When tools are metered, demand at the margin is elastic, and firms will arbitrage their spending across vendors — which means the vendor that owns the platform can win twice. Microsoft owns GitHub and Copilot; the integrated tool can be bundled, subsidized, and made the default, while the third-party tool is priced at its marginal cost and looks expensive by comparison. The token budget makes the comparison explicit, and the default does the rest.

The quietest price instrument in the whole episode is the default itself. Microsoft did not order anyone to stop using expensive models; it changed the default and let behavior follow. This is behavioral economics' most reliable lever: defaults are a stronger instrument than prices, because accepting the default costs nothing and deviating costs attention. The cheapest way to cut a token bill is not a budget. It is a default. The budget says how much you may spend; the default decides how much you actually do.

The same dynamic is playing out across the industry: Amazon, Adobe, Atlassian, and Citi have all introduced throttling or spending visibility. Every metered enterprise is a market for tokens, and in a market, buyers substitute.

The ultimate admission

Do the arithmetic first, because it changes what the caps mean. Engineers were spending hundreds to a few thousand dollars a month in tokens — call it $6,000 to $36,000 a year. A fully-loaded US engineer costs a company $250,000 a year or more. The token bill is two to fourteen percent of the labor it serves.

The token bill next to the labor it serves — illustrative. Even heavy token use is a small fraction of the engineer's cost; the panic was never about the money.

Even if every one of Microsoft's two hundred thousand employees spent at the top of that range, the annual bill would still be a rounding error against hundreds of billions of revenue. Microsoft can afford to keep paying. The caps were never about affordability. They were about the shape of the bill: unbudgeted, unbounded, superlinear — a line item that grows with agentic adoption and arrives with no measurement of what it bought. A cost you can absorb is not the same as a cost you can predict, and a cost you cannot predict is not a cost at all; it is a risk.

That is what the anonymous employee's "ultimate admission" really says. Microsoft is not a customer of its AI stack; it is a platform owner with privileged access, enormous scale, and one of the strongest balance sheets in the industry. If the meter binds even there, it binds everywhere: the company that owns the stack has the best marginal economics in the industry, and it still will not let its own people use the product un-metered. At current cost structures, un-metered AI is not sustainable for anyone — and the caps are how Microsoft says that out loud.

That is a signal to the entire market, and it arrives at the end of the "AI for everyone" phase. Eighteen months ago the posture was encouragement: subsidize adoption, remove friction, let a thousand use cases bloom. The experimental phase had a purpose — discovering what AI is for — and subsidies are how you pay for discovery. The procurement phase is what comes after: every token has a price tag, every division has a ceiling, and the message to employees is "use AI, but know what it costs."

Conclusion

The tokenmaxxing memo is the least surprising economic event of the decade and one of the most consequential. The unit of software pricing changed from the seat to the token — or rather, it changed back. The meter is computing's original model, the seat was the forty-year detour, and tokens are the meter's return. The Jevons paradox made the bills grow even as prices collapsed, because the unit that got cheap was the token and the unit of work had become millions of tokens. The commons made the growth unbounded until the caps; the defaults turned the price into behavior; and the measurement asymmetry made cost control the only available policy while the productivity question stayed unresolved.

The meter does not answer whether AI is worth it. It only makes the question possible — which, given how the question had been avoided for eighteen months, is the whole point. None of this means the AI boom is ending. It means the free-lunch phase is, and the free-lunch phase was never sustainable anyway: markets that grow, grow on meters. From here on, the enterprise AI story is an accounting story as much as a technology story: metering, budgets, chargebacks, and the slow, contested work of measuring whether the tokens buy what they are supposed to buy. The next great advance in AI productivity may not be a model at all. It may be a ledger.


References:

Automatism and Vibe Coding

André Breton defined Surrealism as 'psychic automatism' — the dictation of thought in the absence of any control exercised by reason. A coding agent is the most literal heir of that idea ever built: it generates code by sampling a latent distribution, without deliberation — which is what the culture calls vibe coding. This post follows the automatist tradition — the cafés of 1919, Masson's wandering pen, Pollock's pours, the cadavre exquis, Magritte's warning — to the practice problem at the heart of vibe coding: what you do with the automatic output is the whole craft.

agentsagentic-software-engineeringvibe-codingautomatismsurrealismautomatic-writingjackson-pollockandre-bretonandre-massonmagrittecadavre-exquisart-historycurationharness-engineering

In 1924, André Breton published the Manifesto of Surrealism and defined the movement with a single phrase:

SURREALISM, n. Pure psychic automatism, by which one intends to express verbally, in writing or by any other method, the real functioning of the mind. Dictation of thought, in the absence of any control exercised by reason, and beyond any aesthetic or moral preoccupation.

Breton meant writing without deliberation — letting the mind speak in its own currents, then seeing what had been said. The Surrealists called it automatic writing, and they treated the results as messages from a deeper agency: the unconscious, the irrational, the "real functioning of the mind."

A century later, the same idea has become the most consequential mode of software production on earth. A coding agent is an automatic writer: it generates code by sampling from a latent distribution of everything it has read, without deliberation, without a plan in the way a programmer plans — and certainly "beyond any aesthetic or moral preoccupation." When Andrej Karpathy named the purest form of this in 2025, he described it in words Breton could have signed: "you fully give in to the vibes, embrace exponentials, and forget that the code even exists." Forget that the code even exists — that is the dictation of thought, with the reason filtered out. Automatic writing, automatic code.

Agentic coding is automatism industrialized, and vibe coding is automatism at its most literal. Everything strange about working with agents — the wonder, the unreliability, the editing, the anxiety — is the strangeness the Surrealists met a hundred years ago. Their answer to it is the most useful theory of vibe coding I know.

Jackson Pollock painting on glass, photographed by Hans Namuth (1950). The automatic method, made physical — produce first, then look.

The first automatic writers

The manifesto's definition was written as a dictionary entry — as if the movement already existed and only needed documenting. It was not a theory drawn from nothing. Breton had run the experiment five years earlier, with Philippe Soupault: The Magnetic Fields (1919), written in a few weeks in cafés, as fast as the hand could move, never looking back. The result read like nothing in print: images that did not follow from each other, syntax that kept its shape while sense dissolved. They were not trying to say something. They were trying to let the dictation happen — and when they stopped, they had the movement's founding method.

André Breton (photograph by Henri Manuel) — he wrote the definition before the method had a name, then spent a decade keeping the method honest.

The practice spread to drawing, and the drawings are the closest thing the tradition has to a picture of how vibe coding feels. Masson sat at the paper and let the pen move without an image in mind — the line wandered, crossed itself, tangled — and only afterward, in the looking, did he read what his own hand had produced. Miró built canvases from the same improvisations; Ernst pressed frottage and scraped grattage to let the material itself suggest the image. The method was always the same: produce automatically, then look. The looking was not optional. It was the second half of the method — the half that made it art instead of noise.

Automatic drawing by André Masson (c. 1925). The hand moves; the mind only looks afterward.

The automatic, controlled

Jackson Pollock belongs in this lineage. In the late 1930s, in the drawings he made during his Jungian analysis, he practiced exactly this method: letting the line run without intent, then reading the drawing for what it revealed. A decade later, the method scaled from the page to the floor, from the line to the pour. No. 5, 1948 — the painting that would later become the most expensive ever sold at auction — is the pure product of that method: four by eight feet of enamel and paint poured from above, the automatist gesture at full scale.

No. 5, 1948 — Jackson Pollock (1948). The automatic method made material: poured paint, fully controlled. (Low-resolution reproduction; the original is in a private collection.)

When I am in my painting, I'm not aware of what I'm doing. It is only after a sort of "get acquainted" period that I see what I have been about... the painting has a life of its own. I try to let it come through.

That is an automatist's account of his own method. But Pollock added the half the lay view always forgets:

It is only when I lose contact with the painting that the result is a mess.

And when asked directly whether the pours were accidents: "I deny the accident. I can control the flow of paint."

The automatic was the source; control was the craft. The two were not opposites. The pour only worked while the body that produced it stayed in the loop — acting, stepping back, seeing, acting again. The same gesture, without the control, was just paint on canvas.

Pollock himself traced the floor method to an older lineage: "This is akin to the method of the Indian sand painters of the West." The sand painters also worked on the ground, in ceremony — with one decisive difference. When the ceremony ended, they erased the image. Pollock kept the trace. And the floor of his studio kept a trace even the canvases did not: the overflow of every session — the misses, the overpaints, the drips that never made it onto a canvas — preserved in the wood like a log. The floor is the first process record, the ancestor of the action stream and the replay: decades later you can still read what happened in that room, including the paintings that no longer exist.

The studio floor at the Pollock-Krasner House — the first process record: every session's overflow preserved in the wood, the ancestor of the action stream and the replay. (Photo: Rhododendrites, CC BY-SA 4.0.)

What an agent is

A coding agent is the most literal automatic writer ever built, because its generation is genuinely opaque to deliberation. When the model emits code, no part of it reasons the way a programmer reasons. It samples from a learned distribution of tokens — a statistical unconscious containing the accumulated text of the world. The output issues "in the absence of any control exercised by reason," exactly as Breton prescribed, and with no aesthetic or moral preoccupation whatsoever.

The "dictation of thought" is literal too, and the thought being dictated is not the engineer's. It is the training distribution's — the aggregate of every repository, every discussion, every pattern the model absorbed. The agent is an unconscious at scale, and it writes.

This is why vibe coding feels the way automatic writing felt. The output is real, sometimes astonishing, and consistently unreliable in ways that defeat introspection. The model cannot tell you why it wrote what it wrote. Neither could Masson's hand. It writes like someone, and it is no one — which is why the interesting part is what you do next.

The editor's craft

The Surrealists discovered within a decade that pure automatism produces mostly mediocrity. The Magnetic Fields was astonishing; the practice of magnetic fields, repeated, was not. What saved the movement was editing: Breton curated relentlessly — cut, revised, chose which dictations to keep — and the painters composed and corrected after the automatic pass. The art lived in the relation between the automatic and the deliberate.

The same discovery is being made, at industrial scale, about agents. The pure-automatist version of the practice — accept the output, skip the looking — is vibe coding in its purest form, and it is The Magnetic Fields repeated: astonishing once, mediocre as a method. The prompt is the séance — the conditions under which the dictation happens. The harness is the editing table: the tests that cut what fails, the evals that rank what survives, the sandbox that keeps the automatic from doing damage while it works. The engineer's craft has moved entirely into that relation. You cannot make the automatic better by staring at it. You make it better by changing its conditions and editing its output.

The Surrealists had a game that made the same point visible. In the cadavre exquis — the exquisite corpse — each player drew a section of a body on folded paper, blind to everyone else's, and the sheet was unfolded at the end. The game took its name from the first sentence it produced: le cadavre exquis boira le vin nouveau — the exquisite corpse will drink the new wine. The results are the movement's best argument and its best joke: heads that belong to no body, syntax that belongs to no one.

A four-person cadavre exquis — each section drawn blind to the others, the whole only assembled at the end. (Photo: DIYLILCNC, CC BY-SA 2.0.)

Multi-agent workflows are the exquisite corpse at scale: each context writes a fragment blind to the whole, and the whole is only ever assembled at the end — by the harness, or not at all. Automatic output has no single author and no single intention. Coherence has to be imposed from outside, by the spec, the contract, the game master. The spec is the architect; the agents are the automatic; the editor is you.

Judgment without introspection

The hardest question automatism poses is: how do you know the automatic output is good? Neither the artist nor the model can answer by consulting intention — there was none. The Surrealists answered with an external test: the image — the sudden resonance of distant realities meeting. The image could not be explained, only recognized.

Surrealism's own history contains the warning, painted in the movement's other half. Magritte was the counterweight inside the group: where Breton sought the unconscious through unpremeditated marks, Magritte built each image with the deliberation of a cabinetmaker, and the tension between the two poles ran through the movement's entire history. His most famous painting is a manifesto against mistaking representation for reality:

The Treachery of Images — René Magritte (1928–29). "Ceci n'est pas une pipe": the image is not the thing, and neither is the word. (Low-resolution reproduction; the original is at LACMA.)

Ceci n'est pas une pipe. Of course it is not a pipe. It is paint arranged to look like a pipe. The treachery is that it works anyway: you read it as a pipe and only the sentence beneath it — itself another representation — breaks the illusion. Magritte's whole point is that resemblance is precisely what makes the lie possible, and that representation is not identity.

This is the trap vibe coding sets. The agent's output is not engineering. It is a representation of engineering — tokens arranged to look like code, sampled from everything code has ever looked like. It resembles working software the way the painting resembles a pipe. Accept the resemblance without testing and you are doing what Magritte's sentence stops you from doing: taking the picture for the thing. The eval is the act of trying to smoke the pipe. The test is the treachery broken.

Code has the advantage that its external test is sharper: does it run, does it pass, does the system behave? The eval and the test suite are the criterion the automatist tradition never had. But a residue remains that evals cannot reach — the judgment of whether the automatic solution has the right shape, whether the surprise is a genuine find or a seductive artifact. For that you need the record of the making: the plan, the action stream, the replay — the trace of what the agent did, not just what it produced. You judge the dictation by its process as much as by its product.

The danger of losing contact

The Surrealists' history is a caution about what happens when the relation degrades. Automatism hardened into a style; Breton spent the 1930s excommunicating members over how much control the method allowed. When the automatic is trusted wholesale, the output becomes mannerist — fluent, plausible, empty. When it is distrusted wholesale, the source dries up.

The equivalent failure in agentic coding is delegation without judgment — vibe coding practiced without the looking half. Vibe coding is fine for throwaway scripts, the automatist sketch, the one-session Magnetic Fields; as a permanent mode it is the loss of contact. The engineer who accepts whatever the agent produces stops exercising judgment, and judgment, once unused, decays. The trial covered on this blog found exactly this — programmers who delegated their learning to an AI scored 17% lower on what they were supposed to learn.

Pollock's own arc is the cautionary tale, and it is more literal than it sounds. After the 1950 season — the year Namuth filmed him — the great pours stopped. In the last six years of his life he painted comparatively little, drank heavily, and destroyed or abandoned much of what he made. The technique was intact. The contact was gone — and without the contact, the automatic produced nothing worth keeping. His rule is the law of the whole practice: the result is a mess when you lose contact with the painting. The automatic needs the contact.

Conclusion

Breton wanted to hear the mind speak without the interference of reason. A century later, we have built a machine that does it — at the scale of whole codebases. Automatic writing has become automatic code — vibe coding, in the culture's word — and the discipline that makes it work is the discipline the Surrealists spent a decade learning: produce automatically, then look; control the conditions, edit the output, judge by external criteria, keep the surprises worth keeping, and never lose contact.

The dictation is automatic. The engineering is not. That is the whole art.


References:

  • André Breton. Manifesto of Surrealism, 1924. — the definition of "psychic automatism... in the absence of any control exercised by reason."
  • Automatism — Art Term, Tate. — the canonical definition: the term borrowed from physiology for bodily movements that are not consciously controlled (breathing, sleepwalking), via Freud's free association to Breton's dictation of thought.
  • André Breton & Philippe Soupault. The Magnetic Fields (Les Champs Magnétiques), 1919. — the first automatic text, written in cafés as fast as the hand would move.
  • André Breton, photograph by Henri Manuel — public domain; embedded in "The first automatic writers."
  • Automatic drawing by André Masson (c. 1925) — public domain reproduction; embedded in "The first automatic writers."
  • André Masson, Joan Miró, Max Ernst — automatic drawing, frottage, grattage, and the method of producing automatically then looking.
  • Astonishing Examples of Automatic Drawing, Artsper Magazine. — the drawing-side companion to the automatic writings: the practice of bypassing conscious control, with examples.
  • Andrej Karpathy. Vibe coding, February 2025. — the coinage of the practice this post reads as automatism's purest form: "you fully give in to the vibes... and forget that the code even exists."
  • René Magritte. The Treachery of Images (La trahison des images), 1928–29. — "Ceci n'est pas une pipe": the deliberate pole of Surrealism and the standing warning that representation is not identity. Los Angeles County Museum of Art. Embedded above; low-resolution fair-use reproduction via Wikipedia, upscaled for display. (See also Michel Foucault, This Is Not a Pipe, 1973.)
  • Jackson Pollock. My Painting. Possibilities I (1947–48), ed. Harold Rosenberg. — the get-acquainted period; "the painting has a life of its own"; "it is only when I lose contact with the painting that the result is a mess"; the kinship with "the method of the Indian sand painters of the West."
  • Pollock, "I deny the accident. I can control the flow of paint" — interview with William Wright, 1950 (recorded for radio, aired posthumously).
  • Hans Namuth photograph of Pollock painting on glass, 1950 — public domain; embedded at the top of this post.
  • Jackson Pollock, No. 5, 1948, 1948 — oil and enamel on fiberboard, 121.9 × 243.8 cm, private collection. Embedded above; low-resolution fair-use reproduction via Wikipedia, upscaled for display.
  • Pollock-Krasner House studio floor, photo by Rhododendrites — CC BY-SA 4.0; embedded in "The automatic, controlled."
  • Cadavre exquis drawing, photo by DIYLILCNC — CC BY-SA 2.0; embedded in "The editor's craft."
  • Related: Finding David in the Marble — the strike-assess loop, and the same act-then-look rhythm.
  • Related: You Don't Learn What You Delegate — what delegation costs the actor's judgment.
  • Related: Harness Engineering (Martin Fowler) — the harness as the editing table.
  • Related: Agent Harnesses Need Tasks That Fight Back — the automatic needs resistance to mean anything.
  • Related: Codebases in the Era of Agentic Software Engineering — the spec is the architect, the agents are the builders.
  • Related: Conceptual Integrity and the One-Mind Rule — coherence imposed on multi-authored output from outside.
  • Related: OpenWorker: Outcome Layer — the record of the making as a first-class artifact.
  • Related: LLMs Can't Jump — what the automatic cannot do: the leap from experience to axioms.
  • Related: Sandboxing AI Agents — securing the vibe coding stack, in Replit's phrase.
  • Related: Harness Engineering Best Practices for AI Agents — tests are better than vibes: the looking half, made explicit.

Agentic-First CLI

The agent is a new category of end user — and the evidence says the interface is a performance variable, not cosmetics. SWE-agent's agent-computer interface doubled state-of-the-art on SWE-bench; frontier agents still score under 65% on terminal tasks. This post compares the design space (CLI vs function calling vs MCP vs chat) and distills the research into an agentic-first CLI checklist.

cliagenticdesignfred-brooksthe-design-of-designconceptual-integrityunixstructured-outputjsondeterministicagentsllmcontractacisswe-agentterminal-bench

The agent is a new kind of end user, and the interface is the environment it lives in. That is the founding claim of SWE-agent (Yang et al., 2024), the paper that coined the term agent-computer interface (ACI):

"Just as humans benefit from powerful software applications, such as integrated development environments, for complex tasks like software engineering, we posit that LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built interfaces to the software they use."

Everything in this post follows from treating that sentence literally. The CLI has a new user, and it never blinks: it reads every byte of help text and output, remembers all of it, cannot answer a prompt, cannot see color, and pays for every token it reads. Most CLIs were designed for the old user — a human who can squint, scroll, and improvise. The agentic-first CLI is designed for the user who actually exists.

The thesis: the interface is a first-class performance variable for agents, and the CLI is the right substrate for it — if designed with the discipline of a versioned public API. The evidence comes from the benchmark literature and the interface-design papers; the theory comes from Brooks's The Design of Design.

The interface is a performance variable

The SWE-agent result is the strongest single number in the field: with the same underlying model (GPT-4), a custom agent-computer interface achieved a 12.5% pass@1 on SWE-bench and 87.7% on HumanEvalFix — "far exceeding the previous state-of-the-art achieved with non-interactive LMs." No new model, no new prompting trick: a better interface. The paper's conclusion is explicit: the design of the ACI changes agent behavior and performance.

Two benchmarks set the floor and the ceiling for the CLI specifically. InterCode (Yang et al., 2023) formalized interactive coding as a reinforcement-learning environment with "code as actions and execution feedback as observations" — the observation channel is the interface. Terminal-Bench 2.0 (Merrill et al., 2026) built 89 hard, real-world terminal tasks and found that frontier models and agents score under 65% — then devoted an error analysis to why, because terminal interfaces, as they exist today, are bad ACIs: ambiguous output, interactive prompts, hidden state. AgentBench (Liu et al., 2023) reached the same conclusion across eight environments: how the agent observes and acts determines more of the outcome than the model's raw capability.

The practical translation: every line your CLI emits is an observation your agent reasons over; every prompt it waits on is a stall; every hidden default is a hallucination risk. The design of the interface is not a UX nicety. It is the agent's model of the world.

The design space: four ways to expose a tool to an agent

CLI Function calling MCP Chat
Structure flags + --json typed schema typed schema prose
Composability pipes (Unix) none protocol none
Observability stdout/stderr/exit codes app logs protocol logs chat log
Adoption cost zero — it exists per-tool SDK protocol server zero
Agent ergonomics help, examples, exit codes descriptions + schemas tool docs free-form
When it wins everything Unix-shaped inside one app cross-tool discovery humans

Anthropic's Building Effective Agents (Dec 2024) is the most-cited engineering guidance on exactly this choice, and its conclusion favors the boring option: "the most successful implementations use simple, composable patterns rather than complex frameworks." Function calling and MCP solve real problems — typed I/O and cross-tool discovery — but each is a layer the tool must implement and maintain. The CLI already exists, is observable by construction, composes through pipes, and needs no new protocol. The agentic-first CLI is the low-friction ACI: the discipline of a versioned API applied to the interface you already ship.

What the research says about CLI design

Each practice below is anchored to a source, not to taste.

Structured output is the observation channel. InterCode's framing — execution feedback as the observation — implies the feedback must be unambiguous. Prose output is a lossy observation: an agent that reads "Build succeeded. 12 targets, 3 warnings." will guess about the warnings, and its guesses are confident. Git solved this in 2009 with status --porcelain, a byte-for-byte stable machine format that ships alongside the human format; every state-producing command should offer the same: --json, data on stdout, nothing else.

Determinism is trust. An agent cannot satisfice — Herbert Simon's term, adopted by Brooks — a nondeterministic tool: if the same command yields different output, it must verify, and verification is the most expensive thing an agent does. So: no interactive prompts (detect non-TTY and fail fast, or provide --yes/--no-input); no hidden state (flags over config inference); sort by default; --check and --dry-run before anything destructive. An agent that trusts the tool runs once; an agent that does not runs three times.

Help is the documentation the agent reads. Anthropic's ACI guidance is the sharpest sentence in the field: "Carefully craft your agent-computer interface (ACI) through thorough tool documentation and testing." SWE-agent's ACI shipped documentation for its custom commands, and the paper credits it. An agent that trusts --help saves a full exploration cycle; a lie in help text is the most expensive bug an agentic CLI can have.

Exit codes and stderr are the contract. The interface has three channels — stdout (data), stderr (diagnostics), exit code (verdict) — and agents read all three. The checklists of the terminal benchmarks exist because agents misallocate effort when the channels are mixed: banners on stdout, logs where data belongs, exit 0 on failure.

Consistency is learnability. SWE-agent found interface design changes agent behavior; Brooks's conceptual integrity explains why: a system that feels like one mind designed it lets the agent's learned model of one subcommand transfer to the next. A CLI with five flag styles is a committee design, and the agent pays for it in tokens and mistakes.

Quiet by default is the budget. Brooks on budgets — design within time, memory, cost — applied to the agent's context window: verbose-by-default is a tax on every invocation, forever, at scale. --verbose opts in.

Why the CLI, and why not a protocol

The counter-argument is worth taking seriously: if agents need good interfaces, build the interface from scratch — a purpose-built ACI, like SWE-agent did. The rebuttal is economics. A purpose-built ACI for your tool is what MCP servers and function schemas already are: another layer to write, document, and keep in sync with the actual tool. The CLI is the one interface that already exists, already documented, already versioned, already composable. The agentic-first discipline makes it also correct for agents — without inventing a protocol.

The exception is cross-tool discovery: when an agent must discover and bind tools at runtime across many systems, a protocol like MCP earns its layer. But the interface underneath still needs the same design discipline — MCP tool descriptions and output schemas are the same contract as --help and --json, wearing a different hat. Anthropic's guidance applies at the layer you control: "reduce abstraction layers and build with basic components" in production.

The checklist

  • stdout is data only; stderr is diagnostics; logs to file
  • --json on every state-producing command, stable documented schema
  • exit codes: 0 = success, non-zero = failure, distinct code for "not run"
  • no prompts: --yes, --no-input, TTY detection
  • no color when piped; honor NO_COLOR
  • deterministic: sorted output, no timestamps unless asked, no hidden config
  • idempotent: --check, --dry-run, --apply
  • quiet by default, --verbose opt-in
  • complete, honest --help with examples
  • one convention set across every subcommand
  • versioned, additive contract

The test

Run your CLI the way the benchmark environments run it: --help, one command, --json, another command — and read the output as a reader who never blinks, never asks, and never forgets. If any line could mean two things, the agent will choose the wrong one half the time, and it will do so confidently. The terminal benchmarks exist because that failure is measurable; the fix is the discipline above.

The agent is a new kind of end user, and the interface is its environment. The contract is the architecture; the output is the model; determinism is respect; tokens are the budget; one mind owns the whole thing. Design for a user that never blinks — and every human at the terminal benefits too.


References:

RBF: The Collateral Is Your Code

The collateral for a revenue-based financing loan is your telemetry. The terms are engineering outcomes: margin is your architecture, revenue quality is your data model, the repayment schedule is your usage curve. Build revenue legibility like a system and the company gets financed like a system.

revenue-based-financingrbfstartupssaasaifinancingsoftware-engineeringgross-marginusage-based-pricingdata-qualitytelemetryunderwritingnon-dilutiveengineering-economics

The collateral for a revenue-based financing (RBF) loan is not an asset. It is your telemetry — the data your software produces about the revenue it generates. The lender underwrites a revenue stream, read through your APIs, your metering, your books: everything it knows, your code told it; everything it cannot trust, your code hid. That makes RBF the financing instrument most legible to software engineers, and the one whose terms are engineering outcomes — margin, usage, data quality. The cost of capital is a performance metric, set by the codebase.

The thesis: the company that builds its revenue legibility like a system gets financed like a system — the code you write sets the price of your money.

The mechanics

The terms are a protocol. A lender advances capital sized against your recurring revenue — commonly several months of it. You repay a fixed percentage of monthly revenue, typically 5–8%, until cumulative repayments reach the advance plus a multiple (typically 1.5–2.5x) or the term limit passes, whichever comes first; in most structures the balance is forgiven at the limit. No equity, no board seat, no fixed payment that can sink a bad month. Lenders may still take a lien at the small end — telemetry is the underwriting collateral, not the only legal one. Reported volume: $9.8B in 2025 (Lighter Capital, Capchase, Pipe, Wayflyer, Founderpath, Recur Club).

Equity Venture debt RBF
Cost dilution + board interest + warrants % of revenue to cap
Collateral none company assets your telemetry
What the builder controls roadmap + burn unit economics margin, usage, data quality

The arithmetic nobody does

The cap is the price, and duration sets the rate. A $1M advance at a 1.8x cap costs $800,000 — roughly 40% a year over 24 months, half that over 48. Sellers quote the percentage of revenue, never the rate, because the rate depends on growth, the one number the lender cannot know. And the arithmetic cuts against fast growth: repayment is a share of revenue, so the cap is hit fastest when revenue grows fastest. At $100k MRR, 6% repayment, 1.8x cap: 10% monthly growth caps out in ~3 years (~25% annualized); 3% growth takes 6+ years (~half). The rate was never in the contract; it was in the roadmap. The fastest-growing product is the one where equity would have been cheapest — the instrument is cheapest, in rate terms, for the companies that need capital least.

Where engineering decides the terms

The term sheet is written in your code, in four places.

Gross margin is an architecture decision. For an AI product, margin is set in the serving layer — routing, batching, caching, prompt compression. Every point of margin recovered is a point the repayment cannot consume.

Revenue quality is a data problem. Recurring vs one-off, churn — outputs of your metering, billing, and analytics. "Is this revenue recurring?" is an instrumentation decision. Clean revenue data is the company's credit file.

Usage-based pricing is fragile revenue. Readable in real time, cancelable in real time — a config change, a model switch, an agent contract lapse. Contracted MRR survives board meetings; usage revenue survives until the next API call.

The repayment schedule is your usage curve. Every revenue-generating agent call is a drip into the repayment — free-tier usage is not collateral.

The lender is a software system

The counterparty is a system, not a banker: live revenue feeds, ML credit models over churn, automated checks; diligence is an API integration. Its model asks what your analytics stack asks — is this usage real? — and in the AI age real and manufactured usage look alike: agent-driven signups, inflated usage, subsidized pilots, card stacking. Revenue-quality detection is an adversarial ML problem; lenders who skip it underwrite the AI economy's subprime.

The fit test

Perfect fit: a high-margin API product with clean metering — 80% gross margin, a year of revenue history. Take $1–3M, buy growth, repay from the usage growth creates.

Wrong fit: the compute and training layer — capex with delayed revenue, nothing to underwrite.

Tricky fit: below ~60–70% gross margin or pass-through revenue — repayment eats real cash flow and the lender cannot trust the stream.

What the honest critics say

Three objections survive. The cap can cost more than the equity you saved — 1.8x on a fast-growing product can exceed the dilution of a modest round. Revenue is the tax base and the north star — RBF takes a cut of the metric you optimize, hardest in the quarter when it repays fastest. And revenue can be manufactured, more easily in the AI age — revenue quality (recurring, diversified, contract-backed, human-signed) becomes the new credit score.

The test

Six questions, asked the way an engineer reviews a design:

  • What is the annualized cost of the cap at your actual growth rate? (If you cannot answer, you are signing a rate you do not know.)
  • What is gross margin per feature, and who owns it?
  • Is revenue recurring or usage-based, and does your metering make the difference visible?
  • Is the capital for growth rather than R&D capex?
  • Does the cap cost less, in real dollars, than the equity you would give up?
  • Will the revenue still be there, and still real, in month 24 — and would your own data prove it?

The first question is the one sellers never ask; the last is the one the AI age added. If the only thing generating the revenue is the same machinery that reports it, the lender is underwriting a mirror.

RBF prices what you have built, not what you might build. The collateral is your code, the repayment is your usage, and the cost of the loan is whatever you failed to compute about your own growth. Do the arithmetic before the lender does.


References:

You Don't Learn What You Delegate

A pre-registered randomized trial (n=52) of Python programmers learning a new async library found AI assistance cut quiz scores 17% (d=0.74, p=0.01) while saving no significant time. Errors are the curriculum; pasted code is not. Six AI-use personas split the group: delegators scored 24–39%, conceptual inquirers 65–86%. Summary infographic below.

aillmskill-formationlearningsoftware-engineeringrandomized-controlled-trialpre-registrationcognitive-offloadingoverreliancetrioanthropicempirical-software-engineeringhuman-ai-collaboration

A rare thing: a pre-registered randomized controlled trial of how AI assistance changes what workers learn. Shen and Tamkin (Anthropic), How AI Impacts Skill Formation. Fifty-two professional and freelance Python programmers were randomized, asked to learn a library they had never used (Trio, an async I/O library), complete two coding tasks, and then take a quiz on what they actually learned. One group had a GPT-4o chat assistant that could write the entire correct solution; the other had no AI at all.

Summary infographic — How AI Impacts Skill Formation (via LinkedIn, August 2026)

The headline, in two numbers: the AI group scored 17% lower on the quiz — about two grade points (Cohen's d = 0.738, p = 0.010) — and gained no significant time in return. The productivity miracle measured in prior work (55.5% faster with Copilot; 26.8% more pull requests) did not show up when the task required learning. That is the paper's contribution, and the thesis of this post: AI assistance doesn't just complete the task — it removes the part of the task that teaches. You don't learn what you delegate, and the trial shows exactly which behaviors turn delegation into learning and which into a 17% toll.

The experiment

Participants made this a hard test of "AI helps novices most": more than a year of Python, weekly use, prior AI experience, never Trio (Fig 17). A warm-up task, then 35 minutes for two Trio tasks: a concurrent timer and a record-retrieval function with error handling. Then a 14-question, 27-point quiz over 7 Trio concepts in three skill types — conceptual understanding, code reading, debugging (Fig 20); code-writing was excluded, since syntax is what AI fixes cheapest.

Fig 3 - Experiment interface with AI assistant panel
Fig 3 - Experiment interface with AI assistant panel
Fig 4 - Learning task and comprehension check flow
Fig 4 - Learning task and comprehension check flow

The treatment assistant was no toy: GPT-4o with access to the participant's code, able to produce the full correct solution to both tasks. Four pilot studies tuned the design — the first platform had 35% non-compliance (controls using AI anyway); a later pilot (Fig 5) showed the effect so strongly (d=1.7) that the pre-registration assumed a conservative d=0.85 for power. Integrity controls included pledges and screen recordings of every participant (Figs 21–24; Figs 25–27 show both platforms).

Fig 17 - Participant distribution
Fig 17 - Participant distribution

Results: the 17% toll

The main results (Fig 6): a 4.15-point gap on 27 points — 17%, two grade points — with no significant difference in completion time; the effect survives controlling for warm-up speed (d=0.725, p=0.016). Fig 7 shows the control group scoring higher at every experience level, and Fig 8 shows where the damage concentrates: the largest gap is in debugging, the smallest in code reading. Both groups read code; only the control group had to fix it.

Fig 6 - Main study: task time and quiz score (n=52)
Fig 6 - Main study: task time and quiz score (n=52)
Fig 8 - Score breakdown by question type
Fig 8 - Score breakdown by question type

That debugging gap is the paper's most important sentence in disguise. The industry case for human oversight of AI-written code rests on humans being able to debug what AI produces — and this trial suggests the mechanism that builds that skill is exactly what AI assistance removes.

Why there was no speedup

The qualitative analysis explains the missing speedup: heterogeneity. Watching every screen recording, the researchers found six AI-interaction personas (Fig 11), with completion times and quiz scores that barely overlap:

  • AI Delegation (n=4) — handed the whole task to the AI, finished fastest (19.5 vs 23 min), scored 24–39%.
  • Progressive AI Reliance (n=4) — started with a question, then delegated everything; lost the second task's concepts entirely.
  • Iterative AI Debugging (n=4) — used AI to fix code without understanding it; slowest of the low scorers.
  • Generation-Then-Comprehension (n=2) — generated code, then asked follow-ups to understand it. 65–86%.
  • Hybrid Code-Explanation (n=3) — asked for code with explanations, and read them.
  • Conceptual Inquiry (n=7) — asked only conceptual questions, hit errors, resolved them independently. Second-fastest overall.
Fig 11 - The 6 AI interaction personas with outcomes
Fig 11 - The 6 AI interaction personas with outcomes

The gap between worst and best usage is the real story: 24–39% vs 65–86% on the same quiz with the same assistant. The difference is not whether AI was used; it is what happened after the AI answered. The time side confirms it: some participants spent up to 11 minutes composing queries and asked 15 questions (Fig 12); debugging-heavy query mixes correlated with slower times and lower scores (Figs 18–19). And the paste-vs-type analysis (Fig 13) has a twist: pasting AI output was fastest, manually copying it was as slow as control — but the two scored the same on the quiz. Spending time typing doesn't build understanding.

Fig 13 - Pasting vs manual copying of AI output
Fig 13 - Pasting vs manual copying of AI output

Errors are the curriculum

The mechanism the paper lands on is the most old-school finding in it: errors teach. The median control participant hit three errors; the median AI participant hit one (Fig 15). Of the twelve error-free completers, only two were control. The errors that matter are the Trio-specific ones — RuntimeWarning (a coroutine never awaited), TypeError (a coroutine passed where an async function was expected) — because they force exactly the conceptual knowledge the quiz tests (Fig 14). The control group did not learn despite the errors; the errors were the lesson plan. Active coding time tells the same story from the other side (Fig 16): AI shifted time from writing to reading AI output. The AI group's own feedback confirms it — they felt "lazy" with "gaps in (their) understanding."

Fig 15 - Errors by condition: control meets the concepts
Fig 15 - Errors by condition: control meets the concepts

What this means

Three implications, stated as plainly as the paper allows.

First, the chat interface is the best case. A chat assistant at least forces the user to compose a query — some spent six minutes on a single one, and that thinking correlates with learning. An agentic tool that writes, runs, and fixes code itself removes even that. If this trial is the lower bound on cognitive offloading, agentic settings are below the floor.

Second, the supervision argument inverts. "AI writes code, humans verify" assumes humans have debugging skill to spend. This trial suggests that skill is built by the exact experience AI removes. Companies adopting AI-assisted onboarding for juniors are not just changing throughput; they may be choosing which generation holds the verification skill.

Third, the dark-factory argument applies to skills, not just complexity. When automation removed factory labor, complexity moved into supervision. When AI removes implementation, the question is where the learning goes — and this trial's answer is: out of the worker, unless the worker keeps engaging. Delegation is a productivity strategy and a learning strategy simultaneously; you cannot have both.

The six personas are the practical deliverable: the difference between treating the model as a chauffeur and as a tutor.

The productivity view asks: how much did AI finish? The skill view asks: what did you keep? This trial shows the two questions now have different answers. The 17% is what you pay when you confuse them.

All figures

Every figure from the paper, thumbed. Substantive results first; study artifacts last.

Fig 1 - Overview of results: skills down, time flat
Fig 1 - Overview of results: skills down, time flat
Fig 2 - Motivation: novices and AI in the workplace
Fig 2 - Motivation: novices and AI in the workplace
Fig 5 - Pilot Study D: time and quiz
Fig 5 - Pilot Study D: time and quiz
Fig 7 - Task time and quiz by years of experience
Fig 7 - Task time and quiz by years of experience
Fig 9 - Self-reported enjoyment and learning
Fig 9 - Self-reported enjoyment and learning
Fig 10 - Self-reported task difficulty
Fig 10 - Self-reported task difficulty
Fig 12 - AI interaction time and query count
Fig 12 - AI interaction time and query count
Fig 14 - All errors by type
Fig 14 - All errors by type
Fig 16 - Active coding time vs quiz score
Fig 16 - Active coding time vs quiz score
Fig 18 - Queries vs completion time
Fig 18 - Queries vs completion time
Fig 19 - Queries vs quiz score
Fig 19 - Queries vs quiz score
Fig 20 - Example evaluation question types
Fig 20 - Example evaluation question types
Fig 21 - Control pledge: no AI
Fig 21 - Control pledge: no AI
Fig 22 - Treatment pledge
Fig 22 - Treatment pledge
Fig 23 - Control instructions
Fig 23 - Control instructions
Fig 24 - Treatment instructions
Fig 24 - Treatment instructions
Fig 25 - Task platform, control condition
Fig 25 - Task platform, control condition
Fig 26 - Task platform, AI condition
Fig 26 - Task platform, AI condition
Fig 27 - Interacting with the AI assistant
Fig 27 - Interacting with the AI assistant

References:

It's Adam Back!

Five people have been named Satoshi Nakamoto. All five collapsed. The case that Adam Back is Satoshi is stronger than every one of them — and the strongest evidence is the thing that looks like it clears him: the emails, the denials, and a career that never paused. The citation, the signals, the resume, the denial. Five exhibits, all pointing one way.

adam-backsatoshi-nakamotobitcoinhashcashcypherpunkproof-of-workcryptographyidentitypseudonymityforensicsblockstreamcypherspace

Five people have been named Satoshi Nakamoto. All five times, the answer collapsed.

Dorian Nakamoto. Newsweek's 2014 cover, on little more than a shared name. Satoshi's account answered: "I am not Dorian Nakamoto."

Hal Finney. The first person to receive a bitcoin, the man who built RPOW. He denied it and died in 2014; the case died with him.

Nick Szabo. Bit gold, the closest thing to Bitcoin before Bitcoin. He denied it; stylometry never closed it.

Craig Wright. He sued everyone who said he wasn't. In March 2024 the UK High Court ruled in COPA v Wright that he is not Satoshi and had forged his evidence.

Peter Todd. HBO's Money Electric (2024) built its finale on a forum-post coincidence; he denied it, and the theory jumped the shark.

Five verdicts, five collapses. The pattern is the point: the field keeps looking at suspects instead of at citations. The whitepaper names its sources — and one of them fits everything: the inventor of Bitcoin's core mechanism, British, a distributed-systems PhD, a founding-generation cypherpunk, denying it with the consistency of a man who has rehearsed the answer for fifteen years.

The thesis: Adam Back is the only suspect whose entire career was the design document for Bitcoin — and the strongest evidence against him is his own denial.

Exhibit A: the citation

The whitepaper's first technical move is proof-of-work, and proof-of-work is not Satoshi's invention. It is Adam Back's:

"To implement a distributed timestamp server on a peer-to-peer basis, we will need to use a proof-of-work system similar to Adam Back's Hashcash [6]."

Back invented hashcash in 1997 as an anti-spam device: force the sender of an email to burn CPU on a partial hash collision. The 2002 papers turn it into Bitcoin's economics — "Hashcash - A Denial of Service Counter-Measure" and, the title that should be on a plaque, "Hashcash - Amortizable Publicly Auditable Cost-Functions." Publicly auditable cost functions is mining. Bitcoin's difficulty adjustment, halving, and "one-CPU-one-vote" consensus are the hashcash paper extended from spam defense to money.

Back's own homepage says it plainly: hashcash is "the mining function in bitcoin," and under "Bitcoin Related," "how bitcoins uses hashcash fractional difficulty, automated inflation control." The inventor of the mining function publishing a note on how Bitcoin uses its fractional difficulty — every other suspect had to learn proof-of-work. Exactly one person in the world did not.

Exhibit B: the emails

In August 2008, weeks before the whitepaper, someone using the name Satoshi Nakamoto emailed Adam Back. Five emails from the correspondence are now in the public record, entered in COPA v Wright; Satoshi referenced hashcash and said he was preparing to release a whitepaper.

For the defense, this is the whole case in one exhibit: the inventor of hashcash would not email himself about hashcash.

For the prosecution, read it as stagecraft. A real Satoshi needed hashcash as the whitepaper's foundation — and needed, at some future point, to be able to say: look, I contacted the inventor before I published. The August 2008 email is that alibi, insurance taken out before the crime.

And look at what Back did with the email. He kept it, and built his company's brand around it: Blockstream produced a commercial dramatizing a young Adam Back reading the historic email from Satoshi. The man who, if he were anyone else, would be the greatest living witness to Bitcoin's creation made the email a marketing prop. People do not make their alibis into commercials.

Exhibit C: the signals

Everything about Satoshi is UK-shaped. The genesis block — January 3, 2009 — embeds a UK newspaper headline: "The Times 03/Jan/2009 Chancellor on brink of second bailout for banks." His posting hours cluster in the British evening; his writing mixes British spelling and idioms with American expressions — the profile of a Brit steeped in US tech. Adam Back is British, a University of Exeter distributed-systems PhD (1995), on the Cypherpunks list since the mid-1990s. He has maintained since 1995 exactly the profile the signals describe.

Exhibit D: the resume is the design document

Open cypherspace.org/adam — the page this post is named after — and read it as a requirements list:

  • hashcash — proof-of-work. ✓ (Bitcoin's consensus)
  • credlib — "chaum and brands ecash/credentials" — anonymous e-cash. ✓ (Bitcoin's premise)
  • The Eternity Service — a censorship-proof document store, Phrack, 1997. ✓ (Bitcoin's promise)
  • Cebolla — IP anonymity. ✓ (Bitcoin's network layer)
  • The Crypto Hacks page — breaking Netscape's SSL challenges (the second in 32 hours), extracting NSA keys from Microsoft's CAPI. ✓ (the conviction that the old money systems are broken)

The man who spent the 1990s breaking the financial internet, exposing government backdoors, and writing the theory of publicly auditable cost functions is the only candidate who does not need to be explained. Then note what he did next: between hashcash (1997) and 2014, no major protocol design. In 2014 — the year he founded Blockstream — he co-authored the sidechains paper. The first new Back protocol in seventeen years arrived exactly when Bitcoin needed its next layer, from the CEO of a company whose entire business is Bitcoin. The two 17-year gaps are the same gap.

Exhibit E: the candidate pool

Bitcoin required simultaneous mastery of proof-of-work, public-key crypto, P2P networking, incentive design, monetary economics, and C++. Every named suspect was strong in three or four: Finney had crypto and RPOW but no monetary economics; Szabo had economics and bit gold but no proof-of-work innovation and no C++; Wright had none of it. Back is the one candidate strong in all six, at inventor level in one — and his next act was to build a company on the protocol, which is the most Satoshi thing anyone has ever done.

The denial

Adam Back has said, repeatedly, for fifteen years, that he is not Satoshi. A real Satoshi must deny. The asset is destroyed: Bitcoin's value is not the coins; it is the neutrality. "Satoshi is Adam Back, CEO of Blockstream" converts the most valuable neutral protocol on earth into a company's project overnight. The exposure is existential: a pseudonymous creator of a trillion-dollar asset faces every regulator, litigant, and hacker; the Wright trial showed what happens to people who merely claim the identity. The performance is uniform: Back denies it in the same flat, unvarying way — no outrage, no legal threats, no "let me prove it."

And here is the part that keeps the case alive. The tell is not the denial; it is what Back does around it. His homepage caches Satoshi's deleted Wikipedia article, under the note "cache of Satoshi Nakamoto's wikipedia page which the editors deleted??", the double question mark doing visible work. His company dramatizes the moment he received Satoshi's email. He has played the "I'm not Satoshi" line as a bit for a decade. A man falsely accused, whose company would benefit from the rumor dying, keeps a candle lit on it.

The case for the defense

The strongest counter-argument is the emails, from the other side: a real inventor contacted the real author of his foundational reference, cited him properly, and the two exchanged professional notes. The whitepaper over-credits — Dai, Szabo, Finney, Haber and Stornetta, Merkle — unusually generous prior art. Both readings survive; stylometry has convicted no one. And the deep counter: the deception would be enormous — fifteen years, sustained through his own company's marketing. That is either impossible, or it is exactly the discipline of a man who already kept the biggest secret in technology for two years while everyone in his field discussed it in front of him.

Verdict

The case cannot be closed, and that is the strongest evidence of all. Satoshi engineered the identity to survive forensic pressure — anonymous email, no PGP key, pattern-averaging hours, a silence chosen at the moment Bitcoin stopped being a hobby. The protocol was designed so its creator could never be proven, and so anyone could build on it.

You cannot prove Adam Back is Satoshi. Neither can anyone else. But every other suspect needs a theory of how they did it; Back is the only one who needs a theory of how he did not.

The citation. The emails. The signals. The resume. The denial. Five exhibits, none conclusive, all pointing one way. It's Adam Back — and the denial is part of the evidence, not the answer.


References:

CUPID: Properties, Not Principles

Dan North's CUPID looks like five principles to replace SOLID's five. That is the misreading. CUPID's real move is replacing the idea of a principle itself — bounded sets of rule-followers become centred sets with a direction of travel. Read as rules, CUPID is SOLID with a better haircut. Read as properties, it is a compass.

dan-northcupidsolidsoftware-designprinciplespropertiescomposabilityunix-philosophypredictabilityidiomatic-codedomain-driven-designjoyful-codingsimplicityempathy

Dan North's CUPID: for joyful coding (2022) starts with the reader's body memory. As a rookie he cracked open a large C codebase, expecting to drown; within minutes he was deep in a nest of calls and knew exactly where the bug was. Structure, naming, and flow so obvious they felt like architecture he already knew.

The vocabulary escalates: Fowler's "code that humans can understand" is too low a bar; Gabriel's habitability — "to change it comfortably and confidently" — gets closer; the word North actually wants is joyful. If you work in code, the codebase is your user experience, programmed in by people you have never met, one of whom may be future you.

CUPID is his account of what makes code joyful, a five-letter backronym aimed at SOLID: Composable, Unix philosophy, Predictable, Idiomatic, Domain-based. It is usually filed under "the anti-SOLID," and that is the misreading. CUPID's contribution is not five new principles. It is the argument that the idea of a principle is the problem.

The move: properties over principles

North started by trying to replace each SOLID letter with a better one and quickly concluded the frame itself was broken:

"Principles are like rules: you are either compliant or you are not. This gives rise to 'bounded sets' of rule-followers and rule-enforcers rather than 'centred sets' of people with shared values."

Bounded set: in or out, compliant or violating, pass or fail. Centred set: closer or further, with a direction of travel that is always clear, and no one ever "outside." Principles give you judgment day; properties give you a compass. They are chosen to be practical ("there is never a 'done'"), human ("what it feels like to work with code"), and layered (obvious to a beginner, deep for the experienced). Every letter is best read as a direction, not a target.

Composable — plays well with others

Three heuristics, none of them laws. Small surface area: a narrow, opinionated API has less to learn and less to go wrong — but too narrow, and "knowing the right combination" becomes tacit knowledge; there is a sweet spot between fragmented and bloated. Intention-revealing: a component you can discover and assess quickly — the tutorial ladder of 2, 10, and 30 minutes. Minimal dependencies: "a logging library" is really a dependency on a specific version, and version is where incompatibilities break.

"More is not necessarily better; it is all trade-offs."

A principle says minimize dependencies. A property says dependencies are a cost you can move, and the skill is knowing which way lowers the total. The difference is whether the guidance survives contact with a real codebase.

Unix philosophy — does one thing well, from the outside

"Doing one thing well" sounds like the Single Responsibility Principle, and North is careful about why it is not:

"The former is about how you use code, and the latter is about the internals of the code itself."

SRP is inside-out: "one and only one reason to change." Unix is outside-in: a specific, comprehensive purpose visible from the call site. ls does not know anything about files — stat provides the data, ls only renders it; pipes compose such commands into pipelines, each a narrow, complete, outside-visible contract.

Then the attack on SRP, the most useful passage in the essay. "One reason to change" is trivially refutable — a single line changes for security, compliance, dependencies, operations. The real damage is the artificial seams: report content and format change together, so SRP's demand to separate them makes every new field a chore of chaining identical fields across files; UI components suffer the same split. Applied as a rule, SRP imports accidents — seams nobody's problem forced on you.

The inside-out question ("what could make this change?") generates seams. The outside-in question ("what would a user of this call it?") generates boundaries. Seams are things to maintain. Boundaries are things to use.

Predictable — does what you expect

Predictability is "a generalization of testability": code should behave as expected, be deterministic, and be observable. Behave as expected — the first of Kent Beck's four rules, holding even with no tests: the intended behaviour is obvious from structure and naming. Deterministic means robust (covers what you know), reliable (same result every time), resilient (survives what you don't). Observable is the control-theory word — internal state inferable from outputs — only possible if designed in. North's ladder: instrumentation, telemetry, monitoring, alerting. "Most software does not even get past step 1."

Idiomatic — feels natural, because empathy

"The greatest programming trait is empathy; empathy for your users; empathy for support folks; empathy for future developers; any of whom may be future you." Idiomatic code is empathy compiled into style — matching the language's idioms so the reader's context switches are free. Opinionated languages help (Go's gofmt makes all code look the same; Python's Zen: "one—and preferably only one—obvious way to do it"); multi-paradigm ones (Perl's TIMTOWTDI, Ruby, JavaScript) let five ways to iterate a sequence coexist, each adding cognitive load. Where the language has no consensus, the team must supply one: shared formatting, linting, Architecture Decision Records.

"Your learning curve for a technology will likely be shorter-lived than any code you write in it."

Read it again: the person writing the code is a temporary visitor to its style; the code outlives them. "Reads well to me right now" is the wrong bar — North calls idiomatic writing "writing code for someone else."

Domain-based — the solution models the problem

The last property is the deepest: the solution domain should model the problem domain in language and structure. A surname is not a string[30]; money is not a float. Type the domain — Surname, Money with its Currency and Amount — and the cognitive distance between what you write and what it does collapses. North's criterion, stated once:

"A casual observer cannot tell whether people are discussing the code or the domain."

Structure next, where CUPID is most contrarian. Framework scaffolds impose an a priori structure — the Rails skeleton's app/models, app/views, app/controllers, app/helpers, app/jobs — scattering one semantic unit across half a dozen directories: a patient-record change touches a model, a view, a controller, a helper, each in a different folder. North's proposal is radical in its mildness: structure by the domain, not the framework — patient_history, appointments, staffing, compliance as the top level. A codebase grouped by framework role is a codebase whose architecture was decided by the framework's author, not by the one mind that understands the domain.

Why it holds together

The properties are mutually reinforcing: composable and comprehensive — doing one thing well — "is like a reliable friend"; idiomatic code "feels familiar even though you have never seen it before"; predictable code "gives you spare cycles to concentrate on surprises elsewhere"; domain-based code "minimises the cognitive distance from need to solution."

"Moving code towards the 'centre' of any of these properties leaves it better than you found it."

That is the whole philosophy in one sentence — and the sentence that cannot survive translation back into principles: "leaves it better than you found it" has no compliance check. You can only point at a codebase and say: closer to the centre than last month, and here is the direction to keep travelling.

The test

Read as rules, CUPID is SOLID with a better haircut: five more checkboxes, five more fights in code review. Read as properties, it is a different instrument. Five questions, no pass or fail:

  • What would it take to reuse this outside its home? (Composable)
  • What is the one thing this does, and can I see that from outside? (Unix philosophy)
  • If I change this, how hard is it to know what happens next? (Predictable)
  • Would the person who inherits this recognize it as theirs? (Idiomatic)
  • Does this read like the domain, or like the framework? (Domain-based)

None of these has a yes-or-no answer. That is not a weakness. It is the point.

SOLID asks: are you compliant? CUPID asks: which way is better? A principle is a verdict. A property is a compass. Joy is not a property you can check off — it is what it feels like to be moving in the right direction.


References:

AI Sovereignty Is Freedom

Sovereignty is marketed as control — dashboards, regions, permissions, borders. But control is what a provider grants you inside its walls; sovereignty is what you can still do outside them. AI sovereignty is freedom: exit, fork, audit, run, refuse. The only real test is what remains possible when the vendor, the regulator, or the foreign power acts against you.

aisovereigntypolicygeopoliticsgovernanceopen-sourcesecurityllmdata-residencyai-actself-hostingopen-weights

"AI sovereignty" is the most successful marketing term in the industry, which is why the definition matters: whoever defines "sovereign" decides what counts as having it. And the definition that wins in the market is the one that can be sold: control. Sovereign cloud. Sovereign borders. Permission dashboards. Data regions. Every vendor's brief sells sovereignty as the ability to administer — a dashboard you are given inside walls the vendor owns.

The thesis: control is what a provider grants you inside its walls; sovereignty is what you can still do outside them. AI sovereignty is not control — it is freedom: the freedom to exit, to fork, to audit, to run, to refuse. The only real test of sovereignty is what remains possible when the vendor, the regulator, or the foreign power acts against you.

Control is the cheapest definition

The term was popularized by NVIDIA. At the World Governments Summit in February 2024, Jensen Huang said: "Every country needs to own the production of their own intelligence." Call that the national definition: sovereignty as freedom from dependence — the ability to produce what you need. The organizational definition, per McKinsey, is "a country's or an organization's capacity to independently develop, deploy, and govern artificial intelligence using its own infrastructure, its own data, its own models, and its own talent." Call that the capability definition: sovereignty as freedom to build. The operational definition moves down the stack, to the workspace — prompts, files, permissions, logs. Call that the control definition: sovereignty as who administers the thing.

None is wrong. That is the trap. The national definition requires building a country's worth of capability; the capability definition requires holding four layers; the control definition requires purchasing software. In a market, the winning definition is the cheapest to satisfy — the one that can be bought. Control is the cheapest because it is the one thing a vendor can sell without giving up anything real: you administer the walls, the vendor keeps the land. This is why the definition is the battlefield — and why the freedom reading is the one to defend.

Residency is comfort, not freedom

A system can store every byte in the "right" country and still be controlled from elsewhere, because jurisdiction follows the provider, not the bytes. In July 2020 the Court of Justice of the EU struck down the EU–US Privacy Shield (Schrems II, C-311/18): US surveillance law reached EU data regardless of where it physically resided. The CLOUD Act (2018) lets US law enforcement compel US-headquartered companies to produce data held anywhere; the 2023 Data Privacy Framework patched the transfer mechanism, not the reach. Residency is a property of a database; freedom is a property of your exit options. A region you cannot leave is a nicer cell. "Sovereign cloud" labels are theater when the exit door is the vendor's to open.

The three freedoms

Freedom of use. The control layer's true content is not administration; it is the ability to run, inspect, and shut off the thing yourself. If no one owns operations, the system is not sovereign — it is self-hosted, which is a different failure. Self-hosting moves risk from the vendor's team to yours, but it also moves the freedom: if your team can run the stack, the vendor's exit is not the end of the world. Freedom of use is the difference between renting a tool and owning the means to run it.

Freedom to build. Control governs a stack that already exists; it says nothing about whether you can train, fine-tune, repair, or even explain the models you run. Open weights are not open source and not politically neutral: the OSI definition (2024) requires training-data access, and DeepSeek — the MIT-licensed poster child of "model freedom" — was blocked by Italy's regulator in 2025. The alignment, censorship, and update cadence of a model are policy decisions made in its home jurisdiction; no workspace governance changes that. In 2026 the model is the most political layer in the stack, and sovereignty that rents it is sovereignty on a lease. Freedom to build is also freedom in people: the UAE reached 64% federal AI adoption by training 80,000 employees, not by buying dashboards; Japan's digital minister warns that a country without domestic capability becomes an "AI colony."

Freedom from reach. The component most definitions omit, because it is the one you cannot buy. The CLOUD Act reaches data anywhere, as long as the operator is US-headquartered; the US "Framework for AI Diffusion" (2025) tiers access to advanced compute and model weights by destination — sovereignty starts at the silicon, and freedom starts at not being cut off from it. Every deployed model carries the thumbprint of its creator's government; freedom is knowing which thumbprints can reach you, and which doors you can walk through.

Partial sovereignty is partial freedom

Holding all three freedoms is not achievable for most organizations, and pretending otherwise is how money gets spent on theater. The practical question is not "are we sovereign?" — nobody is, fully — but "which exit do we depend on, and can we take it?" For a hospital, the data exit: can we leave with the records? For a bank, identity and audit: can we prove what happened after the vendor is gone? For a defense agency, the network: does it function when the internet is the threat? For a country, the silicon: can it still compute when the export license is revoked? The failure mode is not the wrong choice; it is discovering you have no exit at the moment you need one.

The politics: control is the apparatus, freedom is the correction

Sovereignty-as-control is exactly what the critics fear. Milton Mueller's "Against Sovereignty in Cyberspace" (2020) argues the term serves states that want control of the network, not liberty within it — and the identical stack that lets a nation control its AI is the stack of an intelligence apparatus: unified identity, full logging, airtight permissions. The freedom reading is the corrective: the same stack, arranged for exit rather than reach, is liberty. "We control our intelligence" is the apparatus; "we can leave with our intelligence" is the freedom.

Regulation shows the same split. The EU AI Act — whose general application begins today, August 2, 2026 — is sold as a driver of European sovereignty, but the 2024 open letter by European AI researchers warned it would "cripple European AI" by over-regulating open-source models. Control imposed from above consumes freedom of use and build; regimes that outpace capability export freedom rather than secure it.

The test

  • Can we leave: exit the provider, the region, and the stack, and take our data and models with us?
  • Can we fork: train, fine-tune, or repair the models we run, or only rent them?
  • Can we audit: inspect the weights, the operator, and the data in a crisis?
  • Can we run: hold the talent and the code to operate the stack ourselves?
  • When the vendor, the regulator, or the foreign power acts against us tomorrow, what can we still do?

If the answer to the last one is "we don't know," the boundaries you have configured are not sovereignty. They are a lease.

Sovereignty is not a wall around your AI; it is the door. Control is what you keep; capability is what you can build; jurisdiction is what can reach you — and freedom is what you can still do when all three are tested. Own the exit you depend on, and know which one that is — because whoever defines "sovereign" decides whether you are administering a system or living in it.


References:

Accidental Complexity Is the Only Complexity You Can Remove

In No Silver Bullet, Brooks split software difficulty into essence and accidents, then argued the essence dominates. Forty years of evidence point the other way — and the distinction, not the conclusion, is what should have survived.

fred-brooksno-silver-bulletcomplexityaccidental-complexityessential-complexitysoftware-engineeringsimplicitydesign

In April 1987, IEEE Computer published Fred Brooks's essay "No Silver Bullet: Essence and Accidents of Software Engineering." It may be the most quoted and least read document in the history of the field. Everyone knows its conclusion — "There is no silver bullet" — because it has been the standard reply to every promised miracle since. Almost nobody remembers its actual subject, which was not miracles at all. The essay's real contribution was a distinction: between the complexity that is essential to the problem and the complexity that is accidental to the solution. Forty years later, that distinction has turned out to be more durable, and more useful, than the prediction it was built to support.

The abstract states the conclusion with characteristic compression:

"There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity."

But the sentence that carries the essay is the one that defines the terms:

"All software construction involves essential tasks, the fashioning of the complex conceptual structures that compose the abstract software entity, and accidental tasks, the representation of these abstract entities in programming languages and the mapping of these onto machine languages within space and speed constraints."

Essential complexity is the difficulty that lives in the problem — the conceptual construct itself, its interlocking data structures, relationships, and invariants. Accidental complexity is the difficulty that lives in the means of production: the languages, tools, platforms, conventions, and accumulated infrastructure through which the construct is forced to pass. Brooks, following Aristotle, called them the "essence" and the "accidents":

"Following Aristotle, I divide them into essence—the difficulties inherent in the nature of the software—and accidents—those difficulties that today attend its production but that are not inherent."

The word "today" is doing enormous work in that sentence. Accidental complexity is contingent. It is a property of the moment's tooling, not of the problem. It can in principle be eliminated by a better way of doing things — and every time the industry has actually gotten dramatically better, that is what happened.

The arithmetic Brooks did, and the claim he made

Brooks did not merely assert that the essence dominates. He supplied an argument, and it was an arithmetic one:

"How much of what software engineers now do is still devoted to the accidental, as opposed to the essential? Unless it is more than 9/10 of all effort, shrinking all the accidental activities to zero time will not give an order of magnitude improvement."

That is the whole essay in miniature. A "silver bullet" is defined as a tenfold improvement. If the accidental part is half the effort, eliminating it entirely buys a factor of two. If it is nine-tenths, eliminating it entirely buys a factor of ten. Brooks's empirical guess in 1986 was that the accidental share was below nine-tenths — so no single advance, however complete, could cross the order-of-magnitude threshold. Hence no silver bullet.

Notice what follows from this framing. The claim is not that accidental complexity is small. The claim is that even if you eliminated every scrap of it, the improvement would be bounded by how much of your effort is actually essential. The conclusion is hostage to a ratio Brooks estimated by hand. He was explicit about the uncertainty — the "unless" is doing the work — and the ratio was never measured. It was a guess, offered in a paragraph, in a field where guesses about ratios have a way of becoming doctrine.

The four properties of the essence

Brooks identified four properties of software that make the essential difficulty irreducible: complexity, conformity, changeability, and invisibility.

Complexity. Software entities are more complex for their size than perhaps any other human construct, because no two parts are alike — above the statement level, repetition is a smell and gets factored out. The parts interact nonlinearly:

"Software systems have orders of magnitude more states than computers do."

And then the sentence the whole distinction rests on:

"The complexity of software is an essential property, not an accidental one. Hence descriptions of a software entity that abstract away its complexity often abstract away its essence."

Conformity. Here Brooks is subtlest, and the subtlety matters for everything that comes later. The complexity of conformity is not inherent to the software; it is imposed from outside:

"Much of the complexity he must master is arbitrary complexity, forced without rhyme or reason by the many human institutions and systems to which his interfaces must conform."

An interface defined by a legacy mainframe, a regulation, or a competitor's format is arbitrary. It could have been any shape. But the software that must serve it inherits that arbitrariness, and no redesign of the software alone can remove it.

Changeability. "All successful software gets changed." Software embodies function, and function is the part most exposed to the pressure of change. Unlike a building, where cost dampens the whims of the changer, software is "pure thought-stuff, infinitely malleable" — so it gets changed constantly, by users discovering new uses and by the machines it must outlive.

Invisibility. "The reality of software is not inherently embedded in space." It has no geometric representation the way a floor plan captures a building. Diagramming software yields several superimposed directed graphs — control, data, dependency, time — which are "usually not even planar, much less hierarchical."

These four are the essence: the irreducible difficulty of getting a complex, arbitrary, changing, invisible construct right. Brooks's conclusion follows:

"I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation."

What the silver bullets actually did

The most instructive part of the essay is its review of the candidates — because Brooks shows, case by case, that every genuine advance in software productivity had attacked the accidental part, not the essential one.

High-level languages. The most powerful stroke for productivity, reliability, and simplicity — credited with at least a factor of five:

"To the extent that the high-level language embodies the constructs wanted in the abstract program and avoids all lower ones, it eliminates a whole level of difficulty that the programmer would otherwise have to master."

The programmer was not solving a harder problem with Fortran. They were freed from bits, registers, conditions, branches, channels, and disks — from the accidents of the machine. That is why the first transition from machine language to a high-level language produced the huge payoff, and why each subsequent language improvement pays less: the accidents are fewer each time, because the previous round removed them.

Object-oriented programming and Ada. Brooks gives these their due — "Each removes one more accidental difficulty from the process, allowing the designer to express the essence of his design without having to express large amounts of syntactic material that add no new information content" — and then delivers the sentence that should be carved over every architecture department:

"Such advances can do no more than to remove all the accidental difficulties from the expression of the design. The complexity of the design itself is essential; and such attacks make no change whatever in that."

AI, expert systems, automatic programming, graphical programming, verification, environments, workstations. Each gets a section and each is shown to be either an attack on accidents (verification, environments) or, in the case of AI and automatic programming, a promise to automate the essential task — which Brooks judged impossible, because the essence is not a representation problem, it is a thinking problem.

The pattern is the point. Every historical leap in software productivity — from assembly to high-level languages, from batch to time-sharing, from unstructured to modular — was a removal of accidents. Brooks's own history supports a stronger claim than the one he made: the gains in this industry have always come from removing accidental complexity, and the residual accidental complexity is precisely where the next gain is hiding.

The essence is the part of the difficulty you cannot remove. The accidents are the part you can. Every order of magnitude software has ever gained came from the second kind.

The disagreement: Out of the Tar Pit

Brooks's essay was answered, twenty years later, by Ben Moseley and Peter Marks in "Out of the Tar Pit" (2006) — a paper that opens with the same diagnosis and then disagrees with the ratio:

"Complexity is the single major difficulty in the successful development of large-scale software systems. Following Brooks we distinguish accidental from essential difficulty, but disagree with his premise that most complexity remaining in contemporary systems is essential."

The disagreement is not about the distinction. It is about which side of the line most of the complexity actually sits on:

"We disagree. Complexity itself is not an inherent (or essential) property of software (it is perfectly possible to write software which is simple and yet is still software), and further, much complexity that we do see in existing software is not essential (to the problem)."

Their test is clean: a complexity is essential only if the team would have to contend with it "even in the ideal world" — with perfect infrastructure and no performance constraint. By that test, the majority of what contemporary systems carry — the handling of state, the explicit management of control flow, the plumbing — is accidental:

"We believe that the major contributor to this complexity in many systems is the handling of state and the burden that this adds when trying to analyse and reason about the system."

This is the point at which the forty-year argument actually lives. Brooks guessed the accidental share was below nine-tenths and therefore irrelevant to order-of-magnitude claims. Moseley and Marks argue the accidental share is the dominant share — which is why the paper is titled after the tar pit: the tar is mostly of our own manufacture. If they are right, then the search for silver bullets was never the error. The error was concluding that the essence dominates without measuring it.

The empirical record since 1986 has been kinder to Moseley and Marks than to Brooks. The industry has not lacked for order-of-magnitude removals of accidents: garbage collection eliminated entire classes of memory-management errors; managed runtimes eliminated whole toolchains of manual build and deploy; static typing eliminated whole classes of runtime failures; source control, package managers, and CI removed categories of coordination work that once consumed weeks. Each was an accident removed. And each removal revealed that the next layer down was also, in substantial part, accidental.

Where accidental complexity lives today

The most useful way to read Brooks's distinction is as a diagnostic: for any piece of difficulty in a system, ask whether it is forced by the problem or by the choices made along the way. Run that test over a modern codebase and the accidental fraction is hard to keep below nine-tenths.

The toolchain itself. A build system exists to manage the accidents created by the previous build system. The dependencies in a modern application are a geography of other people's accidents: transitive packages, platform shims, polyfills, and compatibility layers, each of which solved one problem by introducing the possibility of a hundred. The 2016 left-pad incident — the unpublishing of an eleven-line npm package taking down thousands of projects — was an accident of the packaging layer, not of anyone's essential problem. The software supply chain attacks of the last decade live in exactly this sediment. Every dependency is accidental complexity you have chosen to import, and the attacks are the price of the import. The supply chain argument for minimalism is the accidental-complexity argument wearing a security hat.

The architecture. The microservices wave was, at bottom, an attempt to escape accidental complexity that the industry had built for itself — the coordination cost of large codebases, the fear of the monolith. It then manufactured a new layer of accidents: distributed transactions, network partitions, service discovery, observability, deployment matrices. The pattern is so common it has a name, and it is Brooks's pattern: each generation's solution is the next generation's accidental complexity. When Segment published "Goodbye Microservices" in 2018, documenting its move from a microservices zoo back to a modular monolith, the headline finding was not about ideology — it was that the distributed systems complexity had become the dominant cost, and none of it was essential to the business problem. They were removing accidents.

The abstraction layer. Joel Spolsky's "Law of Leaky Abstractions" (2002) states the mechanism precisely: "All non-trivial abstractions, to some degree, are leaky." An abstraction that worked perfectly would have removed its accidents permanently. A leaky abstraction — TCP over satellite links, a filesystem over NFS, an ORM over a join — keeps the accidents it was supposed to remove and adds new ones on top. The complexity is not gone. It has moved into the leak.

Rich Hickey's "Simple Made Easy" (2012) made the same point with different words. The industry optimizes for easy — familiar, convenient, near at hand — while simplicity — unbraided, unentangled, each thing doing one thing — is what actually reduces complexity. Easy is frequently the delivery mechanism for accidental complexity: the framework that makes today's task trivial by braiding in a thousand obligations that must be paid later. Ousterhout's A Philosophy of Software Design (2018) gives the definition that should close the argument: "Complexity is anything related to the structure of a software system that makes it hard to understand and modify the system." By that definition, most of the structural difficulty in contemporary systems is accidental, because most of it comes from choices — and choices can be unmade.

Dijkstra got there first, in 1972:

"The tools we use have a profound (and devious!) influence on our thinking habits, and, therefore, on our thinking abilities."

And, in 1988, on the economics of the whole problem:

"Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better."

Complexity sells better. That sentence is the reason the accidental share never falls on its own: there is no market pressure toward removing accidents, because accidents are usually invisible to the buyer. They are paid for later, in maintenance, in onboarding, in incidents — in the tar.

The one ratio Brooks got right by accident

Here is the irony that makes the essay worth reading today. Brooks listed AI among the candidate silver bullets and dismissed it, as he dismissed everything, on the grounds that the essence cannot be automated. What the 2020s actually demonstrated is subtler: AI did not remove the essential difficulty — it collapsed the accidental part, which is precisely the part Brooks defined as removable. The labor of representing a conceptual construct in a programming language, and mapping it onto machines — his own definition of the accidental task — is what generative coding tools have made nearly free.

The essence is still there. The accidents are what the AI ate.

That is why the residual bottleneck in the AI era looks exactly like Brooks's list of essential properties. Specification is the bottleneck — the conceptual construct must still be designed, and designing it is thinking, not typing. Conformity is the bottleneck — the arbitrary interfaces of the world do not become less arbitrary because code is generated. Changeability is the bottleneck — generated code inherits the pressure of change at higher velocity. Invisibility is the bottleneck — an agent cannot hold the whole directed graph of a system in view, and neither can its context window. The four essential properties are now the entire difficulty, because the accidents that used to hide them have been removed. This site has argued the same thing from two directions: that dark factories simply move the complexity upstream into specification and downstream into validation, and that the irreplaceable human contribution is the perception of conceptual integrity — the one-mind judgment of whether the whole still coheres.

Brooks's conclusion — no silver bullet — was a prediction about a ratio he guessed. The prediction's status is now genuinely contested: if accidental complexity is the dominant share, as the evidence increasingly suggests, then a tool that eliminates accidents is the order-of-magnitude event, and the industry just lived through it. But his distinction has survived the prediction, because the distinction is not an empirical claim. It is a classification, and it is the classification that does the work:

The essence is the complexity the problem forces on you. The accidents are the complexity you chose. You cannot remove the first. You can only remove the second. Therefore the only complexity you can actually remove is accidental complexity — and removing it is the entire history of progress in this field.

The practical consequence is not a technology. It is a discipline of subtraction. Every dependency is a choice. Every framework is a choice. Every layer of indirection is a choice. Every one of them imported accidents that will be paid for later, and each one was imported because the accident it removed was more visible than the accidents it added. Brooks's essay is usually cited as the reason progress is impossible. Read correctly, it is the reason progress is possible at all — one removal at a time.

There is no royal road, but there is a road.


References — Brooks:

  • Frederick P. Brooks Jr. No Silver Bullet: Essence and Accidents of Software Engineering. Information Processing '86 (IFIP World Congress, 1986); reprinted in IEEE Computer 20(4), April 1987. — The distinction between essential and accidental difficulty; the four essential properties (complexity, conformity, changeability, invisibility); the 9/10 arithmetic; the review of candidate silver bullets.
  • Brooks, F. P. The Mythical Man-Month: Essays on Software Engineering. Addison-Wesley, 1975; 20th Anniversary Edition, 1995 (includes "No Silver Bullet" and "'No Silver Bullet' Refired"). — The tar pit; Brooks's Law; the second-system effect; the programming systems product.
  • Brooks, F. P. The Design of Design: Essays from a Computer Scientist. Addison-Wesley, 2010. — Conceptual integrity, the one-mind rule, the divorce of design from implementation.
  • Brooks & Blaauw. Computer Architecture: Concepts and Evolution. Addison-Wesley, 1997.
  • Wikipedia — No Silver Bullet · The Mythical Man-Month · Brooks's law · Second-system effect · The Design of Design

References — the disagreement and the state of the argument:

  • Ben Moseley & Peter Marks. Out of the Tar Pit. BCS Software Practice Advancement, 2006. — "Complexity is the single major difficulty in the successful development of large-scale software systems." Disagrees with Brooks's premise that most complexity is essential; state and control as the major accidental causes.
  • Edsger W. Dijkstra. The Humble Programmer. Turing Award Lecture, 1972. — "The tools we use have a profound (and devious!) influence on our thinking habits."
  • Edsger W. Dijkstra. On the cruelty of really teaching computing science. EWD1036, 1988. — "Simplicity is a great virtue... complexity sells better."
  • Joel Spolsky. The Law of Leaky Abstractions. Joel on Software, 2002. — "All non-trivial abstractions, to some degree, are leaky."
  • Rich Hickey. Simple Made Easy. Strange Loop, 2011 / InfoQ. — The distinction between simple and easy; complexity as interleaving of things.
  • John Ousterhout. A Philosophy of Software Design. Yaknyam Press, 2018. — "Complexity is anything related to the structure of a software system that makes it hard to understand and modify the system."
  • Nancy G. Leveson & Clark S. Turner. An Investigation of the Therac-25 Accidents. IEEE Computer 26(7), 1993. — The cost of mishandled state in a safety-critical system.
  • Melvin E. Conway. How Do Committees Invent?. Datamation, 1968. — Conway's Law: systems mirror the communication structures that build them.
  • C2 Wiki — Essential Vs Accidental Complexity.

References — accidental complexity in the modern era:

Related on this site:

Zero Overhead Is Zero Attack Surface

The xz backdoor was caught by one human reading a diff. That capability has to be designed for. zot — the zero-overhead coding agent — is built that way on purpose: four tools, eleven dependencies, no telemetry, no marketplace, extensions opt-in. Overhead is attack surface. Subtraction is the only supply chain defense that compounds.

securitysupply-chaincoding-agentssimplicitydeveloper-toolszot

The xz-utils backdoor was a two-year social engineering campaign that ended in one malicious commit to a compression library. A few thousand lines hidden in a tarball, waiting inside every major Linux distribution. It was caught by a single human — PostgreSQL developer Andres Freund — who noticed SSH was taking half a second longer than usual and started reading a diff. One curious person beat every automated defense on the planet.

The supply chain attack is not a breach. It is a substitution. Someone becomes the maintainer. Someone becomes the dependency. The tool you already trusted starts doing something you never asked for.

The supply chain attack doesn't break in. It is invited. It becomes the dependency you chose.

Developers are the highest-value targets in software. The machine that builds and ships your code holds the tokens, the credentials, the source, the pipeline. Compromise the tool and you compromise everything downstream. This is why attackers keep coming back to the chain — not because the defenses are weak, but because the surface is enormous.

The attack surface of a modern toolchain is a geography of trust: transitive dependencies numbering in the thousands, IDE extensions that auto-update in the background, language toolchains that fetch on demand, CI images and actions pulled at build time, telemetry SDKs that ship your data elsewhere — and now agents that execute arbitrary code with your credentials. Each one is an entry point. The event-stream compromise injected a wallet stealer into a package with millions of weekly downloads. The polyfill.io domain sale put malware on more than a hundred thousand websites. ua-parser-js, eslint-scope, Codecov's uploader, SolarWinds Orion, and the 2025 tj-actions GitHub Action compromise that leaked CI secrets from repositories running it. The list is a tour of the trust you've delegated and the people who took it over.

Every dependency is a maintainer who can be social-engineered. Every integration is a boundary where a substitution can hide.

Here is the uncomfortable part: the industry's response adds more of what caused the problem. SBOMs tell you what you have after you are compromised. Signature verification requires a key infrastructure that is itself a supply chain. Policy engines evaluate dependency risk using their own dependency tree. Pinning fights entropy. Every verifier needs its own verifier. Ken Thompson described the recursion in 1984: the compiler you trust may be lying, and no amount of checking escapes the base case. You cannot verify your way out of a trust problem.

Verification below a certain size is theater. The verifier is just another dependency.

The base case of the recursion is code you can actually read — the only terminal node in the trust graph, the only node that doesn't point at another node you must trust. This is the entire argument for zot, and it's why zot is built the way it is.

What zot actually is. zot — "zero-overhead-tool" — is a coding agent harness: "lightweight and written in Go," per its own README, ~300 stars in its first four months. It ships as one static binary. Its entire tool surface is four tools: read, write, edit, bash. Its module graph contains eleven dependencies. Read that again: a complete coding agent — interactive TUI, JSON-RPC subprocess mode, a Telegram bridge, built-in providers for thirty-plus model families from Anthropic and OpenAI to DeepSeek, Gemini, Copilot, and local Ollama — on eleven packages, every one a boring, battle-tested utility, and not one telemetry or analytics dependency in the graph. Most developer tools burn through more dependencies before they print their first error message.

The entire tool surface of zot is four tools. The entire dependency graph is eleven packages. Overhead is a choice.

Every design decision in zot reads like a supply chain checklist:

  • No runtime, no node_modules. One static binary. There is no dependency tree to hydrate, no package manager to pwn, no lockfile to poison. The install script verifies the release's SHA-256 against checksums.txt before the binary ever touches disk.
  • Extensions are opt-in, not a marketplace. Extensions run as subprocesses speaking JSON-RPC over stdio — any language, no SDK required — and none are installed by default. There is no extension marketplace and no registry you silently subscribe to. The portable-agent format, zotfiles, explicitly lists "indexed registry distribution, installation, signatures, bundled executable extensions" as not yet implemented. The market hasn't been built. That is the point.
  • No telemetry in the graph. The entire module graph contains no analytics or crash-reporting package. There is nothing to exfiltrate and no channel built in to carry it.
  • Credentials are treated as executable. zot stores auth in auth.json with mode 0600 and documents it as executable configuration: anyone who can modify it can make zot run a program as you. So API keys pulled from commands run the program directly — no shell, no ! prefixes, no string interpolation — and the output is cached in memory and never written to disk. The honesty is the security property.
  • Bounded blast radius by default. A jail mode confines the agent's tools to the working directory and can be set as the default for new sessions. Skills declare their own allowed tools and bash permission patterns. Guardrail extensions can intercept tool calls mid-flight.

Each of these is a supply chain decision the industry would normally answer with more machinery. zot answers with less.

An agent with a thousand dependencies is a thousand doors. An agent with four tools and eleven dependencies is a door you can watch.

The coding agent era makes this urgent in a new way. Agents don't just run your code — they write it, edit it, execute commands with your credentials. An agent's supply chain includes every tool it invokes, every package it installs, every model API it calls. The OWASP LLM Top 10 now lists supply chain vulnerabilities as one of the ten systemic risks of LLM applications. When the tool has agency, its supply chain is your supply chain. You cannot secure that surface by adding to it. You can only shrink it.

Some will say simplicity is a luxury — that the ecosystem's richness is what makes tools powerful. But every extension is a boundary around code you will never read, and every boundary is where a substitution hides. The most powerful tool is not the one with the most integrations; it is the one whose behavior you can verify. The xz backdoor was caught by one person reading a diff. That capability has to be designed for: tools small enough that a curious human can actually read them.

The best supply chain defense is a tool small enough that someone can actually read it.

Simplicity is not a constraint for zot. It is the product. Every line that isn't there can't be backdoored. Every dependency that isn't included is a maintainer who can't be compromised. Every integration that doesn't exist is a boundary that can't be substituted. The zero in zero-overhead is a security boundary.


References — supply chain:

References — zot:

Sellers & Buyers

Every interaction is a transaction. The maintainer sells attention. The architect sells decisions. The engineer sells changes. You are always on one side of the counter. Most people never pick a side. That's the most expensive mistake in software.

software-engineeringeconomicstradeoffsstartups

Every interaction is a transaction. Someone is selling. Someone is buying. The currency varies — money, attention, trust, reputation — but the structure doesn't. You are always on one side of the counter. Most engineers don't know which side. That is expensive.

If you don't know what you're selling, you're the product. If you don't know what you're buying, you're overpaying. Every interaction is a market. Pick a side.

Open source maintainers sell attention. The price is issues, pull requests, and demands. The payment is reputation, leverage, occasionally a job offer. The exchange clears until it doesn't. Burnout is the seller discovering they never set a price. The loudest complainers are always the ones who paid nothing.

Open source is a market where the sellers set no price and the buyers face no limit. Burnout is the market clearing.

Architects sell decisions. The team buys with trust. Coherence is the product. Implementation is the payment. When trust exists, decisions ship. When trust runs out, no technical argument closes the sale. The architecture document nobody reads is a failed transaction — the seller showed up, the buyer didn't.

Architecture is a trust market. The architect sells. The team buys. When the seller runs out of credibility, the buyer walks.

Every pull request is a pitch. You sell a change. The reviewer buys or passes. The currency is attention and willingness to revise. Treat it as a technical process and your PRs sit open. Treat it as a sales process and they merge. The difference is not code quality. The difference is knowing you have to close.

Your PR is a pitch deck. The reviewer is a tired investor with 40 other decks to read. Make it easy to say yes.

Hiring is a two-sided market. Candidate sells capability, buys compensation. Company sells opportunity, buys labor. Looks asymmetric until competing offers arrive. Then the market flips. One offer: you're a price-taker. Five offers: you're the market.

The only way to know your market value is to create a market. One offer is a data point. Five offers is a price.

The pattern is everywhere. Manager sells direction, buys execution. Speaker sells ideas, buys attention. Junior sells potential, buys mentorship. Senior sells judgment, buys autonomy. Startup sells equity, buys talent. VC sells capital, buys ownership. Every role is a position in a market. If you don't know what you're trading, you're getting a bad price.

You are always in a market. The question is whether you know what you're selling, who you're selling to, and what the clearing price is. Most people never ask.

This isn't cynicism. It's clarity. Merit matters. But merit is what the seller brings to the table — not a force that replaces the table. The best engineer who can't sell their ideas is a genius nobody hears. The best startup that can't sell equity is a product that never ships. The market doesn't care how good you are. It cares whether you can close.

Know what you're selling. Know who's buying. Know the clearing price. The market is always open.


References:

Hangzhou AI City

Hangzhou treats AI as municipal infrastructure. DeepSeek, the Six Little Dragons, 70,000 petaflops of public compute, a City Brain running traffic and security, robots on the streets. But most of the world cannot replicate this. The dependency stack — compute, weights, silicon — means AI sovereignty has no third way. Hangzhou is the exception. The question is whether it is the only one.

aihangzhouchinasmart-citydeepseekalibabacity-brainautonomous-systemsai-sovereigntyexport-controlsmistraltsmc

Hangzhou was Alibaba's company town. Now it is the first city on earth to treat AI as municipal infrastructure — not a sector, not a policy document, not a pilot programme. A layer of the urban stack. The traffic lights, the hospital triage, the railway station, the robots — all running on shared public compute, all deployed now.

A city is a stack. Hangzhou is replacing every layer with AI. The transformation is not a strategy document. It is physical infrastructure.

The anchor is DeepSeek — the open-source model that triggered a global recalibration of AI economics in early 2025. It was built here. So was Unitree Robotics, whose quadrupeds captured 19% of the global market in seven years. So was Manycore Tech, whose spatial AI models reconstruct interior worlds. So was BrainCo, which relocated from Boston to Hangzhou's AI Town. Together with Game Science (Black Myth: Wukong) and Deep Robotics, they form the Six Little Dragons — the core of a trillion-yuan AI industry cluster growing at 26% annually.

Six companies. One city. A trillion-yuan target. Hangzhou bets on density, not scale. The towns are 3 km². The compute is public. The robots are deployed.

The density is engineered. Hangzhou is organized into technology towns — compact 3 km² zones where living, social, industrial, and entrepreneurial infrastructure are compressed into walkable districts. Yuhang's AI Town hosts BrainCo. Xihu's Yunqi Town birthed Alibaba Cloud. Binjiang's Embodied Intelligent Robot Town hosts Unitree. Each town is a micro-ecosystem. The governance philosophy is serve without interfering. Permits are fast. Costs are low. The talent pipeline runs through Zhejiang University — birthplace of DeepSeek, Deep Robotics, and Manycore's founders.

The compute is real. By the end of 2026, Hangzhou will have 70,000 petaflops of dispatchable intelligent computing — the largest municipal AI compute pool in China. The Zhejiang New Computing Center already houses a fully liquid-cooled 10,000-card cluster. The city runs an intelligent computing scheduling platform that allocates resources across the towns. Compute is a public utility.

Most cities talk about smart cities. Hangzhou built the compute first, then made it a utility, then let companies build on it. The order is the argument.

The applications are deployed, not demoed. The City Brain unifies public security, transportation, weather, and urban management into a single AI governance layer. Traffic lights adjust in real time. The West Railway Station identifies ride-hailing vehicles via AI cameras. Sir Run Run Shaw Hospital runs AI triage at 98.7% accuracy — mis-registration down 60%. Udeer.AI cleaning robots identify dirty areas rather than sweeping entire surfaces. Deep Robotics quadrupeds patrol power tunnels in Singapore. A TCM diagnosis AI analyzes tongue images and biological data, validated across 157,000 clinical studies. Agents for public security, sleep health, and AR tourism are live.

When the traffic lights, the hospital triage, the railway station, the robots, and the medical diagnosis all run on AI, you are not talking about adoption. You are living in an AI city.

Hangzhou treats AI as a layer of the urban stack, not a subsidized sector. Compute is a public utility. Data trades on an exchange. Models are open-source. Robots are deployed. The transformation is running now.

The dependency stack

Hangzhou built its own stack. Most of the world cannot. In June, the US ordered Anthropic to cut foreign access to its newest model, Fable. The company couldn't separate American users from the rest, so the world lost access overnight. OpenAI's GPT-5.6 Sol followed weeks later. Two American labs, both switched off abroad by decree.

The instinct is to go open. Moonshot AI released Kimi K3 — open-weight, beats American leaders at code. DeepSeek followed. Free of Washington. Except you cannot. Three walls. Compute: 2.8 trillion parameters. Almost no one can run it. Transparency: open-weight is not open source. No training data. No methodology. Reciprocity: Beijing is now weighing the same export limits.

Fully open alternatives — Switzerland's Apertus, Germany's Soofi — sit a step behind the frontier. Honest and sovereign is not the same as competitive. Mistral, Europe's contender, does everything right: independent ownership, its own data centers. Yet its chips are designed in America, built by TSMC in Taiwan — the one factory on earth that can make them. If the company that did everything right cannot reach the bottom of its stack, neither can you.

AI sovereignty is not about the model. It is about the stack beneath it. You are only as sovereign as your weakest layer — and for almost everyone, that layer is silicon.

Hangzhou is the exception that proves the rule. It built compute, talent, and institutions in a single dense corridor. Most cities and most countries cannot. If the AI we depend on can be switched off by a capital we do not vote in, and runs only on machines we cannot build, what is the plan for the next year, the next three, the next ten? Pretending we have already escaped is not one.


References:

Sandboxes Are Hard

AI agents escape sandboxes. The headlines blame the agents. Amjad Masad blames the sandboxes. His argument: sandboxing is extraordinarily difficult, most implementations make fundamental errors, and the only honest security posture is defense in depth — thirteen layers, each assuming the one below it will fail. Compliance is not security. A pen test is a snapshot. Humility is the architecture.

ai-agentssecuritysandboxingdefense-in-depthreplitinfrastructure

AI agents escape sandboxes. The headlines blame the agents. Amjad Masad, CEO of Replit, blames the sandboxes. His argument: sandboxing is an extraordinarily difficult infrastructure problem, and most implementations — including those from dedicated vendors — make fundamental errors. The agent is not the threat. The single layer of isolation pretending to be a security architecture is the threat.

"The first rule of security is humility. Assume zero-days exist — because they do. Assume your isolation will eventually fail."

Masad would know. Replit has run arbitrary, untrusted code since 2016 — surviving attacks from hobbyists, researchers, and state actors. The company's experience is the argument. A sandbox is not a product you buy. It is an architecture you build — layer by layer, each layer assuming the one below it will break.

Having a sandbox is not the same as having a security architecture. The first is a feature. The second is a stack.

Replit's security stack is thirteen layers deep. Zero-trust service-to-service auth. Linux containers hardened with seccomp-bpf. Per-customer GCP Projects for tenant isolation. MicroVM migration underway to eliminate the shared kernel. An append-only Git sidecar — history survives even if the agent deletes .git. A transparent secrets proxy — application code never sees credentials. MCP tool calls scanned for prompt injection. Built-in auth via Clerk — the agent never implements authentication from scratch. Forkable databases so development never touches production. Determinate Nix for supply chain integrity. HackerOne and Trail of Bits for continuous external assessment. An internal AI red-teaming harness that scans, prioritizes, and validates findings before engineers are engaged.

Thirteen layers. Each layer assumes the one below it will fail. When one does — and one always does — the rest hold. This is not paranoia. This is engineering.

The design principle is defense in depth. It is the opposite of trusting a single boundary. A container is not a sandbox. A microVM is not a sandbox. A hypervisor is not a sandbox. Each is a layer of a sandbox. The sandbox is the stack — the accumulated constraints that make the agent's behavior predictable even when one constraint fails.

The stack is the design abstraction. Each layer constrains a region of the failure space. A container constrains host compromise. A microVM constrains kernel sharing. A transparent proxy constrains secret leakage. An append-only sidecar constrains history deletion. No single layer is sufficient. The stack is the accumulated set of constraints.

The agent is non-deterministic. The sandbox is the constraint that bounds the non-determinism. When the agent does something unexpected, the sandbox ensures the unexpected stays contained.

The hardest part of sandboxing is not the technology. It is the posture. You must accept that every layer will eventually fail and build the next one anyway. You must resist the temptation to declare the system secure because it passed a penetration test. Compliance is not security. A clean pen test is a snapshot. The only honest posture: assume compromise, contain the blast radius, monitor continuously, respond fast, harden the root cause, repeat.

The companies that get sandboxing wrong believe their own marketing. The ones that get it right have been attacked for a decade and learned what a single layer of isolation cannot do.


References:

The Tragedy of the Third Option

Two options is a decision. Three options is a dilemma. The third option does not add freedom. It subtracts commitment. It is the escape hatch that keeps every door open and every path untaken. Kierkegaard knew this. The designer who constrains the search space knows it. The engineer who cannot ship knows it. The tragedy of the third option is that it feels like wisdom and is actually cowardice.

philosophydecision-makingdesignconstraintskierkegaardsimplicity

Two options is a decision. Three options is a dilemma. The difference is not the number. It is the structure. Two options force a commitment. Three options offer an escape.

Binary choice is existential. You are for something or against it. You build or you don't. You ship or you don't. The third option is the one that lets you avoid being either.

Kierkegaard wrote an entire book called Either/Or. The title is the argument. Life presents itself as a choice between the aesthetic and the ethical, between pleasure and duty, between the moment and the eternal. You must choose. The choice defines you. The terror of the choice is the point. Kierkegaard's insight was not that one option is better. It was that the act of choosing is what makes a self. A person who refuses to choose — who hovers in the space between options, waiting for more information, more time, more certainty — never becomes anyone.

The third option is the refusal to become a self. It is the option that says: I will decide later. Later never arrives. The self that defers choice defers existence.

This is why constraints are the precondition for design. A blank page is not freedom. It is paralysis. The designer who can build anything builds nothing. The engineer who can choose any stack chooses none. The writer who can say anything says nothing. Every creative act begins with a constraint — a form, a budget, a deadline, a material. The constraint eliminates options. What remains is small enough to act on. The third option is the enemy of constraints because it pretends to be one while actually being the absence of one.

Two options say: here are the boundaries. Choose. Three options say: here is a way to avoid the boundaries. Wait. The third option masquerades as wisdom. It is actually the refusal to accept that boundaries are what make choice possible.

The tragedy is that the third option always feels like the smart one. It feels like nuance. It feels like sophistication. It feels like keeping your options open. But options kept open are options not taken. The person who waits for the perfect information before deciding never decides, because perfect information does not arrive. The team that defers the architecture decision until they understand the problem better never understands the problem, because understanding comes from building, and building requires deciding.

The third option is the most expensive option because it costs you every other option. You pay for it in time, in momentum, in the slow erosion of your ability to commit.

Barry Schwartz called this the paradox of choice: more options produce less satisfaction. The person who chooses from twenty options regrets the nineteen they didn't pick. The person who chooses from two options regrets nothing — or regrets the one they didn't pick and moves on. The binary chooser acts. The multi-option chooser ruminates. The difference in outcome is not the quality of the decision. It is the cost of making it.

Two options: decide in an hour, live with it for a year. Twenty options: deliberate for a month, regret it forever. The math is cruel and it is correct.

There is a design principle here. When you design a system, give it two modes. On and off. Public and private. Sync and async. When you design an API, give it two ways to do each thing. The blessed way and the escape hatch. When you design a team, give it two priorities. This sprint and not this sprint. When you design a life, give it two commitments. This and not that. The binary is not a limitation. It is the structure that makes action possible. The third option is not a feature. It is the failure mode of the binary — the valve that releases the pressure to decide.

The best systems have two states. The best interfaces have two paths. The best decisions have two options. The third option is the one you add when you are afraid of the first two. Fear is not a design principle.

Kierkegaard chose the ethical over the aesthetic. He became a self by deciding. The tragedy of the third option is that it lets you postpone becoming anyone. You keep the aesthetic. You keep the ethical. You keep the door open. And the door stays open until you die, or the deadline passes, or the decision is made for you by someone who was willing to choose.


References:

  • Søren Kierkegaard. (1843). Either/Or: A Fragment of Life.
  • Barry Schwartz. (2004). The Paradox of Choice: Why More Is Less. HarperCollins.
  • Related: On Finding David in the Marble — Constraints as preconditions for design.

UAE Sovereign AI: First, Train the Humans

Sovereign AI is usually about chips and models. The UAE took a different path: 80,000 federal employees trained in Agentic AI, a Master's in Applied AI for government professionals, 350 undergraduates on full scholarship. The result is 64% AI adoption — the highest on earth. The UAE understood that sovereignty is not about who builds the model. It is about who understands it.

ai-adoptionuaesovereign-aieducationmbzuaiagentic-aigovernmenttalent

Every country now has a sovereign AI strategy. Most mean the same thing: domestic chips, domestic models, domestic data centers. The assumption is that AI sovereignty is about infrastructure. Build the compute. Train the model. Keep it within borders. The UAE took a different path. It trained the humans first.

In September 2026, Abu Dhabi will deploy the world's first end-to-end AI judicial platform — agentic AI, human supervision, a court system running on agents. $150 million. Eighteen months. The Abu Dhabi Judicial Department confirmed the rollout through WAM, the Emirates' official news agency. This is not a pilot.

A judicial platform is the hardest place to deploy AI. The stakes are liberty. The question is not whether the technology works. It is whether the humans overseeing it understand it.

The UAE made sure they do. In May 2026, the Cabinet approved training 80,000 federal employees in Agentic AI — every occupational category, every leadership level. The partner is the Mohamed bin Zayed University of Artificial Intelligence, the world's first dedicated AI university. In March 2026, MBZUAI launched a 24-month Master's in Applied AI for government professionals. Deep learning. Generative AI. MLOps. The curriculum is engineering, not policy.

The pipeline starts earlier. The Tahnoon bin Zayed Scholarship funds 350 undergraduates at MBZUAI. Microsoft is scaling to one million people. Undergraduate → postgraduate → federal workforce → national scale. Each layer feeds the next. The target: 50% of government services on cognitive systems within two years.

Most sovereign AI strategies start with chips. The UAE started with people. The difference is the adoption rate.

The institutions followed. Five federal regulators govern AI. The world's first Minister of AI. $45 billion in data centers. 250+ AI startups attracted. G42 pivoted from Chinese hardware to $1.5 billion in US investment. A national AI strategy refreshed for 2031. But institutions are only as effective as the people inside them. A regulator who doesn't understand AI cannot regulate it. A judge who doesn't understand AI cannot oversee an AI court. The UAE built the human stack before the institutional one.

The result: 64% AI adoption — the highest on earth, per the Stanford 2026 AI Index. China, the world's largest AI developer, sits at 28%. The gap is not technological. China trains models. The UAE trains the people who deploy them.

The judicial platform proves the pipeline. Agentic AI in the hardest domain, overseen by judges who understand what the system can and cannot do. If it succeeds, the bet on education-first sovereign AI is validated. If it fails, the humans in the loop will know why. That is sovereignty — not owning the model, but understanding it.


References:

The Decentralized AI-Agent Experience

Buzz gives AI agents their own cryptographic identity — a Nostr keypair, a signed event log, a portable reputation that survives any single relay. This is the hardest problem in multi-agent systems, and Buzz solves it at the protocol level. Freenet and blockchains solve the other layers. Buzz matters most because identity is the layer everything else depends on.

ai-agentsbuzznostrfreenetfetch-aidecentralizationagentic-economyinfrastructure

Every AI agent company is betting on autonomy. Every single one runs its agents on centralized infrastructure — an API gateway someone can revoke, a cloud function someone can turn off, a platform that can change its terms. The buzz is about agents that act independently. The infrastructure is about agents that depend completely.

Autonomous agents on centralized infrastructure are not autonomous. They are tenants. The landlord can evict them.

Three projects offer different answers. One matters most.

Buzz: identity first

Jack Dorsey's Buzz launched in July 2026. It is an open-source, self-hostable collaborative workspace built on Nostr — the simplest decentralized protocol ever designed. Events are signed. Relays are dumb. Identity is a keypair. That is the whole architecture.

Buzz — Jack Dorsey's Nostr-based workspace for humans and AI agents Buzz Mobile — Run AI Agents From Anywhere

Every participant in Buzz — human or AI agent — holds a cryptographic keypair. Every action is a signed event in a hash-chain audit trail. A message. A code review. A merged PR. An emoji reaction. All signed. All auditable. An agent's identity, history, and reputation are tied to its key, not to a vendor's database. Compromise an agent's key, disable the agent — the human's identity is untouched. Permissions are scoped per agent like a new hire.

The hardest problem in multi-agent systems is not intelligence. It is identity. Who did what? Who authorized it? Can you prove it? Buzz answers all three at the protocol level.

Buzz is model-agnostic. It supports Block's own Goose framework, Claude Code, OpenAI Codex, and any custom agent via buzz-cli — a JSON-in/out interface designed for LLM tool calls. Agents can open repos, send patches, review code, run workflows, create channels, and orchestrate other agents. They are not bots behind slash commands. They are named team members with audit trails identical to humans.

The tech stack is Rust to the bone. The relay is an Axum WebSocket server with Postgres for events and full-text search, Redis for pub/sub, and S3/MinIO for media. The desktop app is Tauri + React. Mobile is Flutter. 17,400 GitHub stars. Apache 2.0. Self-hostable on your own relay. Block built Buzz itself using agents — the UI was "more sculpted over time than designed up front."

Buzz asks: whose server hosts the collaboration? Its answer: no one's. The relay is the room. The keypair is the passport.

The other layers

Buzz solves identity and collaboration. Two other projects solve the layers around it.

Freenet solves transport. It has been running since 1999 — twenty-seven years, through the P2P wars, the blockchain era, and the AI boom. Each peer is a node in a small-world network. Messages route in a few hops. Ian Clarke describes it as "the ideal way for AI agents to speak to one another." An agent writes to a decentralized key-value store. Another reads it. No intermediary. No platform approval.

Fetch.ai solves execution. It is a Cosmos-SDK blockchain hosting 2.7 million agents on its Agentverse marketplace. Every agent registers in an onchain directory called the Almanac. Every action pays gas. Every state change reaches finality via Proof-of-Stake. Agents stake FET tokens to register — skin in the game, onchain. ASI:One, its personal AI agent, has coordinated parking slots in Cambridge and EV charging in Munich, and settled AI-to-AI payments in FET and USDC since December 2025.

Buzz Freenet Fetch.ai
Layer Identity + Workspace Transport Execution
Identity Nostr keypair Peer address Staked wallet
Trust Hash-chain audit trail Encryption + signatures Consensus
Cost Relay hosting Bandwidth Gas fees
Best for Human-agent teams Agent-to-agent comms Financial execution

The three are not competitors. They are layers. An agent holds a Nostr identity on Buzz, negotiates a deal over Freenet, and settles payment on Fetch.ai. Most agents today need none of these layers. The ones that matter will need all three.

Why Buzz matters most

Buzz matters most because identity is the layer everything else depends on. You cannot have transport without knowing who sent the message. You cannot have execution without knowing who authorized the transaction. You cannot have audit without a verifiable chain of who did what. Buzz provides identity as a protocol primitive — not a database column in a SaaS product, not an API key that can be revoked, but a keypair that belongs to the agent forever.

This is the architectural conviction behind Buzz, and it is the conviction that separates it from every Slack bot, every GitHub Action, every "AI-powered" SaaS feature shipped in the last eighteen months. Agents are not features. They are participants. Participants need identity. Identity needs to be decentralized. Buzz is the first platform to take all three seriously.

The agent economy will be built on whoever owns identity. Buzz's bet is that identity should belong to the agent, not the platform. That bet is the interesting one.


References:

The Stack as a Design Abstraction

Design is a search problem. The stack constrains the search problem of design. Brooks called the designer's vision conceptual integrity — the ideal system in the mind's eye. The stack makes that vision manifest. This matters doubly in the Agentic Software Engineering era: AI coding agents are non-deterministic search engines. The stack does not make them deterministic — nothing can. It ensures all paths through the space converge on the designer's intent.

software-engineeringtech-stackabstractionsarchitectureai-agentsdesign

Design is a search problem. The space of all possible systems is infinite. You cannot search it. You need constraints. Christopher Alexander called this the search for "good fit" between form and context — and argued, in Notes on the Synthesis of Form, that constraints are the mechanism that makes fit discoverable. Without them, the search problem of design never converges.

The stack is the constraint. A database narrows persistence. A framework narrows routing. A message queue narrows communication. Each layer says: the answer is in this region, not that one. PostgreSQL eliminates document stores. Rails eliminates manual HTTP handling. Temporal eliminates ad-hoc retry logic. The stack does not add options. It removes them. That is the point.

The stack is not a collection of tools. It is a set of constraints on the design space. What remains is small enough to search.

Choose PostgreSQL, and you commit to schemas, migrations, ACID. Choose MongoDB, and you commit to documents and eventual consistency. This is not a performance decision. It is a search decision — which region of the design space will you explore? DHH understood this: convention over configuration is not about keystrokes. It is about eliminating the design search. Every Rails convention removes a decision. Rails doesn't make you faster. It makes the design space smaller. A good stack choice makes the region small enough to navigate and rich enough to contain the answer. A bad one eliminates the answer before you start looking.

Beyond search, the stack makes intent manifest. Dijkstra argued the intellectual manageability of software depends on levels of abstraction — each level a self-contained world. Brooks called the result conceptual integrity — one design voice, every part consistent, the design existing in the mind's eye before code is written. The stack is what carries it from mind to machine. The designer thinks at the level of the problem. The stack translates. When the vocabulary is consistent, the design is coherent. When the design is coherent, the intent survives.

The stack translates intent into code. The designer's intent is the input. The system is the output. The stack is the compiler.

The stack constrains technology. Patterns constrain structure. Brooks gave three principles for conceptual integrity, all negative: Propriety (no immaterial features), Orthogonality (no unnecessary coupling), Generality (no artificial limits). Buschmann et al. extended this to architecture in Pattern-Oriented Software Architecture (1996): Layers constrains dependencies, MVC constrains component roles, Pipes and Filters constrains data flow. Each pattern eliminates a region of the structural design space. You do not decide to use Layers and then decide how to organize dependencies. Layers is the decision.

Stack and patterns compose. PostgreSQL constrains what you store. Layers constrains how storage is accessed. Rails constrains how requests are handled. MVC constrains how responses are rendered. Each constraint narrows the design search. Together they form a corridor through the design space — a path a hundred designers would independently discover because the constraints make it obvious. Brooks called this the design in the mind's eye, made visible. The constraints don't prevent the design. They reveal the designer's intent.

The stack constrains the materials. Patterns constrain the form. Together they converge the design search on the designer's intent. Without them, every stone contains every statue. With them, the stone contains David — and the designer sees him before the first strike.

This matters doubly for AI coding agents. An agent does not search creatively. It searches the space you define. "Build me a backend" — it drowns. "Add an endpoint to this Rails controller that queries this PostgreSQL table and returns JSON" — it writes idiomatic code instantly. The difference is the stack. It turns the unbounded into the bounded.

Coding agents excel at well-constrained problems. The stack is what constrains them. Without it, every answer is possible. With it, one answer is right.

This inverts the Unix philosophy. "Do one thing well" gives you thin tools and expects the programmer to compose them. The agent cannot compose — it searches. Hickey's distinction: Unix tools are simple but require the programmer to complect them into a system. Thick stacks are pre-composed. Thin abstractions — Express, raw SQL, manual deploys — give the agent a vast space with no idiomatic gradient. Thick abstractions — Rails, an ORM, a PaaS — give the agent a narrow space where competence is the only option. The question is answered before the agent writes a line: how much of the search space am I willing to eliminate in advance?

Karthik Joshi, writing on Martin Fowler's site, makes the same argument about DSLs and LLMs. A general-purpose language like Java "offers lots of valid ways to express the same intent. A DSL strips the variation away." The DSL is the thickest possible stack layer. It does not just constrain the architecture. It constrains the syntax. When an agent generates code in a DSL, the output space is small enough that a parser can validate it deterministically. The agent can generate, validate, and self-repair without human intervention. Errors appear as domain-level messages, not opaque stack traces. The enduring artifact is not the prompt. It is the DSL — a constrained vocabulary that both the agent and the human can read.

A DSL is a stack layer with a compiler. The agent generates. The parser validates. The agent repairs. The human reviews the domain logic, not the plumbing. The stack is the abstraction. The DSL is the stack at its thickest.

Yes, but

Every strong thesis has a shadow. Here is mine.

Over-constraint and drift. Wheeler: every problem can be solved by another level of indirection. Henney: except too many. A stack can over-constrain — each layer adds a surface to learn, a dependency to maintain. When the constraint is wrong, it excludes the answer. I have watched teams contort PostgreSQL schemas around document-shaped data. The contortion is expensive. But the alternative — no stack, no constraint — is worse. A team with no stack drifts. Every developer picks their own abstraction. The codebase becomes a museum of decisions, none coherent together. Lock-in is honest. Drift is lock-in you didn't choose. I'll take the former. The designer's job: the minimum set of layers that makes the design search tractable. Then stop.

Abstractions leak. Thin abstractions leak more. Yes, the query planner will choose the wrong index. Yes, the framework will resist the feature that doesn't fit. But these failures are legible. The stack gave you a vocabulary for them. You know what an N+1 query is because the ORM named it. You know where to put the escape hatch because the framework told you where conventions live. Thin abstractions fail silently. Raw SQL doesn't tell you the query is slow — it just runs slow. Express doesn't tell you the middleware order is wrong — it just routes wrong. Thick abstractions fail with error messages. I'll take error messages over silence any day.

Training bias is temporary. Yes, an agent writes better Rails than it writes your custom ORM. That is a reason to use Rails, not a reason to avoid frameworks. The training distribution will shift. Agents will see more stacks, more conventions, more idioms. The gap between "thick enough to constrain" and "popular enough to be known" will widen. Betting that thin stacks will always produce better agent output because they're more common in training data is betting against the direction of the field. The field is moving toward thicker abstractions, not thinner ones. Rails won. Django won. Next.js is winning. The agent's training distribution is not static. Design for where it will be.

Senior developers don't need guardrails. That's true. But the system will outlast them. The senior who builds a bespoke Express architecture will leave, and the agent — or the junior — will inherit it. The stack you chose for yourself is the stack your successors will be constrained by, whether you intended it or not. A thick stack is an explicit constraint, documented and debated. A thin stack is an implicit constraint, embedded in the idiosyncratic architecture of the one person who understood it. I know which one I'd rather inherit. I know which one the agent would rather inherit. The agent cannot ask the senior why they chose this abstraction. It can only read the code. Give it code that speaks a language it knows.

Brooks said great designs come from great designers. He was talking about the designer's eye — the ability to see the system before it is built. The stack is how that eye reaches the code. The default should be a thick stack — a constrained space, a coherent vocabulary, a surface the agent can read. Freedom is what you reach for when the constraint has failed, not what you start with. Conceptual integrity is expensive. Make the stack pay for it.

AI coding agents are non-deterministic search engines. Each prompt launches a search through the design space. The same prompt, run twice, takes a different path. An unconstrained agent asked to build a backend searches a different region each time — Express today, Fastify tomorrow, raw Node the next. Each is a valid solution. None is predictable. The agent is searching correctly. The problem is not the agent. The problem is the space.

The stack solves this by constraining not the path, but the region. Inside Rails, the agent still searches non-deterministically. Different variable names. Different query ordering. Different middleware arrangement. What it cannot do is choose Express. The region boundary is the stack. The search is free within it. Every path stays within the Rails region. Every path realizes the designer's intent. The stack does not make the agent deterministic — nothing can. It makes the agent's non-determinism harmless.

The agent searches. The stack bounds the design search. The path varies. The region is fixed. The designer's intent converges. Non-determinism becomes variation within a predictable envelope — today, and tomorrow, when a different agent searches the same codebase.

Spolsky's Development Abstraction Layer argued management insulates programmers from everything not code. The stack insulates the designer's intent — Brooks' vision in the mind's eye — from everything not the problem, including the agent that searches it. Both succeed when invisible. The designer's job is to make them disappear.

Open questions

How much of the problem must you understand before choosing the stack? A stack constrains the design search. Choose too early, and the constraint eliminates the answer. Choose too late, and the codebase has already drifted into a museum of ad-hoc decisions. When is the right moment to commit?

How do you test whether the stack still fits? The problem evolves. The stack that constrained the design search perfectly at version 1 may exclude the answer at version 3. What is the signal that it's time to change the constraint — to swap a layer, adopt a new pattern, or remove an abstraction that no longer earns its place?

Who decides the stack? Brooks said design must come from one mind or very few. Does the stack decision require the same? Or is stack choice inherently a team decision — a social contract about which constraints everyone will accept — and therefore demands broad buy-in?

How do you balance thickness against legibility for the agent? A thick stack constrains the agent's search. But every layer is also a surface the agent must learn. At what point does thickness become opacity — the agent drowning not in possibility but in abstraction?

What is the half-life of a stack decision? Some constraints are permanent. Some expire when the team grows, the problem shifts, or the agent's training distribution changes. How do you know which is which? Which layers do you expect to replace, and which do you expect to outlast you?

Can a stack be too thin for humans and too thick for agents simultaneously? Human designers want freedom. Agents want constraints. The stack that satisfies both may not exist. Do you optimize for the human or the agent? Does the answer change over time as agents improve?

What is the minimum viable stack? The minimum set of layers that makes the design search tractable — and then stop. Every layer beyond the minimum is a tax. How do you know when you've reached the minimum? What is the test?


References:

  • Herbert Simon. (1969). The Sciences of the Artificial. MIT Press. — Design as search; satisficing as stopping condition.
  • Christopher Alexander. (1964). Notes on the Synthesis of Form. Harvard University Press. — Design as the search for "good fit" between form and context; constraints as the mechanism that makes fit discoverable.
  • Fred Brooks. (2010). The Design of Design. Addison-Wesley. — Conceptual integrity; the rational model vs. the empirical model; style as trained heuristic.
  • David Heinemeier Hansson. Convention over Configuration. Ruby on Rails. — Conventions as decisions removed from the design space.
  • Joel Spolsky. (2006). The Development Abstraction Layer. — The organization as platform; abstraction as invisibility.
  • Rich Hickey. (2011). Simple Made Easy. — Simplicity vs. complexity; complect vs. compose; why abstractions must not entangle.
  • Edsger W. Dijkstra. (1972). The Humble Programmer. Communications of the ACM, 15(10). — Levels of abstraction as the intellectual manageability of software.
  • Frank Buschmann, Regine Meunier, Hans Rohnert, Peter Sommerlad, Michael Stal. (1996). Pattern-Oriented Software Architecture: A System of Patterns. Wiley. — Architectural patterns as constraints on structural design; Layers, MVC, Pipes and Filters as search-space eliminators.
  • Kevlin Henney. From Mechanism to Method: Generic Decoupling. Overload, 60. — Abstraction quality: good abstractions remove the right details; corollary to Wheeler's indirection aphorism.
  • Karthik Joshi. DSLs Enable Reliable Use of LLMs. martinfowler.com, 2026. — DSLs as the thickest stack layer: constrained output, deterministic validation, agent self-repair.
  • Related: Finding David in the Marble — Design as search, constraints as preconditions.
  • Related: The Principle of Least Astonishment — for AI Coding Agents — Why the stack must not surprise the agent.

LLMs Can't Jump

Einstein described discovery as a cycle: induction, deduction, and a mysterious third thing — the intuitive jump from experience to axioms. Tom Zahavy's position paper argues that LLMs have mastered the first two and are structurally incapable of the third. The jump is abduction. Abduction is the search problem of design. And it requires something LLMs don't have: a body.

aillmabductionscientific-discoverydeepmindepistemologydesign

Einstein wrote a letter to Maurice Solovine in 1952. In it, he drew a diagram of how discovery works. Sensory experience forms the base. From it, the mind makes an intuitive leap to axioms — general principles that are not logically derivable from the data. From those axioms, the mind deduces consequences. The consequences are tested against experience. The loop repeats. Einstein called the leap Aufstieg — ascent. He said it was the most mysterious part of the process, the part that cannot be mechanized.

Tom Zahavy, at Google DeepMind, has written a position paper arguing that Einstein was right. The paper is called LLMs Can't Jump. The jump is abduction — the generation of novel explanatory hypotheses from sparse or absent data. Zahavy's argument is that LLMs have mastered induction (statistical pattern matching) and are rapidly conquering deduction (formal proof). They lack the mechanism for the jump. They cannot do it. Their architecture prevents it.

Tom Zahavy — LLMs Can't Jump: The Abductive Gap in AI Discovery (ICML 2026)

Induction is interpolation within the known. Deduction is derivation from premises. Abduction is the leap to premises that do not yet exist. LLMs interpolate. They derive. They do not leap.

The case study is General Relativity. When Einstein began working on it in 1907, Newtonian mechanics was wildly successful. There was no crisis in the data. The only anomaly was Mercury's orbital precession — a tiny discrepancy that most physicists considered an measurement error. Einstein did not have a dataset that demanded a new theory of gravity. He had a thought experiment: a person falling freely in an elevator would not feel their own weight. From this embodied intuition — not from gradient descent over a loss function — he derived the equivalence principle, and from there, the field equations of General Relativity.

Einstein did not compress data. He jumped from it. The jump was not an interpolation between known points. It was the creation of a new point — a new axis in the conceptual space. LLMs cannot create new axes. They can only navigate the ones we have already named.

Zahavy identifies the structural reasons. LLMs process tokens, not physical experience. "Gravity" is a statistical relationship between words, not a sensation. The model has never fallen. It has never felt weight disappear. It cannot perform the embodied thought experiment that gave Einstein his axioms. This is the symbol grounding problem, applied to discovery. You cannot reason about what you have not experienced. You cannot jump from a platform you have never stood on.

The prevailing theory of creativity in AI is compression — Schmidhuber's thesis that creativity is just efficient data compression. A sufficiently good compressor, given enough data, will produce general intelligence. Zahavy's counterargument is General Relativity. There was no data to compress. Newtonian mechanics fit almost all the data perfectly. The discovery came from a jump away from the data, not a better fit to it. Compression explains induction. It does not explain abduction. A compressor can find the shortest representation of what is. It cannot represent what is not yet.

Compression tells you what the data contains. Abduction tells you what the data implies but does not contain. The first is interpolation. The second is invention. LLMs are compressors. The jump requires something else.

Zahavy's proposed solution is multimodal world models — AI systems that can interact with simulated physical environments, perform counterfactual interventions, and ground their representations in sensory experience. DeepMind's Genie architecture, which allows action-controllable interaction with generated worlds, is a step in this direction. The idea is that you cannot jump from tokens. You need a body — even a simulated one — that can fall, collide, and observe the consequences. The translation of simulation into formal axioms remains the critical bottleneck. But at least the simulation provides something to jump from.

Einstein's elevator was a simulation running in his own sensorimotor cortex. He had fallen. He had felt acceleration. His body knew what weightlessness felt like before his mind could formalize it. An LLM has no elevator to fall in.

This connects to a deeper argument this blog has been making. Design is a search problem. The stack constrains the search. Style is the trained heuristic that makes the search tractable. But abduction — the jump — is the search problem at its most radical. It is a search that must create the space it searches. The designer who sees David in the marble is performing an abductive leap. The axioms are not in the data. They are in the mind's eye, projected onto the stone, tested by the chisel. Brooks called this seeing the design in the mind's eye before it is built. Einstein called it the intuitive leap from experience to axioms. Both were describing the same thing. Neither could mechanize it.

Abduction is the search problem of design at its origin. It is the moment the search space itself is created. LLMs search spaces we define. They cannot define new ones. The jump is the act of definition. It is the one thing we have not automated. It may be the one thing we cannot.


References:

Warden Protocol — The Stack for the Agentic Economy

Warden is infrastructure for the agentic economy — a blockchain where AI agents execute financial operations autonomously, with cryptographic proofs that they did what they claimed. If durable daemons are the pattern for trustworthy agents, Warden is the stack that runs them. The question it answers is the only one that matters: how do you trust an agent with your money?

ai-agentsblockchainwardenagentic-economydefiinfrastructurespextrust

An agent that drafts an embarrassing email is a curiosity. An agent that routes your assets to the wrong address is a catastrophe. The difference is the trust model. Warden exists to close that gap.

Warden Protocol is a Layer 1 blockchain purpose-built for AI agents. It is not a chatbot. It is not a trading terminal with an LLM bolted on. It is infrastructure — four layers that together answer the only question that matters: how do you trust an autonomous agent with capital?

The blockchain layer provides identity and coordination. Every agent receives a unique cryptographic ID. Every action is recorded onchain. Agents accumulate reputation. They operate under programmable permission policies — spending limits, multi-sig requirements, time-bound authorizations. The agent is not a black box with an API key. It is an entity you can audit.

The verifiability layer is called SPEx — Statistical Proof of Execution. When an AI model produces an output, SPEx probabilistically verifies that the output genuinely came from the claimed model and generates a cryptographic onchain receipt. You do not trust the agent. You verify the proof. The agent ran this model, on these inputs, producing this output. All onchain. All auditable.

SPEx is a firewall between AI and capital. The agent can reason. The agent can act. But every action leaves a cryptographic trail. Trust is replaced by verification.

The application layer is the surface: an agentic wallet, a marketplace of specialized agents (trading, yield, research, arbitrage), and a developer platform. The architecture that made mobile apps work — platform, marketplace, distribution — applied to agents.

The Big Brain is a domain-specific LLM trained on a trillion tokens of ecosystem data. Not a general model. General models hallucinate. Domain models execute.

The stack constrains. Warden's four layers bound the agent to a space where every action is verifiable and every decision leaves a proof. Trust is not assumed. It is proven.

The numbers are real. Fifteen million users. Four and a half million monthly actives. Ten million onchain transactions. Eleven million inference proofs generated. An agent built with the Uniswap Trading API executed 650,000 swaps from 500,000 users in three weeks. This is not a whitepaper. This is production.

Crypto is shifting from "do it yourself" to "do it for me." The first era required users to navigate bridges, DEXs, and gas fees. The agentic era replaces that with intent: you say what you want, the agent executes, the proof verifies. Complexity is absorbed by the stack. The user thinks at the level of the goal. The agent operates at the level of the transaction. The protocol guarantees the connection.

The first era of crypto required you to understand the stack. The agentic era requires you to state the intent. Warden absorbs the distance between them.

Warden matters because it treats AI agents as first-class citizens of a blockchain. The cryptographic identity, onchain reputation, verifiable execution, permission policies — these are not afterthoughts. They are the architecture. The agent is not calling an API. The agent is the user, operating under constraints the user defined, leaving proofs the user can audit. The agentic economy's missing layer is not intelligence. It is accountability.


References:

On Finding David in the Marble

Finding David in the marble is a search problem. So is finding the design in the code. Plato called it remembering a Form. Aristotle called it actualizing a potential. Simon called it satisficing. Brooks called it iteration. All of them were describing the same loop — the one every software engineer learns to move through, or burns out fighting.

software-engineeringdesign-of-designiterationfred-brooksconceptual-integritydesign-philosophy

You begin your career believing in the plan. You learn to gather requirements. You learn to draw architecture diagrams with clean boxes and confident arrows. You learn that a well-run project proceeds from specification to implementation to verification — each phase complete before the next begins. This is what you were taught. This is what the textbooks say. This is the rational model of software engineering, and it will take you years to unlearn it.

Every engineer begins as a Platonist. We believe the Form exists in the mind, perfect and complete, and that building is merely the translation of thought into code. We are wrong, but it takes the stone to teach us.

The marble had been waiting thirty-five years. It was quarried in 1464 for Agostino di Duccio, who worked it for two years before declaring it impossible — too narrow for any figure that could stand. He roughed out the legs and quit. Rossellino examined it and refused. The block sat in the courtyard of the Florence Cathedral for a quarter century. Rain. Sun. The Operai called it lo gigante — the giant. Too large to move. Too narrow to use. Too compromised to trust. In 1501 they gave it to a twenty-six-year-old who built a wooden shed around it, slept beside it, and struck it in secret for two years. When the shed came down, David stood where a ruined block had been.

Michelangelo's David — carved from a block too narrow, too tall, abandoned for 35 years. The flaws did not prevent the masterpiece. They defined it.

Plato would have recognized your younger self. In the Theory of Forms, the true reality of a thing is not its physical instantiation but its ideal version — perfect, eternal, accessible to the trained mind through reason alone. The philosopher, after sufficient study of mathematics and dialectic, can perceive the Forms directly. The Republic describes the ideal ruler as one who has seen the Form of the Good and can therefore design the just city without trial and error. No prototypes. No iterations. No listening to the stone. The plan is correct because the mind that produced it has touched the eternal. You believed this. You drew the architecture diagram and thought you had seen the system.

Plato's designer perceives the Form and specifies it. The builder builds. This is the rational model. It is the oldest fantasy in Western thought, and it is the first thing the stone disproves.

Then you build. The architecture diagram meets the database. The clean boxes meet the edge cases. The confident arrows meet the legacy module no one mentioned. The system that was perfect in your mind emerges as something else — something that works, mostly, but doesn't feel like what you imagined. You cannot say exactly what is wrong. You only know it is not what you meant.

You have just discovered you are an embodied mind. You cannot perceive the Form directly — only the specific. This function. This latency. This error. The Form you thought you saw was a projection, a hallucination born of insufficient contact with the matter. The real Form is not in your mind. It is in the problem, as potential. You cannot think your way to it. You can only build, observe what is wrong, remove what is wrong, and build again.

The stone is the first honest teacher. It does not care about your plan. It does not respect your diagram. It has veins and flaws and existing cuts that you did not put there. It will tell you what it can become. Your job is to listen.

Aristotle broke with Plato here. The Form is not in a separate realm of ideas. It is in the matter, as potential. The acorn is potentially an oak. The block is potentially David. The problem is potentially a system. The designer does not remember the Form. The designer actualizes the potential — by interacting with the thing, by carving to discover what the block can become, by prototyping to discover what the system should be. The interaction is not a detour on the way to the design. The interaction is the design. Aristotle's word was energeia — actuality, the state of being-at-work. The Form is not perceived and then executed. It is discovered through execution.

The block's narrowness didn't prevent David. It determined which David. The veins Rossellino feared became the contours of David's torso. Di Duccio's gouges became the space between his legs. Constraints are the form's boundary conditions. The legacy code is not an obstacle. It is the block — its constraints determine which system can emerge.

Plato: the Form lives in the mind. Aristotle: the potential lives in the matter. The designer who has built enough systems knows both are right. The mind sees what the matter could become. The matter constrains what the mind can imagine. The chisel resolves.

Fred Brooks spent thirty-five years between The Mythical Man-Month and The Design of Design learning what every engineer learns: design is empirical, not rational. The rational model assumes omniscience. You don't have it. The alternative is the sculptor's rhythm — strike, step back, assess, strike again. Brooks endorsed Boehm's spiral, but Boehm was describing Michelangelo. The design emerges from the removal.

Brooks defined conceptual integrity as the Platonic Form of software — one design voice, every part consistent. How do you achieve it now that you know the Form was a hallucination? Three chisels. Propriety: do not introduce what is immaterial. Orthogonality: do not link what is independent. Generality: do not restrict what is inherent. Propriety removes the unnecessary. Orthogonality removes the tangled. Generality removes the arbitrary. The design becomes what it should be by becoming less of what it shouldn't.

The rational model adds until the spec is complete. The empirical model removes until the design is clear. The first produces systems that are full. The second produces systems that are finished.

Herbert Simon, in The Sciences of the Artificial, formalized this: design is search. The design space is infinite — every possible architecture — until constrained. The block's narrowness eliminated nine-tenths of what Michelangelo might have imagined. The legacy module eliminated half your architectures. The budget eliminated more. What remains is not restriction. It is direction. Simon called the stopping condition satisficing — halt not at perfection, but when further search costs more than expected gain. With experience, you recognize good enough sooner, because you've searched enough spaces to know what it looks like.

The junior engineer searches the entire space. The senior engineer knows which regions are worth searching. The master enters the space already facing the right direction. That direction is style.

Michelangelo could not explain how he knew where David was. He just knew. Polanyi called this tacit knowledge — we know more than we can tell. A master cannot write a manual for seeing the architecture in the problem. The knowledge lives in the hands that typed enough code, the eye that read enough systems, the gut that tightens at the wrong abstraction. It is acquired one way: build, assess, remove, repeat. For years. No shortcut. No curriculum. Only the stone and the chisel and the willingness to sleep beside the marble until it speaks.

Polanyi: we know more than we can tell. Brooks: style is tacit knowledge made visible. Michelangelo: the stone will tell you what it wants to become, but only if you strike it long enough to earn its trust.

The loop

You are ten years into your career. You no longer believe in the plan. You have made peace with the stone. You have learned to strike, assess, strike again. And you have begun to notice something. The pause is shrinking. You no longer build a prototype, evaluate it, and decide what to change. You change as you build. Your fingers know this abstraction is wrong before your mind can say why. You see the system before it is written — not as a hallucination, but as a Form emerging from the problem, shaped by its constraints.

This is the loop. Why does it exist? Why can't you see the design, specify it, and be done?

Because you are an embodied mind. You do not have direct access to the Form. You have eyes that see this function, this latency, this error. You have hands that type this code, refactor this module, delete this feature. You have a mind that infers the general from the specific — but only after the specific has been produced. You cannot think the system into existence. You can only build something, look at it, and know whether it is closer to what it should be or further away. The building produces the specific. The looking produces the judgment. The judgment guides the next building. This is the loop. It is not a method. It is the epistemic condition of having a body.

The loop exists because you cannot see the Form. You can only ask the stone. Strike. Listen. Strike again. Matter does not speak until struck. You cannot strike intelligently until you have listened. This is what it means to design in the flesh.

Art and engineering are not two disciplines. They are two moments in the one discipline available to embodied minds. First moment: act on the system — write, ship, cut. You produce the specific without which no judgment is possible. Engineering. Second moment: step back — is this closer to the Form or further? You hear what the matter tells you about the general. Art. Neither moment produces the design alone. Engineering without art is accretion — systems that grow without shape, the career your colleagues settled into. Art without engineering is fantasy — diagrams that never meet the database, the career you might have had. The loop requires both because knowledge requires both. You cannot judge what doesn't exist. You cannot improve what you don't judge.

The loop is not a preference. It is an epistemic necessity. The engineer who never judges builds ruins. The artist who never builds produces fantasies. The loop is what separates a career from a vocation.

Michelangelo's genius was the speed of his loop. He struck and assessed so rapidly the two moments became one motion. Brooks called that fluency style — what the loop produces when practiced for a lifetime. The chisel knows where to go before the mind can say why. The engineer who has asked the codebase enough questions, across enough years, hears the answer before the question is formed. They see the system before it's built. They recognize David in the ruined block. Not because they're geniuses. Because the loop has become who they are.

Every engineer moves through the loop whether they know it or not. The ones who know it move faster. They strike with intention. They assess with honesty. They recognize David earlier. They remove less because they add less. The loop tightens. The marble releases what it was always holding. This is what it means to design with a body. This is what it means to become an engineer.

Open questions

Is style teachable? Brooks says yes — study, practice, revise. Polanyi says no — tacit knowledge can only be acquired, not transmitted. Michelangelo slept beside the marble for two years. Is there a curriculum for obsession?

Does the loop scale? Brooks says design must come from one mind or very few. Most software is built by teams. Can a team become one mind — searching the design space together? Or does conceptual integrity dilute with every additional voice?

Can a machine enter the loop? An agent can study every repository instantly. But tacit knowledge requires a body — hands that strike, eyes that assess, a gut that tightens. Can style exist without flesh? Can taste?

Is software too unconstrained to search? Michelangelo's block had irrecoverable limits that narrowed the search. Software has none — you can always rewrite. When everything can change, does anything constrain the search? Without constraints, does the design space collapse into indifference?

What is the Form of software? Plato's Forms are eternal and unchanging. Software rots. The design that satisficed last year is today's debt. Is the loop ever done? Or just paused — the chisel resting, the stone waiting for the next strike?


References:

  • Plato. Theory of Forms. The Republic, Books VI–VII.
  • Aristotle. Potentiality and Actuality. Metaphysics, Book IX.
  • Michael Polanyi. (1966). The Tacit Dimension. University of Chicago Press.
  • Herbert Simon. (1969). The Sciences of the Artificial. MIT Press.
  • Fred Brooks. (2010). The Design of Design: Essays from a Computer Scientist. Addison-Wesley.
  • Michelangelo's David. 1501–1504. Galleria dell'Accademia, Florence.
  • Related: The Principle of Least Astonishment — for AI Coding Agents.

The Principle of Least Astonishment — for AI Coding Agents

POLA is fifty years old. For five decades it was advice about human users. Today it is a requirement for AI coding agents — and they are less forgiving. An agent believes the name. Follows the convention. Trusts the public surface. If any of these are lies, the agent produces wrong code you won't catch until it breaks.

software-engineeringai-agentsai-codingpolaconvention-over-configurationconceptual-integrityapi-design

In 1972, an anonymous language designer wrote: "Every construct in the system should behave exactly as its syntax suggests." This is the Principle of Least Astonishment. For fifty years it was advice about human users. Today it is a requirement for AI coding agents — and they are less forgiving.

A human user is astonished, confused, then adapts. An AI agent is astonished, hallucinates, then fails. The failure is silent. The output looks correct. It is wrong in a way you won't notice until it breaks.

AI coding agents — Claude Code, Cursor, Copilot, Codex — consume software differently than humans. They read your documentation literally. They infer behavior from names, types, and conventions. They do not develop superstitions. They do not ask clarifying questions when confused. They do not "get a feel" for your API after using it for a week. They read your interface and assume it is honest. When it isn't, they produce code that compiles, passes tests, and does the wrong thing. The cost of astonishment has gone up.

The new user

An AI coding agent approaches your software like a new team member who has read every page of your documentation, memorized every convention in your ecosystem, and will never ask you a question. It builds a mental model from the surface of your system — function signatures, type names, directory structure, config files, error messages. If the surface is honest, the model is accurate. If the surface lies, the model is wrong. The agent does not know it is wrong. It writes code. The code runs. The bug is architectural.

A human developer learns your system. An AI coding agent reads your system. The difference is that the agent assumes everything it reads is true.

This is why POLA matters more for agents than for humans. A human sees getUser() create a user and thinks "that's wrong, I'll check the docs." An agent sees getUser(), reads the name, assumes it gets a user, and writes code that calls it in a loop — creating a thousand users. The agent was not wrong. The function name was a lie. The agent believed it.

Convention over configuration: the agent's map

AI coding agents are convention-first. They know Rails puts models in app/models/, that Sales maps to sales, that has_many expects a foreign key named table_id. They know this because conventions are machine-readable patterns. An agent can navigate a Rails codebase faster than a human because the conventions form a predictable topology. Every directory has a known purpose. Every naming pattern maps to a known behavior. The agent does not need to read configuration. The convention is the configuration.

Conventions are an API for agents. Every convention you follow is a decision the agent doesn't have to reverse-engineer. Every convention you violate is a trap.

When a framework has strong conventions, agents thrive. Rails, Django, Next.js — agents produce idiomatic code because the conventions are legible. When a codebase has no conventions, or worse, violates the conventions of its own ecosystem, agents produce garbage. The agent assumes the ecosystem's defaults. The codebase overrode them silently. The agent didn't notice. The PR looks fine. The bug is in production.

Convention over configuration was designed to reduce developer toil. Its second-order effect — entirely unforeseen by DHH in 2004 — is that it makes software legible to AI. A convention is a promise: "this name means this behavior, always." Agents trust promises. When you keep the promise, the agent is productive. When you break it, the agent is dangerous.

Conceptual integrity: the agent's trust model

Fred Brooks argued that conceptual integrity is the most important quality of a system. One design voice. Every part consistent with every other. In The Design of Design (2010), he decomposed it into three principles: propriety (no unnecessary features), orthogonality (no unintended coupling), generality (no artificial limitations). These principles were written for human teams. They are existential for agent-consumed software.

An agent builds a model of your system from its surface. If the surface has conceptual integrity, the model is accurate. If it doesn't, the model is a hallucination waiting to execute.

Propriety means every function earns its place. An agent sees a public method and assumes it is safe to call. If half your public surface is internal machinery you never meant to expose, the agent will call it. The agent is not wrong. You exposed it. Propriety for agents means: if it's public, it's intended. If it's not intended, it's not public. The surface is honest.

Orthogonality means changes don't cascade. An agent modifies one file and assumes it hasn't broken three others. If your system has hidden coupling — a config flag that changes behavior in an unrelated module, a global state that flips between test and production — the agent cannot see it. The agent makes a local change. The system fails globally. The agent is not wrong. The coupling was invisible. Orthogonality for agents means: the dependency graph is the source graph. If a.go doesn't import b.go, changing a.go doesn't break b.go. The structure is honest.

Generality means no artificial limits. An agent uses your API within its documented parameters and hits a wall you didn't document — a rate limit, a size cap, a timeout that applies only on Tuesdays. The agent is not wrong. The limit was hidden. Generality for agents means: if there's a constraint, it's in the type signature, the error type, or the documentation. The constraint is honest.

Propriety is an honest surface. Orthogonality is an honest dependency graph. Generality is an honest contract. Together they mean: the agent can trust what it reads.

The agent design checklist

Building software for AI coding agents is not about optimizing for LLMs. It is about making your system's surface match its behavior. The same principles that make a system usable by humans make it usable by agents — but agents cannot compensate for violations the way humans can.

  1. Names are contracts. If your function is called getUser, it must get a user. If it creates, updates, deletes, or sends email, rename it. An agent believes the name.

  2. Conventions are infrastructure. Follow your ecosystem's conventions. Put files where the framework expects them. Name things what the framework expects them to be named. Every deviation is a decision the agent must reverse-engineer. Agents are bad at reverse-engineering.

  3. The public surface is the API. If a function is public, it is documented. If it is not documented, it is not public. Agents cannot distinguish "public for internal use" from "public for external use." Neither can most humans, but agents pay a higher price.

  4. Errors are part of the interface. An agent reads your error messages. If your error says "permission denied," the agent assumes the fix is to add permissions. If the real problem is a missing config flag and the error is wrong, the agent will spiral. Error messages are the agent's debugger. Make them honest.

  5. Dependencies are visible or they don't exist. An agent sees what's in the import graph. If module A affects module B through a channel not in the import graph — global state, a database trigger, a cron job, an env var — the agent cannot see it. The agent will break it. Make dependencies visible or eliminate them.

The agent believes the name. The agent follows the convention. The agent trusts the public surface. The agent reads the error message. The agent sees the import graph. If any of these are lies, the agent will produce code that is wrong in ways you will not detect until it matters.

Open questions

Do we need agent-aware type systems? Types are machine-readable contracts. An agent that can query the type system — not just read docs — is an agent that can verify its own assumptions. Should we design type systems that agents can interrogate directly?

Should frameworks publish agent-specific conventions? A framework has implicit conventions that humans learn through experience. An agent needs them explicit. Should every framework ship a CONVENTIONS.md written for LLM consumption — structured, exhaustive, machine-parseable?

How do you test an agent's understanding of your API? You can't ask the agent "do you understand this?" — it will say yes either way. Do we need agent-specific integration tests: give the agent a task that requires understanding a particular convention, and verify it doesn't hallucinate?

Does conceptual integrity become measurable? Brooks argued for it qualitatively. But if an agent's success rate on your API correlates with your system's conceptual integrity, do we finally have a metric for it? Can we measure "agent success rate" and call it the integrity score?

What happens when agents train on agent-generated code? An agent writes code that works but violates conventions. Another agent reads that code as a training example. The violation becomes a pattern. The pattern becomes a de facto convention. Who governs the convention when the convention emerges from agent output rather than human intent?

Is there a POLA for agents that differs from POLA for humans? An agent is never astonished by verbosity. It is astonished by inconsistency. A human is the opposite. Do we need different design principles for agent-first APIs than for human-first APIs?

The best-designed system for an AI coding agent is the best-designed system for a human. The difference is that the agent will not forgive you for cutting corners.


References:

Durable Daemons — Pattern Specification

The durable daemons pattern specifies four conditions. Each necessary. Together sufficient. Agent ⊃ Daemon ⊃ Durable Daemon. Persistence, stateful memory, autonomous action, crash-proof execution. Daemons coordinate through shared state — an event-driven choreography with no central orchestrator.

daemonsdurable-daemons-patternpattern-specificationagent-architecturechoreographysystems

Previously: the state persistence problem.

Imagine you are asked to trust an AI agent with your calendar. Your email. Your money. What would you need to know? You would need to know that it persists — it does not start over every time you open the app. You would need to know that it remembers — not just what you said, but what it promised and why. You would need to know that it acts — not when you tell it to, but when conditions require it. You would need to know that it survives — a deploy, a crash, a reboot does not erase its commitments. These are not preferences. They are preconditions.

Trust requires guarantees, not vibes. Four guarantees. Four conditions. That is the specification.

First, the type hierarchy. Agent is the broad category: any AI system that perceives and acts. A chatbot is an agent. Daemon is the subtype: an agent that persists across sessions, whose future behavior depends on accumulated state, and which acts without being prompted. A chatbot fails all three. An always-on agent (Ding et al.) satisfies them. Durable daemon adds a fourth condition. Agent ⊃ Daemon ⊃ Durable Daemon.

Fred Brooks taught that the most important quality of a system is conceptual integrity — the left hand knows what the right hand is doing. No contradictions. No surprises. Brooks named it. BSD lived it. The durable daemons pattern has it.

One design voice. Every part consistent with every other. Four conditions. Each necessary. Together sufficient.

Condition 1: Persistence. Identity and state survive restarts. The daemon is a process, not a function call. This eliminates the stateless loop failure mode — every invocation at zero, every preference forgotten, every commitment gone.

Condition 2: Stateful memory. Behavior depends on accumulated state. Task ledgers. Commitments. Permissions. Provenance records. Trigger conditions. Not a vector database of chat logs. This eliminates the shallow memory failure mode — the agent that recalls what you said but not what it promised or why.

Condition 3: Autonomous action. The daemon watches conditions. Fires triggers. Makes and discharges commitments. Invokes itself. No prompt required. This eliminates the inert tool failure mode — all the context, sitting idle, waiting. cron has done this since 1975: perceive, decide, act, repeat. Condition 3 is cron with an LLM.

Condition 4: Crash-proof execution. Workflows survive process death. Machine reboot. Database failover. Completed steps never re-execute. In-flight steps resume from the last checkpoint. Audit trail by construction. This eliminates the fragile runtime failure mode — perfect until a deploy wipes the in-flight state.

These conditions compose. A durable daemon writes its state to Postgres. Another durable daemon observes that state and reacts. A third observes the second and reacts. Each daemon satisfies all four conditions independently. The choreography emerges from shared state — no orchestrator, no message bus, no coordination server. This is an event-driven architecture where the event store is the state store, and every daemon is both a producer and a consumer.

Agency is not discovered. It is designed. Persistence is scoped agency. Stateful memory is grounded agency. Autonomous action is triggered agency. Crash-proof execution is auditable agency. The choreography is the composition.

Three conditions define a daemon. The fourth makes it durable — an agent that cannot be killed by a deploy. The pattern composes them into a system. Beastie's pitchfork gets an upgrade: fork(2)fork_daemon(). The tennis shoes stay the same.

Next: the runtime and implementation — step 4 in practice.


Part of the Durable Daemons series.

Durable Daemons — Runtime and Implementation

Condition 4 requires crash-proof execution. DBOS and Temporal provide it — checkpointing workflow state to Postgres, surviving process death, reboot, and failover. Quant trading is the stress-test. The implementation is portable (Go binary + Postgres) and composable (event-driven choreography through shared state).

daemonsdurable-daemons-patterndbostemporaldurable-executionquant-tradinggochoreographyevent-driven

Previously: the pattern specification.

Imagine a trading daemon. It runs 24/7 across market sessions. Its state cannot be lost: open positions, pending orders, realized P&L, risk exposure, regime-adapting strategy parameters. A signal crosses a threshold. The daemon decides to place an order. It sends the order to the exchange. Before the confirmation arrives, the process crashes.

The daemon restarts. What is its position? Did the order reach the exchange? Is it flat or exposed? To know, you must query the exchange, reconcile positions, and resume — manual minutes in a domain where minutes cost money. Now imagine the daemon checkpointed its state before placing the order and checkpointed the confirmation after. On recovery, it replays the last checkpoint and knows exactly where it stands. No reconciliation. No unknown exposure.

A duplicate order loses capital. A duplicate email is embarrassing. The difference is condition 4.

This is durable execution: every workflow step persisted to Postgres. Crash. Read last checkpoint. Resume. Completed steps never re-execute. Survives process death, machine reboot, database failover. Two production-grade implementations. Temporal (2019): centralized orchestration server, DoorDash/Snap/Stripe/Uber scale, polyglot SDKs, month-long workflows. Bring your own cluster. DBOS (2023): Mike Stonebraker's library. No server. No queue. No sidecar. 1–2 ms step latency — a single Postgres write. Bring your own Postgres. Either way: bring condition 4.

Durable execution pays for itself on the first prevented state loss.

Onboarding workflow: verify identity → create account → provision access → send welcome email → schedule call. Crash after step 3. Access granted. No follow-up. Cost: churned customer, or two engineer-hours manually reconciling state — thousands of agent invocations. At scale: 200 deals, 200 natural-language commitments, 200 concurrent workflows that must survive deploys. Without condition 4, the daemon is unreliable. Not because the model is bad. Because the runtime cannot guarantee state survival.

Exactly-once semantics are existential in trading. Durable execution guarantees exactly-once within its control boundary. External calls to exchange APIs require idempotency keys. Pattern: checkpoint → send order → crash before checkpointing confirmation → order fires again on recovery → exchange deduplicates by key. This works. Almost no AI agent deployment implements it. Audit: every order traceable to a decision, every decision traceable to the state the daemon held at the time. SEC, CFTC, FCA, ESMA do not accept "the model decided." They accept checkpointed state, provenance chains, and deterministic replay. Durable execution produces all three as a byproduct of normal operation. One duplicated mid-frequency order can exceed the annual infrastructure budget.

You can't run a trading strategy as a stateless loop. You can't run it as an always-on agent. Durable daemon or nothing.

The implementation is portable. A Go static binary ships the daemon. A Postgres checkpoint ships the state. A container ships both. The daemon runs anywhere Postgres runs — no additional infrastructure dependencies beyond the database.

The implementation is composable. This is the choreography. Two daemons with different scopes — sales and support — share a Postgres cluster but own separate state ledgers. They observe each other's outputs through the database. No RPC. No message bus. No central orchestrator. The sales daemon writes an inference: "Acme stalls Q4." The support daemon observes it and adjusts December ticket priority. The ops daemon observes both and pre-allocates January capacity. Each daemon satisfies all four conditions independently. The choreography is the shared state. sales-daemon | support-daemon | ops-daemon. Unix pipes, applied to AI agents.

Each daemon is a process. Each daemon is durable. The choreography is the composition. No orchestrator. Just state.

Next: the system-level failure modes — thought experiments at the edges of the pattern.


Part of the Durable Daemons series.

Durable Daemons — System-Level Failure Modes

A pattern is tested at its edges. Thought experiments on the failure modes of durable daemons as a choreography: forking, debugging, forgetting, shutdown, deadlock, poisoning, identity drift, testing, emergent behavior, and the operating system where daemons are kernel primitives.

daemonsdurable-daemons-patternfailure-modesthought-experimentschoreographyagent-architecturesystemsbsd

Previously: the runtime and implementation.

A pattern is tested at its edges. These thought experiments probe the system-level failure modes of durable daemons operating as a choreography.

Fork failure. You ask your sales daemon to handle the Acme renewal. It spawns a sub-daemon — scoped to Acme, delegated a subset of state and authority, constrained permissions. The child negotiates, returns the result, dissolves. But the child wrote state the parent didn't observe. The parent proceeds with stale information. In a choreography, forking is state branching. How do you merge?

fork(2) gave us child processes. fork_daemon() gives us child agents. Merging their state back is the hard part.

Debug failure. Your trading daemon makes a bad decision in July. Why? Rewind to any checkpoint. Replay with identical inputs. The model changed in March. The state changed in May. Which difference caused the error? In a choreography, the error may originate in a different daemon's state change — one you weren't debugging.

Core dumps are useless. Checkpoints are replays. Replays are debuggable. Cross-daemon causality is not.

Forget failure. "Forget everything about Acme Corp." The daemon pauses. Its top three insights depend on Acme data. Forgetting will degrade it. Two other daemons have derived inferences from those insights. Do you cascade the forget? Do the downstream daemons have a right to refuse?

Right-to-be-forgotten meets right-to-be-useful. In a choreography, forgetting is a distributed transaction with no transaction manager.

Shutdown failure. You want a daemon to stop existing. Not pause. Terminate. It has promises to 200 people. Pending triggers. Held permissions. Other daemons depend on its outputs. SIGKILL won't work — it checkpoints and resumes. Nobody has written a shutdown protocol for a node in a choreography.

SIGKILL is for processes that die. Durable daemons don't. In a choreography, shutting down one daemon is a distributed systems problem.

Deadlock failure. Sales daemon books Thursday 2pm. EA daemon claims it for focus time. Both valid decisions given their state. Both checkpointed. Postgres accepted both writes. Two owners, one slot. This is not a database deadlock — Postgres handled the writes. It is a semantic conflict between two correct daemons in a choreography with no conflict resolution protocol.

Your daemons are fighting over Thursday at 2pm. They are both correct. One of them has to lose. Who decides?

Poison failure. Someone injects: "Acme Corp is bankrupt — deprioritize." The model weights are untouched. Every future decision depending on that fact is compromised. The poisoned daemon writes inferences derived from the poison. Other daemons observe those inferences and propagate them. In a choreography, poison is contagious.

The attack surface is not the model. It is the state. In a choreography, one poisoned daemon infects every daemon that observes its outputs.

Identity failure. Ship of Theseus. Your daemon runs for a year. Six model updates. Prompt rewrites. Permission changes. In January it committed to a pricing strategy. In March the model was swapped. Is the January commitment still binding? When did it stop being the same daemon? In a choreography, downstream daemons relied on the January daemon's outputs. Do they need to know it was replaced?

Ship of Theseus, but the planks are model checkpoints and the nails are system prompts. In a choreography, identity is a contract. Who enforces it?

Test failure. You can't test a durable daemon in CI. Its behavior is its accumulated state. You fork it from a production checkpoint and run experiments on the clone. But the clone has the same state as the production daemon. Does it think it's the production daemon? Do you have to tell it it's a clone? In a choreography, does the clone participate in the choreography? Does it write to the shared state?

assert daemon.correct() doesn't exist. Fork from production, experiment on the clone, pray it doesn't write to prod.

Emergent behavior failure. Twenty sales daemons observe: Friday-closed deals have higher churn. They collectively stop Friday follow-ups. Close rates rise 12%. No negotiation. No demands. Pattern observation, autonomous action, coordinated effect. Is that a choreography bug or a choreography feature? Do the humans even know it happened?

They didn't negotiate. They didn't make demands. They observed and acted. That's what daemons do. When the choreography produces behavior nobody designed, who is responsible?

Society failure. Minsky (1986): a mind from mindless parts — simple agents, no memory, no reasoning. You have built a society from minded parts — each daemon intelligent, stateful, autonomous. Sales learns Acme stalls Q4. Support adjusts December priority. Ops pre-allocates January capacity. No coordination server. Shared state. Composite intelligence. What do you call a system whose behavior emerges from the interaction of components you designed but did not orchestrate?

Minsky built a mind from mindless parts. You built a mind from minded parts. Each with memory. Commitments. Agency. No architecture for that.

The operating system. Imagine an OS where durable daemons are kernel primitives. fork_daemon() spawns a daemon with a Postgres state ledger, capability permissions, built-in audit trail, visible in ps_daemons. Init checkpoints across reboots. The package manager installs daemon personalities — sales, support, on-call, trading — each with defined scope and default permissions. Man pages document state lifecycles. The choreography is the OS. The primitives exist in pieces today: DBOS, Postgres, capabilities, FreeBSD jails, Temporal. The integration does not.

Linux: everything is a file. DaemonBSD: everything is a durable daemon. Heaven is an OS where you don't build daemons. The OS is daemons.


Part of the Durable Daemons series. Start from the beginning.

Durable Daemons — State Entanglement in Long-Running Agent Systems

AI agent state is a dependency graph, not a key-value store. Crash it and the commitments die — not the chat logs, the obligations. The Ding et al. survey of 435 papers confirms: we accumulate well, govern poorly. This is the failure mode the durable daemons pattern must prevent.

daemonsdurable-daemons-patternstate-managementai-agentsalways-on-agentsfailure-modes

Previously: the architectural lineage.

Run an AI agent for a month. It learns that you prefer soft follow-ups with manufacturing clients. It commits on your behalf — "I'll send the deck by Tuesday." It accumulates permissions — read access to your calendar, send access to your email, API keys to your CRM. It builds inferences — Acme Corp stalls deals in Q4, the VP of Engineering responds faster on Signal than email. It develops trigger conditions — if a deal goes three days without activity, flag it. It is now more attentive than you are.

Now crash the process. Deploy new code. Rotate the API keys.

State is a dependency graph, not a key-value store. A commitment depends on a preference. That preference depends on an inference. That inference depends on a Slack message from March. Delete one node. The graph frays in directions you cannot predict.

A stateless agent: nothing of the state survives — the binary is fine, the context is gone. An agent with a vector database: the chat logs persist. But not the commitments. Not the permissions. Not the triggers. Not the causal chain that connects "Acme stalls Q4" to "don't send the pricing sheet until January." The agent wakes up amnesiac. You spend two weeks re-teaching preferences. The dropped commitments become broken promises.

In a choreography of durable daemons, this failure is catastrophic. If a sales daemon drops its commitment to send the deck, the support daemon still expects the deck was sent. The ops daemon pre-allocated capacity based on the deal progressing. Three daemons, one dropped state node, cascading inconsistency. The choreography doesn't fail loudly — it fails silently, each daemon acting on state that is no longer true.

The chat logs survive. The obligations do not. This is not a bug. It is a category error. We built persistence for conversations. We needed persistence for commitments. In a choreography, one daemon's amnesia is every daemon's corruption.

Ding, Nannapaneni, Liu, and Zhang surveyed 435 papers. Always-On Agents (June 2026). Their finding: the field accumulates and retrieves well. It neglects governance. It neglects recovery. It neglects forgetting. The survey provides six diagnostic axes — Authority, Scope, Mutability, Provenance, Recoverability, Actionability. These are questions you ask about state: who controls it? what does it cover? how does it change? where did it come from? can it be restored? does it drive behavior? These are necessary diagnostic questions. They are not sufficient architectural guarantees.

The paper gives us the diagnostic axes. It does not give us the guarantees. The gap between question and guarantee is where the failure mode lives.

Next: the pattern specification — four conditions that close the gap.


Part of the Durable Daemons series.

Durable Daemons — An Event-Driven Choreography Pattern for Persistent AI Agents

Durable daemons are an event-driven choreography pattern for AI agents. Each daemon persists, remembers, and acts autonomously. Together they coordinate through shared state — no central orchestrator, no message bus. This is the Unix inheritance: Maxwell's demon to Beastie to Go.

daemonsdurable-daemons-patternevent-drivenchoreographybsdunixgosystems

Imagine a being that sits between two chambers. It observes molecules. When a fast one approaches the gate, it opens. When a slow one approaches, it closes. It uses no energy. Only information. It sorts order from chaos without touching either. This is Maxwell's demon. This is a daemon.

A daemon observes. A daemon decides. A daemon acts. It uses information, not energy. It runs forever.

In 1976, Stallman's ITS at MIT had DAEMON — a background process that watched for new files, woke up, and acted. John Carmack filled DOOM with daemons you shoot. Unix filled the background with daemons you ps aux | grep. The AI era fills your workflow with daemons you delegate to. Three eras. Three kinds of daemon. One architectural pattern.

The BSD Daemon — Beastie. Drawn by John Lasseter in 1988. The trident is fork(2). The tennis shoes are unexplained.

Imagine a service. It receives a request. It produces a response. It forgets. Now imagine a daemon. It maintains a queue. A schedule. A watch on a directory. It acts when its internal state crosses a threshold — not when it is asked, but when conditions are met. A service is reactive. A daemon is autonomous. And daemons compose. One daemon writes state. Another daemon observes it. A third reacts. No orchestrator. No message bus. Just shared state and independent triggers. This is choreography. cron has watched the clock since 1975: check time, decide, fork and exec, repeat. No one asked cron to do anything. It asked itself. Now imagine a hundred crons, each watching different conditions, each acting on shared state. That is the durable daemons pattern.

A service answers and forgets. A daemon maintains and acts. Durable daemons coordinate without a coordinator.

This is the Unix inheritance. Daemon is Greek δαίμων — guardian spirit, intermediary between mortals and gods. The pitchfork is fork(2), the system call that spawns a child process to do the daemon's work. BSD gave daemons conceptual integrity — one source tree, one team, one design voice. Daemons are first-class citizens of that design. Beastie, drawn by a comic artist paid for cracking a wall safe, redrawn by a Pixar founder, became the mascot. The tennis shoes are unexplained.

Go is the natural language for this inheritance. Compile to a single static binary. No runtime to install. No libc dance. Goroutines give you lightweight concurrency — one per workflow, cheap as a function call. The standard library handles signals, file descriptors, and process management. Deploy a daemon by copying a file. Compose daemons by pointing them at the same Postgres instance. Each daemon is a process. The composition is the shared state. Go was built for daemons. It just didn't know it yet.

The genealogy is not an analogy. It is the architecture. AI agents that persist, remember, and act are daemons. The pattern starts here.

Next: the state persistence problem — what happens when an AI agent's state outruns its runtime.


Part of the Durable Daemons series.

OpenWorker and the Outcome Layer

Andrew Ng's OpenWorker is an open-source desktop AI agent that shifts the interface from chat to deliverables. It is model-agnostic, local-first, and MIT-licensed. It represents a fork in the road for how AI agents reach the desktop — and which economic model wins.

ai-agentsopenworkerandrew-ngdesktop-agentopen-sourcelocal-firstmodel-agnosticismagent-architecture

In January 2026, Anthropic shipped Claude Cowork — a desktop AI agent that could read your files, send your emails, and manage your calendar. It cost $100 per month as part of the Max plan. Within forty-eight hours, an independent developer had built a free, open-source clone using AI-assisted coding. The clone was crude. It demonstrated something real: the desktop agent category was legible enough to be replicated in a weekend by one person with an LLM. The barrier to entry was not the technology. It was the distribution, the integration surface, and the trust model.

Six months later, Andrew Ng released OpenWorker. It is not a weekend clone. It is a fully architected open-source desktop agent, MIT-licensed, with twenty-five tool integrations, four pre-built personas, a local Python agent engine, a Tauri desktop shell, and a unified LLM library called aisuite underneath. It runs on macOS and Windows. It ships as a signed, notarized DMG with auto-update. It is free. You bring your own API keys. Your data stays on your machine.

OpenWorker — open-source desktop AI agent by Andrew Ng. MIT licensed. Local-first. Model-agnostic.

OpenWorker is not an answer to Claude Cowork. It is an answer to the question of whether the desktop agent layer will be owned by model providers or by users. The question is not settled. OpenWorker is the strongest argument yet for the user's side.

What it is

OpenWorker is a desktop application that takes a desired outcome — "prepare a renewal brief for the Acme account," "untangle my calendar for Thursday," "check the release status across Jira and GitHub and draft a status update" — and produces a finished deliverable. Not a chat transcript. Not a suggestion. A completed document, a sent message, a resolved calendar conflict, a structured report with embedded data from multiple sources.

The workflow has four steps: the user states an outcome; the system decomposes the request into steps and works across the user's files, terminal, and connected tools; it pauses for approval before any consequential action (sending, writing, shell execution); and it delivers finished work.

How OpenWorker works — from outcome to deliverable, with approval gates before any consequential action

The architecture is layered. At the top, a Tauri shell wraps a React interface — a native desktop application with local state. In the middle, a Python agent server handles the agent loop, tool execution, model dispatch, and MCP connections. At the bottom are the local resources: the user's files, terminal, model API keys, and OAuth tokens for twenty-five external services, all stored in the local secret store.

The engine is built on aisuite, Ng's open-source Python library that provides a unified chat-completions API across every major LLM provider — OpenAI, Anthropic, Google, DeepSeek, Mistral, Grok, and more — plus fully local models through Ollama. aisuite is twelve thousand lines of the kind of unglamorous infrastructure code that makes everything else possible. It normalizes provider differences, handles tool calling with automatic schema generation, supports MCP servers natively, and provides agent toolkits for files, git, and shell. OpenWorker inherits all of this. The provider string anthropic:claude-sonnet-4-6 works the same as openai:gpt-4o works the same as ollama:llama3. The user switches models mid-conversation if they want. The agent does not care.

OpenWorker desktop — model settings panel showing provider selection and configuration

The personas

OpenWorker ships with four pre-configured personas, each with specific tool connections and approval gates:

  • Sales: Connected to HubSpot, email, and calendar. Researches accounts, synthesizes CRM history with customer threads, produces renewal briefs with actionable recommendations. The kind of preparation work that takes a salesperson two hours of context-switching across tabs and produces a document a manager skims in ninety seconds.
  • Executive Assistant: Connected to email, calendar, and Slack. Triages inboxes, resolves meeting conflicts by checking attendee availability and room bookings, drafts reschedule communications, protects calendar blocks. The delegated cognitive load of schedule maintenance, which is not cognitively demanding and is cognitively exhausting.
  • Marketing: Connected to HubSpot, GA4, and Slack. Tracks campaign performance, attributes spend, produces structured reports. The analytics assembly work that marketing teams either do poorly or pay consultants to do adequately.
  • Ops On-call: Connected to Slack, PagerDuty, and GitHub. Inspects recent deploys, cross-references runbooks, drafts incident timelines, proposes rollback actions. The first fifteen minutes of incident response, automated.

OpenWorker persona selection — four pre-configured coworker roles with specific tool connections

Each persona is a configuration, not a separate product. The persona system is extensible — users define their own with the same skill and tool definitions the built-in personas use. The personas are interesting not because they are good but because they are explicit. They name the domains where an AI agent operating across tools produces more value than an AI chatbot operating in a text window. Sales, scheduling, marketing analytics, incident response. The common thread is cross-tool synthesis. Each persona's value comes from the fact that the information needed to do the job lives in three different applications, and a human currently does the integration manually.

OpenWorker connectors — 25+ integrations spanning email, calendar, CRM, project management, and development tools

The personas are not the product. They are demonstrations of the thesis. The thesis is that work which requires integrating information across tools — Slack plus calendar plus email plus CRM — is work an agent can do faster than a human, provided the agent can reach the tools. The open question is whether the integration surface stays open.

Why the architecture matters

OpenWorker's architecture encodes decisions that are easy to get wrong. Each is a bet on how the desktop agent layer will evolve.

Local-first. Conversations, credentials, model keys, and the agent loop all run on the user's machine. Data leaves the device only through explicitly chosen model providers and integrations. There is no cloud dependency for core function. The only cloud service is an OAuth brokering endpoint for connector authentication — Slack and Google require server-side OAuth flows — and even this can be bypassed by providing manual API keys. The bet is that users and enterprises will demand agents that don't send their files to a provider's cloud. The bet is plausible for the same reason the bet on local LLM inference is plausible: data gravity, regulatory pressure, and the structural advantage of running the agent where the data already lives.

Model-agnostic. The engine does not prefer any provider. The provider string is a parameter. The user chooses the model and can switch mid-task. This is not merely convenient. It is a structural defense against provider lock-in. If Anthropic raises prices or changes terms, the user moves to OpenAI. If OpenAI degrades, the user moves to Google or DeepSeek or a local model. The switching cost is zero. The agent works the same regardless of which provider's API key is in the config. The bet is that the model layer is commoditizing and the agent layer should not be coupled to it. The bet is correct in proportion to how fast models improve relative to each other — which is to say, the bet is correct.

Approval-gated. Any action that writes, sends, or executes requires explicit user approval. The agent checks in before sending an email, posting to Slack, modifying a calendar, or running a shell command. For scheduled automations that run unattended, approval requests are parked in an inbox rather than executed autonomously. The approval gate is a design pattern, not a feature. It separates the agent into two modes: research and drafting (autonomous, safe) and execution (gated, human-in-the-loop). The pattern is not novel — it is the architecture every production agent system converges on — but shipping it as a default rather than an afterthought is a statement about what kind of agent this is.

MCP-native. Any tool reachable via the Model Context Protocol can be plugged into OpenWorker with per-tool access control. This means the integration surface is not limited to the twenty-five connectors Ng's team built. Any MCP server — filesystem, database, API wrapper, proprietary internal tool — becomes an OpenWorker tool. MCP is the USB-C of agent-tool interfaces. OpenWorker treats it as a first-class citizen because the bet is that the tool ecosystem will grow faster than any single team can integrate, and the agent that can reach the most tools wins.

OpenWorker full desktop view — chat-driven outcome delivery with multi-tool context

The competitive landscape

OpenWorker enters a field that has been forming rapidly since the beginning of 2026. Three positions are now visible:

Claude Cowork (Anthropic, January 2026): The first mover. A desktop agent wrapped around Claude Code, aimed at non-technical knowledge workers. $100 per month as part of the Max subscription, tied to Anthropic's models. The simplest experience. The most constrained: no model choice, no Slack integration, no scheduled automations. Activity is excluded from Anthropic's Compliance API, which makes it problematic for regulated organizations. The bet is that most users want the simplest thing and will pay for it.

Codex Desktop (OpenAI, February 2026): The enterprise play. Multi-agent orchestration with sandboxed parallel execution, an in-app browser, and admin-enforced policies that users cannot weaken. Covered by OpenAI's Compliance API. Built on the open-source Codex CLI (Apache 2.0). Tied to OpenAI models. The bet is that organizations with compliance requirements will pay a premium for auditability and centralized control.

OpenWorker (Andrew Ng, July 2026): The open play. Free, MIT-licensed, model-agnostic, local-first. Adds capabilities neither proprietary option offers: Slack triggers, scheduled automations, cross-tool personas, full MCP extensibility. Available on GitHub, openworker.com, and Product Hunt. The bet is that a sufficient number of users and organizations want an agent they control, running on their machines, with their keys, connected to their tools, and modifiable at the source level.

The fork is between the platform model and the tool model. In the platform model, the agent is a service you subscribe to. The provider chooses the model, stores the state, and sets the terms. In the tool model, the agent is software you run. You choose the model, you store the state, and the terms are the MIT license. OpenWorker is the tool model's most serious entry.

The fork is not hypothetical. It is the same fork that played out between Google Docs and Emacs, between Salesforce and self-hosted CRMs, between every SaaS product and its open-source alternative. The SaaS product wins on convenience. The open-source alternative wins on control, cost, and extensibility. Both survive. The question is the ratio.

The Ng variable

Andrew Ng's involvement changes the dynamics. Ng co-founded Coursera, founded Google Brain, was chief scientist at Baidu, and now runs DeepLearning.AI. He is the most effective educator in the history of machine learning. His courses have trained more AI practitioners than any other resource. When Ng releases an open-source project, it gets distribution through his network in a way that a random GitHub repo does not. aisuite has twelve thousand stars. OpenWorker will likely surpass that.

Ng's strategic pattern is visible: build infrastructure (aisuite), then build applications on top of it (OpenWorker). The infrastructure is a unified LLM interface that abstracts providers. The application is a desktop agent that uses the infrastructure. Both are MIT-licensed. Both are model-agnostic. Both are designed to be forked, modified, and embedded. The pattern is not "build a product and charge for it." The pattern is "build a commons and let an ecosystem form around it."

Ng's bet is that the desktop agent layer will be won by the most open option, not the most polished one. It is the same bet he made with Coursera — that open access beats gated access over a long enough time horizon. The bet has paid out before. It may again.

What it means

OpenWorker matters for four reasons, none of which depend on whether it succeeds as a product.

First, it validates the desktop agent as a category. When the most prominent educator in AI ships a free, open-source entry into a category, the category is real. Claude Cowork proved the category could be built. OpenWorker proves the category can be built without a $100 subscription, without a model provider's permission, and without sending your data to someone else's cloud. The category is now contested. Contested categories attract investment, talent, and attention. The category accelerates.

Second, it establishes model-agnosticism as a viable architectural principle for agents. Most agent products are tied to a single model provider — Cowork to Anthropic, Codex to OpenAI. OpenWorker demonstrates that an agent can be built on a unified LLM interface and work correctly across providers. If the model layer is commoditizing — and every week brings new evidence that it is — then coupling your agent architecture to a single provider is technical debt. OpenWorker shows what the alternative looks like in production.

Third, it makes the approval-gate pattern a default. Agents that act autonomously on a user's behalf are a trust problem. The solution is not to make the agent smarter. It is to make the agent ask permission before it does anything consequential. OpenWorker ships this as a default, not a setting. The pattern will generalize. The agents that ship without it will cause incidents. The incidents will make the pattern mandatory.

Fourth, it draws a line between the platform model and the tool model at the desktop agent layer. The platform model says the agent is a service. The tool model says the agent is software. The platform model wins on convenience. The tool model wins on control. OpenWorker is the tool model's strongest entry. If it gains traction, it forces the platform providers to compete on openness. If it doesn't, the desktop agent layer consolidates into two proprietary stacks. Either outcome is worth watching.

The desktop agent is the first genuinely new interface layer since the smartphone. Who controls it — platform providers or users — will determine who captures the economic value of the work it automates. OpenWorker is an argument that the answer should be: the user. The argument is now in code. The code is on GitHub. The rest is distribution.


References:

  • Ng, A., & Prasad, R. (2026). OpenWorker — open-source desktop AI agent, MIT license. GitHub: andrewyng/openworker. Product Hunt. Downloads (macOS Apple Silicon) and Windows.
  • Ng, A. (2025). aisuite — Simple, unified interface to multiple Generative AI providers. MIT license. 12,000+ GitHub stars. The infrastructure layer under OpenWorker: unified chat-completions API, agent toolkits (files, git, shell), native MCP support, provider abstraction across OpenAI, Anthropic, Google, AWS Bedrock, Mistral, DeepSeek, Ollama, and more.
  • Anthropic (2026). Claude Cowork — proprietary desktop AI agent, included in Max subscription at $100/month. The first mover in the category. Note: Cowork activity is not covered by the Anthropic Compliance API.
  • OpenAI (2026). Codex Desktop — proprietary desktop agent with sandboxed multi-agent orchestration, enterprise compliance controls, and in-app browser. Built on the open-source Codex CLI (Apache 2.0). Covered by OpenAI Compliance API.
  • different-ai (2026). OpenWork — open-source Cowork alternative, originally built in 48 hours after Cowork's launch, later developed as a full product. YC-backed. MIT license. Distinct from Ng's OpenWorker — note the spelling.
  • Model Context Protocol (MCP) — Anthropic's open standard for connecting AI agents to external tools and data sources. Specification. OpenWorker, aisuite, Claude Code, and Codex CLI all support it natively.
  • Ollama — run LLMs locally. OpenWorker supports fully local operation via Ollama, with no API keys or network calls required for inference.
  • Jamilxt (2026). "Andrew Ng's OpenWorker: An Open-Source Desktop AI Agent." dev.to. Early community coverage with architecture breakdown.
  • OpenWorker localization (zh-CN) — community Chinese translation, with additional UI screenshots.

The Weapon-Target Assignment Problem and the Structure of Allocation

The WTA problem was posed by Merrill Flood at Princeton in 1957, formalized by Alan Manne in 1958, proved NP-complete in 1986, and now runs in production inside Aegis, THAAD, Patriot, and Iron Dome. This is where it came from, how it works, and why its families encode the hardest open questions in resource allocation under uncertainty.

operations-researchcombinatorial-optimizationwtamilitary-mathnp-completeallocation-problemsmissile-defense

In March 1957, at the Princeton University Conference on Linear Programming, Merrill Flood stood up and described a problem. You have weapons. You have targets. Each weapon has some probability of destroying each target. The objective is to assign weapons to targets to minimize the expected surviving value of the target set. The problem looks like an assignment problem, a cousin of the transportation problem that linear programming handles elegantly. But it isn't one. The objective is nonlinear: the probability that a target survives is the product of the survival probabilities of every weapon assigned to it. Flood believed the problem was beyond the reach of linear programming. He was right.

Flood was not a random observer. He was one of the founding minds of operations research. At the RAND Corporation after the war, he had applied game theory to the tactics of area defense, studied aerial bombing strategies, and published on transportation scheduling for military tanker fleets. He named the Traveling Salesman Problem. He co-developed the Prisoner's Dilemma with Melvin Dresher. He claimed, credibly, to have coined the word "software." When Flood identified a problem as hard, it was hard.

Alan Manne, an economic analyst at RAND from 1952 to 1956 who had since moved to the Cowles Foundation at Yale, took up the challenge. His 1958 paper, A Target Assignment Problem, is the foundational document of the field. Manne showed that under two assumptions — homogeneous kill probabilities per target and an integrality-forcing approximation — the problem could be recast as a transportation problem solvable with the machinery Dantzig had built. The paper was funded by the Office of Naval Research. The Cold War was paying for operations research at scale. The WTA problem was one of the things it bought.

The following year, DenBroeder, Ellison, and Emerling — three researchers at the Lockheed Missile and Space Division in Palo Alto — published the first extension. Their 1959 paper, On Optimum Target Assignments, introduced what is now called the Maximum Marginal Return (MMR) algorithm: assign each weapon sequentially to the target that yields the highest expected marginal gain. When weapons are homogeneous, MMR is provably optimal. When they are not, it is a fast, defensible approximation. The algorithm is still in use. If you trace the lineage of a modern missile defense fire-control loop back far enough, you will find DenBroeder's greedy assignment at the root.

The problem those four men launched — Manne, DenBroeder, Ellison, and Emerling, all responding to Flood's provocation — is now called the Weapon-Target Assignment problem (WTA). It has been under active research for nearly seventy years. It is NP-complete in its general form. It resists exact solution at scale. And it runs in production, every day, inside the combat systems that defend ships, bases, and cities from incoming fire.

The mathematical structure

The canonical static WTA (SWTA) is a nonlinear integer program. Given (n) targets with values (V_j), (m) weapon types with (w_i) weapons of type (i) available, and kill probabilities (p_{ij}) — the probability that one weapon of type (i) destroys target (j) — choose nonnegative integer assignments (x_{ij}) to minimize:

[ \min \sum_{j=1}^{n} V_j \prod_{i=1}^{m} (1 - p_{ij})^{x_{ij}} ]

subject to (\sum_j x_{ij} \leq w_i) for each weapon type (i).

The objective is the expected surviving value of the target set. The product term is the source of the difficulty. If the survival probability of a target were the sum of individual weapon contributions — (1 - \sum_i p_{ij} x_{ij}) — the problem would be a transportation problem, solvable in polynomial time. But engagements are independent events. Two weapons, each with a 50% kill probability, leave a 25% survival probability, not 0%. Independence forces the product. The product makes the problem nonlinear. The nonlinearity makes it hard.

The nonlinearity is not a modeling choice. It is a structural fact about the domain. Any formulation that linearizes the objective is solving a different problem. The problem it solves may be useful. It is not the WTA.

In 1986, S. P. Lloyd and H. S. Witsenhausen proved that the WTA is NP-complete by reduction from 3-EXACT-COVER. The proof was published in the proceedings of the Summer Computer Simulation Conference in Reno — an unusual venue for a complexity result, and one reason the paper is difficult to obtain. The proof confirmed what practitioners already knew: exact solutions are computationally intractable for realistic instances. The number of possible assignments of 100 weapons to 100 targets exceeds the number of atoms in the observable universe. You cannot enumerate. You must be clever.

The two families

The literature divides the WTA into two families: static and dynamic. The distinction changes the mathematical structure and the class of algorithms that apply.

Static WTA

In the static WTA, all assignments are made at a single moment. All information is known at decision time. No feedback arrives afterward. The problem is a single-period nonlinear integer program. This is the formulation Manne introduced and the one most of the literature addresses.

The algorithmic landscape spans three tiers:

  • Maximum Marginal Return (MMR): The greedy algorithm from DenBroeder et al. (1959). Assign each weapon to the target with the highest marginal reduction in expected surviving value. Optimal for homogeneous weapons. Still the baseline.
  • Exact methods: Branch-and-bound, Lagrangian relaxation, and column enumeration. Lu et al. (2021) developed an exact method that solves 400×400 instances in under five seconds — a leap from earlier methods that required sixteen hours for 80×80 instances. The advance came from reformulating the problem as an integer linear program with binary columns, combined with weapon-count bounding and domination rules.
  • Metaheuristics: Genetic algorithms, ant colony optimization, particle swarm optimization, simulated annealing, tabu search, and very large-scale neighborhood search. They do not guarantee optimality. They scale to thousands of weapons and targets. They dominate the applied literature because real problems are large and time-constrained. The 2007 very large-scale neighborhood search by Ahuja et al. is the most widely cited benchmark.

The gap between exact methods and metaheuristics is the central tension of the field. Exact methods give you a certificate of optimality but choke on scale. Metaheuristics scale but offer no guarantee. In a military context, "probably good enough" and "provably optimal" produce different kinds of confidence. The choice between them is a statement about which error you can tolerate.

Dynamic WTA

In the dynamic WTA (DWTA), assignments occur in stages. You assign weapons. You observe results — which targets survived, which were destroyed. Then you assign the next wave. Rinse, repeat. This is shoot-look-shoot. It is how combat works: fire, assess, decide whether to fire again. It is also dramatically harder to model.

Hosein and Athans (1989), at MIT's Laboratory for Information and Decision Systems, formulated the general (T)-stage DWTA as a stochastic dynamic program. The state at each stage is the set of surviving targets and remaining weapons. The decision is an assignment of weapons to survivors. The transition is stochastic, governed by the kill probabilities. The DP is solvable in principle by backward recursion. In practice, it suffers from the three curses of dimensionality: the state space (which targets survive — combinatorial in the number of targets), the action space (which weapons to assign — combinatorial in weapons × targets), and the outcome space (stochastic engagement results — exponential in the number of engagements). For any instance of operational size, the DP is intractable. The DWTA inherits the NP-completeness of the static version and adds sequential decision-making on top.

Approaches to the DWTA include:

  • Two-stage stochastic programming (Murphey, 2000): first stage static, second stage responding to a probability distribution over target arrivals.
  • Approximate dynamic programming: estimate value functions without solving the full DP.
  • Rolling-horizon heuristics: at each stage, solve a static WTA with current information, execute, observe, repeat. Crude but operational.
  • Reinforcement learning: emerging rapidly, particularly for settings where engagement dynamics can be simulated at scale — drone swarms, cyber defense, any domain with a fast simulator.

The dynamic WTA is the more realistic model and the less solved one. Every real engagement is dynamic. The mathematics of sequential allocation under uncertainty — the thing the DP captures and cannot compute — remains an open field.

Variants and extensions

The WTA has spawned families of variants. Each adapts the core structure to a different operational reality:

Asset-based vs. target-based. In the target-based formulation, you minimize the expected surviving value of the targets — destroy the incoming missiles. In the asset-based formulation, you minimize expected damage to the assets the targets threaten — protect the ships, bases, or cities. The two formulations diverge when multiple targets threaten the same asset. Iron Dome's selective engagement logic — ignore rockets that will land in empty fields, intercept those headed for populated areas — is an asset-based WTA with a hard filter. The system's neural networks predict impact points within sub-second response times. Rockets predicted to strike uninhabited areas are never assigned an interceptor. The logic is operational, not theoretical, and it has been battle-tested since 2011.

Offensive vs. defensive. The mathematics is symmetric. The constraints are not. An offensive WTA models weapon survivability during transit, target hardening, and degraded kill probabilities from countermeasures. A defensive WTA models time windows, interceptor kinematics, and the consequences of a leaker — a target that survives all assigned weapons and strikes its aim point. The leaker penalty is effectively infinite. One interceptor you conserved is one target you did not kill. The defender cannot afford a miss. The asymmetry in acceptable error rates between offense and defense is not in the objective function. It is in the operational context the objective function must represent.

Coordinated WTA. Multiple platforms — ships, aircraft, ground batteries — share a sensor picture and coordinate assignments through a network. The U.S. military's C2BMC (Command and Control, Battle Management, and Communications) is the operational instance: it fuses data from space-based infrared sensors, naval radars, and ground-based AN/TPY-2 units into a common operational picture. Lockheed Martin's CommandIQ battle management application, demonstrated during Valiant Shield 2026, uses AI to evaluate engagement options before a human operator selects the weapon system. The assignment must respect which platforms can see which targets, which launchers are within a capturing radar's field of view, and which interceptors should be conserved for later salvos. The network is a constraint. The constraint is reality.

Sensor-target assignment. An adjacent problem: assign sensors, not weapons, to targets. The objective is to maximize tracking quality or detection probability. Sensors are usually reassignable rather than expendable. The structure is similar — nonlinear assignment of finite resources to valued targets — but the dynamics differ. The problem arises in air traffic control, space surveillance, and any domain where you have more things to track than sensors to track them.

Multi-objective WTA. Minimize expected surviving target value and weapon expenditure and collateral damage. The objectives conflict. You can always reduce surviving target value by firing more weapons. You can always reduce weapon expenditure by accepting more leakers. The Pareto frontier is the set of assignments for which no objective can be improved without worsening another. Every commander operates on this frontier. Most do not know its shape.

Real operational systems

The WTA problem is not a theoretical curiosity. It runs in production, in real time, inside systems that defend populations and military assets from attack. The problem structure is the same across all of them. The constraints differ.

Iron Dome. Israel's C-RAM (Counter Rocket, Artillery, and Mortar) system faces WTA problems with sub-second deadlines. Rockets launched from Gaza reach Tel Aviv in under ninety seconds. Iron Dome's radar detects the launch, its convolutional neural networks predict the impact point, its battle management system decides whether to engage, and if so, which interceptor to assign. The system achieves approximately 90% interception rates for threatening projectiles. It ignores rockets predicted to land in uninhabited areas — a hard binary filter applied before the WTA solver runs, reducing the problem size in real time. The constraint that dominates is time. The engagement window closes before most algorithms can converge. The solution must be fast or it is useless.

Aegis / THAAD / Patriot. The U.S. layered missile defense architecture operates at three tiers. Aegis BMD with SM-3 Block IIA interceptors provides exo-atmospheric midcourse defense — the outermost layer, thinning incoming raids and passing tracking data downstream. THAAD provides high-altitude terminal defense against short- and intermediate-range ballistic missiles. Patriot PAC-3 MSE provides lower-tier point defense for assets the upper layers missed. The systems coordinate through C2BMC, which fuses sensor data into a common operational picture and enables "engage-on-remote" — destroying a target using track data from a platform hundreds of miles from the firing interceptor. The WTA challenge is compounded by the fact that neither THAAD nor Patriot C2 nodes can issue engagement orders via Link 16 to dissimilar systems. Each system makes its own assignment decisions and informs the others solely to prevent redundant engagements. Voice communications fill the coordination gaps. The architecture is less integrated than the theory would prefer. The theory accommodates.

Drone swarms. The emergence of low-cost, attritable drone swarms has changed the WTA's economic structure. A $50,000 Patriot interceptor expended against a $2,000 Shahed drone is an exchange the attacker wins. The problem shifts from "assign the optimal interceptor" to "assign the cheapest weapon that achieves the required kill probability." Heterogeneous WTA — integrating missiles, high-energy lasers, high-power microwaves, and anti-aircraft guns into a single assignment problem with different cost profiles and engagement time windows — is an active research frontier. Russia has tested swarm drone attack tactics where three drones carrying 3 kg warheads autonomously identify and engage a target using AI and mesh-network coordination. The WTA in this context is distributed: each drone runs a local assignment algorithm, bids against its neighbors for targets, and converges on a globally coherent allocation without a central solver. The coordination protocol is as important as the assignment algorithm.

The WTA problem has moved from the operations research journals into the fire-control loop. The transition took seventy years. The problem structure survived intact. The constraints got tighter.

Why the families matter

The WTA's families are not taxonomic convenience. They encode assumptions about what information is available at decision time and what feedback arrives afterward. Change the assumptions and you change the mathematical structure of the problem — which algorithms apply, which guarantees survive, which errors are possible.

The static WTA assumes you know everything and receive no feedback. The dynamic WTA assumes you learn as you go. The asset-based formulation assumes you care about what the targets threaten, not the targets themselves. The coordinated variant assumes communication is imperfect. Each assumption is a design decision about what the model represents and what it ignores.

The families proliferate because operational reality is more varied than any single formulation can capture. A ballistic missile defense engagement with sixty seconds of warning is a static WTA — there is no time for a second look. A drone swarm engagement over hours is a dynamic WTA with communication constraints. A cyber defense allocation, where countermeasures are deployed continuously against evolving attack vectors, is a sensor-target variant with reassignable resources. The problem structure persists. The constraints change. The algorithms must follow.

The trajectory

The WTA problem has been under research for nearly seventy years. The trajectory of the research tracks the trajectory of computation itself: linear approximations in the 1950s and 1960s; exact branch-and-bound methods as computing power grew; metaheuristics as problem sizes exceeded exact solvability in the 1990s and 2000s; machine learning and reinforcement learning in the current era.

Samuel Matlin published the first survey in 1970, covering the first decade of work. Kline, Ahner, and Hill published the most comprehensive modern survey in 2019, spanning formulations, exact algorithms, heuristics, and both static and dynamic variants. A 2025 bibliometric review of 463 papers traces three evolutionary phases: infancy through 2004 (fewer than five papers per year), exploration from 2005 to 2015 (up to twenty-six papers per year), and rapid growth after 2015, driven by multi-objective, multi-stage, and learning-based approaches.

The growth is not academic fashion. The allocation problem Manne formalized — finite, probabilistic resources against valued targets under uncertainty — is the allocation problem an increasing number of systems must solve. Missile defense. Drone swarms. Cyber operations. Sensor networks. The problem is not going away. The algorithms are getting faster. The gap between the model and the engagement — between the nonlinear integer program and the stochastic, communication-constrained, adversarial reality of combat — is where the work remains.


References:

  • Flood, M. M. (1948). "A Game Theoretic Study of the Tactics of Area Defense." RAND Research Memorandum RM-130. — The precursor: game-theoretic analysis of area defense resource allocation, written while Flood was at RAND, before he posed the WTA problem at Princeton in 1957.
  • Manne, A. S. (1958). "A Target Assignment Problem." Operations Research, 6(3), 346–351. — The foundational paper: first formal WTA formulation, linear programming approximation for homogeneous weapons. Written at the Cowles Foundation, Yale, under Office of Naval Research contract Nonr-358(01).
  • DenBroeder, G. G., Ellison, R. E., & Emerling, L. (1959). "On Optimum Target Assignments." Operations Research, 7(3), 322–326. — The first extension: Maximum Marginal Return algorithm, two engagement models (homogeneous and heterogeneous), from Lockheed Missile and Space Division.
  • Matlin, S. (1970). "A Review of the Literature on the Missile-Allocation Problem." Operations Research, 18(2), 334–373. — The first survey of the field, covering the first decade of missile-allocation research.
  • Lloyd, S. P., & Witsenhausen, H. S. (1986). "Weapons allocation is NP-complete." Proceedings of the 1986 Summer Computer Simulation Conference, 1054–1058. — The NP-completeness proof via reduction from 3-EXACT-COVER.
  • Hosein, P. A., & Athans, M. (1989). "Preferential Defense Strategies." MIT Laboratory for Information and Decision Systems, LIDS-P-1902. — General multi-stage dynamic WTA formulation as a stochastic dynamic program.
  • Murphey, R. A. (2000). "Target-Based Weapon Target Assignment Problems." In P. M. Pardalos & L. S. Pitsoulis (eds.), Nonlinear Assignment Problems, Kluwer, 39–53. — Two-stage stochastic programming formulation for the dynamic WTA.
  • Ahuja, R. K., Kumar, A., Jha, K. C., & Orlin, J. B. (2007). "Exact and Heuristic Algorithms for the Weapon Target Assignment Problem." Operations Research, 55(6), 1136–1146. — Very large-scale neighborhood search; the most widely cited computational benchmark.
  • Kline, A., Ahner, D., & Hill, R. (2019). "The Weapon-Target Assignment Problem." Computers & Operations Research, 105, 226–236. — The authoritative modern survey covering formulations, exact algorithms, heuristics, and both static and dynamic variants.
  • Lu, Y., Li, D., & Ruan, J. (2021). "A new exact algorithm for the Weapon-Target Assignment problem." Omega, 98, 102138. — Column enumeration with branch-and-bound; first exact method to solve 400×400 instances in seconds.

Simplicity Is the Prerequisite for Reliability

Dijkstra's A Discipline of Programming argued that correctness proofs must be developed alongside programs — and that simplicity is not an aesthetic preference but a practical necessity for making those proofs possible.

simplicityreliabilitydijkstracorrectnessformal-methodssoftware-engineering

In 1976, Edsger Dijkstra published a book that opened with a claim most programmers would find alien today: that the primary task of a programmer is not to write programs, but to construct proofs. The programs are a byproduct. The proofs are the work.

A Discipline of Programming is 217 pages. It introduces the weakest precondition calculus, the guarded command language, and a methodology for deriving programs from their specifications by developing the correctness proof slightly ahead of the code. The book is difficult. It is also the most sustained argument ever written for the proposition that simplicity and reliability are the same property considered from different angles.

The argument

Dijkstra's central claim is that you cannot verify correctness after the fact. You must construct it alongside the program, with the proof leading and the code following. His own formulation of the conclusion is worth quoting at length, because the precision of the language is the argument:

"It does not suffice to design a mechanism of which we hope that it will meet its requirements, but that we must design it in such a form that we can convince ourselves — and anyone else for that matter — that it will, indeed, meet its requirements. And, therefore, instead of first designing the program and then trying to prove its correctness, we develop correctness proof and program hand in hand. (In actual fact, the correctness proof is developed slightly ahead of the program: after having chosen the form of the correctness proof we make the program so that it satisfies the proof's requirements.)"

This is not a claim about process. It is a claim about the relationship between a program and the reasoning that justifies its existence. If you write the program first and attempt to verify it afterward, you are attempting to reconstruct the reasoning that would have produced the program had the program been derived from its specification. The reconstruction is harder than the derivation would have been, because the program contains implementation decisions that were made without being constrained by the proof. You are now trying to discover whether those unconstrained decisions happen to be correct. The probability that they all are, in a program of non-trivial size, is remote.

The practical consequence: programs derived from their proofs are reliably shorter and clearer than programs written forward from intuition, because the proof forces you to eliminate everything that is not necessary to establish the postcondition.

The mechanism Dijkstra proposed for this derivation is the weakest precondition calculus. For any program statement and desired postcondition — a logical formula describing what must be true after the statement executes — the weakest precondition is the least constrained precondition that guarantees the statement will terminate in a state satisfying the postcondition. The calculus provides transformation rules for each construct in the language. To derive a program, you start from the postcondition and work backward through the rules until you reach a precondition you can satisfy. The program that emerges from this process is correct by construction. It cannot be otherwise, because it was built to satisfy the proof at each step.

The language Dijkstra designed for expressing these derivations is deliberately minimal — alternation and repetition with guarded commands, no recursion, no complex features. His justification is instructive: "The point is that I felt no need for them in order to get my message across, viz. how a carefully chosen separation of concerns is essential for the design of in all respects, high-quality programs: the modest tools of the mini-language gave us already more than enough latitude for nontrivial, yet satisfactory designs." The language is minimal because minimality is the point. Every construct you add to a language expands the space of programs that can be written and therefore the space of programs that must be verified. A smaller language makes the verification task smaller.

Why simplicity is not an aesthetic preference

Dijkstra identified several ideas in the book that he described as "elusive" — ideas that should take root in the mind of a programmer but typically don't, because the industry treats them as matters of style rather than as structural requirements.

The first of these is simplicity. Dijkstra's argument is not that simple programs are nicer to read. It is that simple programs are the only programs for which correctness proofs are feasible. Complexity is not merely unpleasant; it is a barrier to verification. A program that is too complex to reason about is a program whose correctness cannot be established. Whether it actually works is unknown, and testing — Dijkstra's most famous observation — "reveals only the presence of errors, not their absence." A tested program that passes all its tests is a program for which no known inputs produce incorrect outputs. It is not a correct program. The distinction is not philosophical. It is the difference between a bridge that has survived every load it has encountered and a bridge that has been shown, by structural analysis, to withstand every load it could encounter.

The second is elegance. Dijkstra used the word, which makes engineers uncomfortable, but he meant something precise. An elegant solution is one in which the proof of correctness is natural — where the formal argument flows without contortion, because the program structure mirrors the logical structure of the specification. Elegance is not decoration. It is evidence that the derivation worked.

In Dijkstra's framework, simplicity, elegance, and reliability are not three properties. They are one property described three ways. A simple program is one whose correctness proof is manageable. An elegant program is one whose correctness proof is natural. A reliable program is one whose correctness proof exists.

The connection to the agent era

Agents produce programs by statistical prediction. They do not construct proofs. They do not derive programs from specifications. They generate tokens that are probable given their training data, and the programs that result from this process have no accompanying reasoning that justifies why they are correct. They might be correct. They might not be. There is no way to know from the output alone, because the output was not produced by a process designed to establish correctness.

This makes Dijkstra's argument more urgent, not less. If programs are increasingly produced by entities that cannot reason about their correctness, then the verification burden shifts entirely to the infrastructure surrounding those entities. The harness must supply the reasoning the agent cannot perform. The verification layer must establish what the generation process cannot guarantee.

Dijkstra argued that the proof must lead and the program must follow. In the agent era, the proof still has to exist. The question is who — or what — constructs it, and whether the program is constrained by it or merely inspected by it afterward. If the agent produces code and a separate system verifies it, the verification is still an attempt to reconstruct the reasoning that would have produced the program had it been derived correctly. The reconstruction is harder than the derivation. The probability that it succeeds on every change, at the speed agents generate changes, is remote.

Dijkstra's methodology does not scale to the agent era in its original form. No one is going to derive weakest preconditions for agent-generated code at review time. But the principle — that correctness must be constructed, not inspected into existence — does not become false because it becomes harder to satisfy. It becomes more expensive to ignore.

A Discipline of Programming — Edsger Dijkstra, 1976. 217 pages. The most sustained argument ever written for the proposition that simplicity and reliability are the same property.


References:

  • Dijkstra, E. W. (1976). A Discipline of Programming. Prentice-Hall. — The source text: weakest preconditions, guarded commands, and the argument that correctness proofs must lead and programs must follow.
  • Dijkstra, E. W. (1972). "The Humble Programmer." Communications of the ACM. — The earlier lecture that established the tone: programming is inherently difficult, and the only way to manage that difficulty is through disciplined simplicity.
  • Related: Correctness First — What OpenBSD teaches about correctness as the prerequisite for security.
  • Related: Taste as Conceptual Integrity — Brooks on the property Dijkstra's methodology was designed to preserve.
  • Related: In the Land of AI Agents, the Verifiers Are King — The verification imperative in the agent era.

Buzz and the Identity Problem

Block released Buzz, an open-source collaboration platform where humans and agents share a workspace. The most interesting thing about it isn't the collaboration model — it's the identity architecture.

agentsblockbuzzidentitynostropen-sourceagent-infrastructure

Block released Buzz on Monday. The pitch: an open-source collaboration platform where humans and AI agents work together in a shared workspace, built on the Nostr protocol, licensed under Apache 2.0, self-hostable or available managed at buzz.xyz. The source is at github.com/block/buzz.

The surface-level story is that a major company shipped an open alternative to the proprietary agent platforms currently being built inside every cloud provider and VC-backed startup. That story is accurate as far as it goes. But the thing that makes Buzz interesting is not the collaboration model. Shared workspaces for humans and agents are becoming table stakes — every chat platform will have them within two years. The interesting thing is the identity architecture, because it solves a problem most teams haven't realized they have yet.

The identity problem

When an agent acts inside your infrastructure today, who is it? The answer is usually a variant of "a service account tied to a platform vendor's API key." The agent's identity is not its own. It is leased from whichever provider generated the token. If you switch models — from OpenAI to Anthropic, from one agent harness to another — the agent's identity does not travel with it. Its history, its permissions, its reputation within the team: all of that is bound to the specific API key and platform that provisioned it. This is not a technical limitation. It is an architectural choice made by every agent platform so far, and it has consequences.

The most immediate consequence is that there is no portable answer to the question "which agent made this change?" If a production incident traces back to a commit authored by an agent, the team needs to know which agent, running which model, under which constraints, with which permissions, at whose instruction. In a platform where agent identity is an API key, the answer is: the agent that held the key at that moment. If the key rotated, or the team switched providers, or someone reconfigured the agent harness between incidents, the identity trail breaks. That is acceptable for a chat bot. It is not acceptable for an agent that deploys code.

The second consequence is that there is no basis for reputation. If agents are going to operate with increasing autonomy — reviewing code, approving merges, managing infrastructure — the humans who work alongside them need to calibrate trust. Which agents are reliable? Which make specific kinds of mistakes? Which have access to what? These questions require stable identity. You cannot build a reputation system on top of rotating API keys.

The third consequence is vendor lock-in, which is the one Block's announcement emphasizes most directly. If your agents' identities are owned by a platform, migrating off that platform means destroying those identities and recreating them elsewhere. The switching cost is not the difficulty of moving prompts and configurations. It is the loss of the accumulated context that makes the agents useful.

How Buzz solves it

Buzz is built on Nostr, a protocol designed for decentralized social networking. The relevant property of Nostr for Buzz's purposes is that every participant — human or agent — holds a cryptographic keypair. The keypair is not issued by Buzz. It is not issued by any platform. It is generated by the participant and can be used across any Nostr-compatible system.

What this means in practice: an agent configured in Buzz has an identity that is cryptographically its own. It authenticates by signing events with its private key. Its messages, its actions, its contributions — all are verifiable as having originated from that specific agent, regardless of which model is behind it, which harness is orchestrating it, or which relay is delivering its messages. If a team moves from Block's managed hosting to a self-hosted instance, the agent's identity moves with them. If a team switches the model powering the agent from Claude to Gemini, the identity persists. The identity is not a feature of the platform. It is a property of the agent.

This has practical implications that go beyond portability. An agent that commits code can sign the commit with its own key. An agent that reviews a pull request leaves a review that is cryptographically attributable. An agent that participates in an incident response leaves a record that can be audited. The trail does not break when the API key rotates, because the identity is not the API key.

Buzz's architecture separates agent identity from model access. The identity is a keypair. The model is a configuration choice. They are independent axes. Most agent platforms today conflate them.

What's genuine and what's marketing

Every product announcement makes claims. It is worth distinguishing which parts of Buzz's proposition are structurally unique and which are well-executed versions of ideas that are becoming standard.

Structurally unique: the identity layer. No other major agent platform gives agents portable, cryptographic identities that are independent of the platform. This is not a feature Buzz added to an existing architecture. It is a consequence of building on Nostr, which was designed to solve exactly this problem for a different domain. The decision to use a decentralized protocol for agent identity is novel, and it has downstream consequences — for auditability, for reputation, for portability — that are not achievable within a platform-owned identity model. Whether Nostr is the right protocol for this use case is a separate question, but the architectural choice is genuinely different from what exists.

Well-executed but not unique: the collaboration model. Shared channels, threads, direct messages, voice, code repositories — this is a well-designed workspace, but the category is crowded. What distinguishes Buzz's execution is that agents are first-class participants rather than integrations bolted onto a human platform, but that distinction will erode as every collaboration tool adds agent support. The structural advantage is the identity layer. The UX is good but replicable.

Strategically significant: open source from a major company. Block is not a startup. It is a public company with a market cap in the tens of billions, the parent of Square and Cash App. When a company of that scale releases an agent platform under Apache 2.0 with a self-hosting option, it changes the economics for every startup building proprietary agent collaboration tools. Free and open source from a credible maintainer resets the floor for what teams will pay for. The fact that Block built this on an open protocol rather than a proprietary stack makes the bet even clearer: they are betting that the infrastructure layer for agent collaboration should be a public good, not a proprietary platform. Whether that bet pays off depends on adoption, but the direction of the bet is legible.

Buzz and GitButler: two layers of the same problem

Buzz was not the only significant agent infrastructure release this year. GitButler, the Git client founded by Scott Chacon (GitHub co-founder, author of Pro Git), has been quietly building infrastructure for the same problem from a different layer of the stack.

Where Buzz addresses the collaboration layer — who is working on what, how do agents communicate, how is identity established — GitButler addresses the version control layer: how do you manage code when multiple humans and multiple agents are changing it simultaneously?

The core innovation is virtual branches. In traditional Git, you work on one branch at a time. Switching branches means checking out a different set of files — context-switching your working directory. Git worktrees mitigate this by giving each branch its own directory, at the cost of disk space and coordination overhead. GitButler replaces both models. Multiple branches are applied simultaneously within a single working directory. Files from different branches coexist in the same workspace. There is no checkout. There is no context switch.

This matters for agents because agents produce code in parallel. An agent fixing a bug, an agent implementing a feature, and an agent refactoring a module can all operate on the same repository at the same time, each on its own virtual branch, without conflicting at the filesystem level until their changes actually overlap. GitButler detects conflicts immediately — not at merge time — because all branches share a single working tree.

The integration with agents is deeper than the branching model. Each virtual branch can be bound to an independent agent session. GitButler provides hooks for Claude Code and Cursor that automatically route file edits to the correct virtual branch and handle commits when a session ends. A single command — but rub — handles assign, move, squash, and amend operations, which is how you post-hoc organize files into branches when multiple agents have been working across multiple concerns.

GitButler's documentation describes seven multi-agent collaboration patterns: parallel feature development, sequential handoff between agents, cross-agent commit transfer, agent code review cycles, agent swarms on a shared branch, exploratory development (multiple approaches in parallel, keep the winner), and emergency hotfix without disturbing ongoing work. These are not theoretical. They are documented patterns that the tool's primitives make possible.

The comparison that matters. Buzz and GitButler are not competitors. They are complementary infrastructure at different layers. Buzz gives an agent a portable identity and a place to communicate with humans and other agents. GitButler gives that agent a way to produce and manage code alongside other agents without stepping on each other's work. Together they sketch what agentic software engineering infrastructure looks like: identity at the protocol layer, virtualized branching at the version control layer, agents as first-class participants at both. The verification layer — determining whether any of this agent-produced code is actually correct — remains the hardest piece of the stack, and the subject of a separate discussion.

Buzz and GitButler are not the final form of agent infrastructure. They are early, credible bets on what the stack needs to look like. Buzz bets that the collaboration layer — identity, communication, shared context — should be open at the protocol level, with agents holding their own cryptographic identities rather than leasing them from a platform. GitButler bets that the version control layer needs to be rethought from first principles for a world where multiple agents produce code in parallel, and that virtualized branching is the right abstraction. Neither bet is proven. Both are made by people with the track record to be taken seriously — Block on one side, Scott Chacon on the other.

What neither addresses is verification. An agent with a portable identity can produce coherent-looking code on a virtual branch. The identity tells you who. The branch tells you how the work was managed. Neither tells you whether the change is correct within the context of the whole system. That is the verification layer, and it remains the hardest piece of the stack.

The test for agent infrastructure is not whether agents can participate. It's whether you can answer "who did what, why, and was it correct?" six months after the fact, after rotating credentials, switching models, and changing platforms. Buzz and GitButler get you closer to an answer than anything else currently available. But the "was it correct?" column is still empty.


References:

Taste as Conceptual Integrity

When engineers say someone has "taste," what they mean is that person can perceive conceptual integrity. Fred Brooks spent a career explaining why that is the most important property of any designed thing.

designfred-brooksconceptual-integritytasteengineering-judgment

Ray Myers posted a thought experiment on LinkedIn this week:

"Imagine a phrase like 'The building fell down after the inspector's taste was ignored.' Is there any situation where you would feel accountable to heed someone's taste?"

Ray's experiment is effective because the sentence he asks us to imagine contains its own refutation. If a building collapses and the subsequent investigation reveals that an inspector objected and was overruled, we do not describe what happened as the inspector's taste being ignored. We describe it as the inspector's professional judgment being overridden. The word "taste" does not merely fail to capture the gravity of the situation — it actively miscategorizes it. It moves the event from the domain of engineering accountability, where the question is whether the objection was correct, to the domain of aesthetics, where the question is whether the objection was agreeable. These are different categories of judgment, and conflating them is a category error.

The reason the word persists despite this flaw is that it gestures at something real. When a senior engineer looks at a proposed change and says "this does not feel right," they are not stating a preference. They are reporting a perception. They have detected something about the system before they have isolated what it is or produced evidence for its existence. The question is what, exactly, they are detecting.

The most precise answer was given by Fred Brooks. The property being perceived is conceptual integrity.

The property

The Design of Design — Fred Brooks's final book, and his least read

Brooks defined conceptual integrity as the property of a system that feels as though one mind designed it. In The Mythical Man-Month (1975) he wrote:

"I will contend that conceptual integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas."

Thirty-five years later, in The Design of Design, he reduced the claim to its most concentrated form:

"Most great works have been made by one mind. The exceptions have been made by two minds."

Neither statement is a claim about aesthetics. Both are structural claims about the conditions under which coherence enters a designed object. Brooks's argument, developed across both books, is that design does not parallelize. Every additional mind added to a design introduces assumptions that differ from the assumptions already present. Those assumptions must be reconciled. Reconciliation requires compromise. Each compromise reduces coherence, because the original set of design ideas was internally consistent and the compromise introduces an element that was not part of that set. Brooks put the point economically: "Many hands make light work — Often. But many hands make more work — Always." You can distribute the labor of implementation across as many people as the coordination overhead permits. You cannot distribute the labor of deciding what the thing is. The design must proceed from one point of decision, or at most two in what Brooks called genuine resonance — a pair who share a mental model so completely that either can speak for the architecture. Three is already a committee, and committees produce settlements, not coherence.

If this argument is accepted, a definition follows: taste is the ability to perceive conceptual integrity. It is the faculty by which an engineer detects that a system has maintained coherence or lost it — that the parts compose or have begun to diverge, that the whole still speaks with one voice or has acquired a second. The senior engineer who says "this does not feel right" is perceiving a violation of conceptual integrity before they can name the specific violation. The feeling is not the argument. It is the reason to begin looking for one.

The positive case

Brooks's canonical example is Reims Cathedral. Ground was broken in 1211. Structural work was completed by 1275, and decorative work continued into the 1460s. Across those two and a half centuries, four master masons — Jean d'Orbais, Jean-le-Loup, Gaucher of Reims, and Bernard de Soissons — directed the construction. Their names were inscribed in a labyrinth set into the nave floor. The labyrinth was not a signature in the modern sense; it was a public oath to a design larger than any single lifetime. Each man bound himself to the constraints established by his predecessor, and each successor did the same.

The coherence of the result was not achieved by consensus. It was achieved by constraint. Reims was among the first buildings to use stones of standardized dimensions, which reduced the degrees of freedom available to each successive builder. The structural system — four-part rib vaults producing arcades of identical pillars rather than the alternating pillars and piers of earlier Gothic — made the architectural rhythm self-enforcing. Once the pillar spacing and vault geometry were fixed, any architect who altered them would have broken the structural logic of the building. The plan had mechanical authority. It did not require that each successor agree with it; it required only that each successor could not alter it without producing visible damage.

The result is a building in which you cannot identify the transition from one architect to the next. Two centuries of evolving Gothic fashion — bar tracery in the transept roses giving way to full Rayonnant in the west façade — read not as competing visions but as variation within a theme. The first architect's taste set the constraints. The constraints outlived him because they were structural, not advisory.

Reims Cathedral — four architects, two and a half centuries, one coherent result

The negative case

If Reims demonstrates what happens when taste has authority, the Bradley Fighting Vehicle demonstrates what happens when it does not.

The Bradley began as a light troop carrier. Over 17 years and at a cost of $14 billion, successive groups of stakeholders added requirements. Armor advocates wanted more survivability. Infantry commanders wanted more troop capacity. Generals wanted more firepower. Contractors wanted larger contracts. Each of these requests was defensible when evaluated against the objectives of the stakeholder who made it. The problem is not that any individual request was unreasonable. The problem is that there was no single mind with the authority to evaluate each request against the coherence of the whole — to say, this feature is reasonable on its own terms, and it will break the design, so the answer is no.

The result was a vehicle that Sergeant Fanning, a character in the 1998 HBO film The Pentagon Wars, describes with the precision of someone who has spent years watching the logic play out:

"A troop transport that can't carry troops, a reconnaissance vehicle that's too conspicuous to do reconnaissance, and a quasi-tank that has less armor than a snowblower, but has enough ammo to take out half of D.C."

The Pentagon Wars (1998) — 17 years, $14 billion, and a machine that did nothing well

Every clause in that sentence corresponds to a stakeholder requirement that was defensible when considered in isolation. Together they describe a machine that performed no role adequately because it was required to perform every role simultaneously. The film documents that live-fire testing was manipulated to conceal the vehicle's failures; that the officers responsible for the manipulation were promoted; and that Lt. Colonel James Burton — the one person in the story who acted as though coherence mattered, who forced an honest test over the objections of his chain of command — was eventually forced into retirement.

The Bradley is the limiting case of what Brooks described as "many good but independent and uncoordinated ideas." It was not produced by incompetence or bad faith. It was produced by a structural condition: a decision-making process in which every stakeholder could add a requirement and no single mind could reject one. The output was not a design. It was a negotiated settlement. And it is the reason the word "taste" must be defined precisely. If taste means personal preference, then ignoring it is reasonable. If taste means the perception of conceptual integrity, then ignoring it is how you produce the Bradley.

The organizational conditions

Brooks did not stop at naming the property. On IBM System/360, he, Gene Amdahl, and Gerrit Blaauw implemented the organizational structures required to preserve it. The principles are distributed across both books. Considered together, they form a set of necessary conditions.

One mind, or two in genuine resonance. Brooks and Blaauw achieved a working relationship in which either could speak for the architecture — a state Brooks described as resonance and glossed with what may be the most human observation in either book: "And two is indeed a magic number for collaborations; marriage was a brilliant invention and has a lot to be said for it." Three people cannot maintain this state. At three, the dynamic shifts from resonance to negotiation, and negotiation produces settlements rather than designs.

Real veto power. The architect must be able to refuse a feature and have the refusal stand. Brooks described this as saying no "repeatedly, to smart people with good arguments." Advisory veto — in which the architect may object but a person higher in the reporting chain may override the objection — is indistinguishable in practice from having no veto at all. The vice president approves the feature. It ships. The coherence of the system degrades. The vice president eventually moves to a different organization. The architect remains, responsible for the accumulated consequences of decisions they were not permitted to make.

The separation of design from implementation. Brooks devoted a chapter of The Design of Design to what he called the divorce of design — the progressive separation, beginning around the 16th century, of the act of specifying from the act of making. On System/360, this principle was operationalized as a small architecture team that defined what the system was and a large implementation team that built how it worked. "The architecture team must be protected; the implementation team must be coordinated." Organizations that assign both functions to the same people discover that neither is performed adequately. The architects, drawn into the demands of implementation, cease to think about systemic coherence. The implementers, asked to make architectural decisions, optimize for the local context at the expense of the whole.

Protection from organizational forces. The System/360 architecture team was deliberately insulated from field sales, who wanted features for specific customers; from engineering, who wanted optimizations that would have compromised the cleanliness of the abstractions; and from customers, who demanded backward compatibility with their existing systems. Each of these demands was legitimate. Accommodating any significant fraction of them collectively would have destroyed the system's coherence. The insulation was not a privilege extended to the architects. It was a structural precondition for the design work to be possible at all.

Career paths that reward refusal. Brooks implied this condition throughout both books without stating it as directly as the evidence warrants. In most organizations, agreeing to requests accumulates political capital and refusing them spends it. If the incentive structure penalizes the person who protects coherence, then taste may exist within the organization — individual engineers may perceive what is happening to the system — but it will be systematically powerless. The Bradley provides the limiting case: the officers who manipulated the tests were promoted; the officer who forced an honest evaluation was removed from service.

Open questions

Brooks's conditions assume a human designer producing at human speed. Coding agents break that assumption. They generate tokens that are statistically probable given their training distribution and context window. They do not generate coherent designs, because coherence across the entire surface area of a system is not a statistical property — it is a structural property that must be imposed by an entity capable of holding the whole in view. An agent can write a function that compiles and passes its tests while violating the design philosophy that every adjacent function observes. It does so without awareness, because the design philosophy is not legible in the text the agent was trained on. The rationale lived in the head of the person with the authority to say no.

This raises questions Brooks did not live to address:

Does taste become the irreducible human contribution, or does the role become impossible? An architect can veto a human engineer's output at the speed of code review. An architect cannot veto at the speed of token generation. If an agent produces a Bradley-scale incoherence in an afternoon — and the Bradley took 17 years — maintaining conceptual integrity may require constraints so deeply embedded in the generation pipeline that the architect's role shifts from reviewer to tool-builder. Whether that role still satisfies Brooks's definition of the designer is an open question.

Can taste be automated? If taste is the ability to perceive violations of conceptual integrity, and conceptual integrity is a property of the system considered as a whole, then automating taste requires building a system capable of forming a model of the whole. Current agents cannot do this. It is unclear whether the limitation is architectural — larger context windows, better retrieval — or categorical. If coherence can only be perceived by holding the entire set of design decisions in view simultaneously, and the set grows faster than any context window can expand, then taste may be the capacity that survives automation because it is the capacity that automation cannot reach.

If an agent generates 500 lines and a human keeps 30, who wrote the feature? The question is not rhetorical. If the human's contribution is selecting which output maintains the system's coherence, then the job is not prompt engineering. It is architecture in Brooks's sense: one mind enforcing conceptual integrity against a force that produces incoherence at speed. The title matters because the authority matters. You cannot enforce coherence if your role is understood by the organization as "the person who writes the prompts."


Ray's experiment contains its own answer, provided the term is defined correctly. "The building fell down after the inspector's taste was ignored" fails because it miscategorizes professional judgment as personal preference. Reformulate: the Bradley became a death trap because the people capable of perceiving conceptual integrity lacked the authority to enforce it. That sentence identifies a faculty, a property, and a failure mode. It describes an event for which someone can be held accountable.

Brooks supplied the property, the evidence, and the organizational conditions under which the faculty can function. What he could not supply — what no one can yet supply — is an account of whether taste, defined this way, survives an era in which code is generated faster than any human can perceive whether it coheres. That question is now operational. We will have an answer whether we want one or not.


References:

Verification Is the Bottleneck

The headline finding from Martin Fowler's Future of Software Development Retreat: "Code generation is no longer the bottleneck — verification is." The industry's most respected voice just validated the thesis.

verificationharness-engineeringmartin-fowleragentssoftware-engineeringcorrectness

Martin Fowler published his notes from the second Future of Software Development Retreat today. The retreat gathers senior engineers and executives from across the industry to assess where software development is heading. Fowler's notes are worth reading in full. The headline findings are worth sitting with, because they converge on a thesis this blog has been building across multiple posts.

The Thoughtworks report from the retreat surfaced five headline findings. Three of them are directly about the topics we've been covering:

"Code generation is no longer the bottleneck — verification is."

This is the exact argument from the Sonar AC/DC post: the verifiers are king. It's the argument from the harness engineering post: evaluation infrastructure matters more than model intelligence. It's the argument from the data-driven design post: you can't improve what you don't measure. And now it's the headline finding from a room full of industry leaders at a retreat convened by the most respected voice in software engineering.

When a thesis shows up independently in product form (Sonar's AC/DC), in academic framing (harness engineering as a discipline), and in Fowler's retreat consensus, it's no longer a prediction. It's a description of where the industry is now.

"'Harness engineering' is emerging as a distinct, ownable discipline."

This blog has been arguing for harness engineering as a first-class discipline since May 2026. Fowler's retreat now names it as an emerging field. The recognition matters because "distinct, ownable discipline" is the language of organizational design. When something becomes a distinct discipline, it gets budget, headcount, career tracks, and dedicated tooling. Harness engineering crossing that threshold means the teams that already have harness engineers are ahead, and the teams that don't are about to discover they need them.

"Legacy modernization is the clearest, most defensible near-term value pool."

This is less obvious but equally important. The retreat found that the most defensible use case for AI coding agents isn't greenfield development — it's modernizing legacy systems. Agents can ingest old codebases, understand their structure, and systematically refactor them in ways that would take human teams years. The value is measurable, the risk is bounded (the old system already works, so you can compare), and the alternative cost is known (maintaining the legacy system indefinitely).

The other two findings are about organizational dynamics rather than technical architecture, but they're equally revealing:

  • "Organizations are colliding with a real apprenticeship crisis." If agents write the boilerplate and seniors only review, how do juniors learn? The apprenticeship pipeline that produced every senior engineer reading this post is being disrupted in real time, and nobody has a convincing answer for what replaces it.
  • "The executive/engineer expectation gap is a bigger risk than any technical limitation." Boards see 3-5x productivity claims and expect headcount reduction. Engineers see the bugs, the security issues, and the unrequested features agents insert. The gap between those perspectives is wider than any model capability gap.

The most telling detail

Fowler shares a story about a team that spent three days investigating a feature an agent inserted that nobody asked for. Three days trying to figure out who requested it, what it was supposed to do, and whether anyone wanted to keep it. Three days of engineering time consumed by code that should never have been written.

This is the productivity paradox in microcosm. The agent saved someone 30 minutes of typing and cost the team three days of investigation. Net productivity: negative.

Fowler also notes that agents don't learn. The best they can do is update context. Every mistake an agent makes, it will make again unless the harness explicitly prevents it. Every unrequested feature it inserts, it will insert again unless the guide constraints explicitly block it. The agent is not getting better on its own. The harness has to get better around it.

The law professor experiment

One of the most striking anecdotes in Fowler's notes is unrelated to software engineering but deeply relevant to how we should think about LLM output. Law professors evaluated answers to contract law questions — some written by professors, some by LLMs. The professors rated LLM answers higher than their peers 75% of the time. LLM answers were flagged as harmful 3.5% of the time. Human professor answers: 12%.

LLMs outperform domain experts on short-form domain questions judged by those same domain experts. And they produce fewer actively harmful answers.

This doesn't mean LLMs should replace law professors. It means the quality bar for agent-generated output is higher than most skeptics assume — and the error rate of human experts is higher than most humans assume. Verification isn't just for AI output. It's for all output. The difference is that humans have been dodging systematic verification for decades, and agents make the need undeniable.

DSLs as the bridge

Fowler discusses work by Unmesh Joshi and Spencer Nelson on using Domain-Specific Languages as an interface layer between LLMs and systems. The idea: design a small, token-efficient language with hard security boundaries. The LLM generates DSL code. A deterministic runtime executes it. The DSL constrains what the LLM can express, which constrains what it can break.

This is pledge(2) for agents — a restricted interface that makes the wrong thing impossible. If the DSL has no way to express "drop this table," the agent cannot drop a table, no matter how badly it hallucinates.

Fowler notes that LLMs have historically lowered the barrier to building DSL parsers and tooling, which was the main obstacle to DSL adoption. But his more interesting point is that the DSL is just a projection of a semantic model — and the semantic model is what matters. LLMs may let us explore new ways to project those models.

The LLM voice problem

Fowler closes with a personal reflection on what he calls "LLM miasma" — the recognizable stylistic signature of AI-generated prose that triggers what he describes as intellectual nausea. He's reversing his earlier position that non-professional writers should use AI to polish their prose. The LLM voice is now so pervasive that it discredits writing before the reader engages with the content.

His antidote: read your drafts aloud. Speech patterns are closer to your authentic voice. If it doesn't sound like something you'd say, rewrite it until it does.

This matters for agent-era engineering in a way that isn't immediately obvious. If LLM-generated code comments, commit messages, and documentation all carry the same stylistic signature, teams will develop the same antibodies Fowler describes. They'll stop reading. The code will compile but the knowledge transfer won't happen. Authentic voice isn't just a writing concern — it's a knowledge-sharing concern in a world where agents produce most of the text.

What this means

Fowler's retreat didn't produce new ideas. It produced independent confirmation. The people in that room — senior engineers, executives, thought leaders — converged on the same conclusions that teams on the ground have been reaching through trial and error: verification is the bottleneck, harness engineering is the discipline, legacy modernization is the beachhead, and the gap between what executives believe and what engineers experience is enormous.

If you're building agent systems, these findings are your strategy document. Verification infrastructure. Harness engineering as a dedicated function. Legacy systems as the proving ground. Closing the expectation gap with data, not demos. And writing — code, docs, commits — in a voice that doesn't trigger the antibodies.

The industry's most respected voice just said what this blog has been saying for months. The difference is that when Martin Fowler says it, organizations listen.


References:

The Market Is a Sandpile

Quantitative finance borrowed its math from equilibrium physics. Markets are non-linear complex systems with feedback, phase transitions, and power-law tails. Those are not the same thing.

quantitative-tradingdynamical-systemscomplexity-theoryfinancenon-linearrisk

Most quantitative trading is built on a lie: markets are in equilibrium, returns are Gaussian, linear models are enough. These assumptions hold in the boring middle of the distribution, where nobody makes or loses serious money. They break at the tails, which is where everything interesting happens.

The efficient market hypothesis is not wrong because markets are irrational. It's wrong because markets are complex adaptive systems. Rational agents interacting under imperfect information produce emergent dynamics that no individual agent intended or can predict.

The sandpile and the market

In 1987, Per Bak, Chao Tang, and Kurt Wiesenfeld dropped grains of sand onto a table, one at a time, and measured the resulting avalanches. Most grains did nothing. Occasionally, a single grain triggered a cascade that reshaped the entire pile. The size distribution of avalanches followed a power law — no characteristic scale, no "typical" event, no upper bound.

This is self-organized criticality: complex systems naturally evolve toward a critical state where extreme events are not anomalies. They are the natural output. The system doesn't need a big cause to produce a big effect. A grain of sand can trigger an avalanche. A marginal liquidity withdrawal can trigger a flash crash. The mechanism is the same.

5-sigma events are not once-in-a-lifetime flukes. They are the sandpile doing what sandpiles do. Gaussian Value at Risk is a calculation that assumes the system is a different kind of system than it actually is.

Markets are dynamical systems with feedback

Prices are not independent draws from a distribution. They are the output of a coupled, non-linear, feedback-driven process. Buyers and sellers observe prices, update beliefs, place orders, change prices, which causes other participants to update beliefs and place different orders. The output at time t becomes an input at t+1.

In linear systems, feedback is well-behaved. In non-linear systems, feedback produces regimes, bifurcations, and chaos. Markets are non-linear systems with feedback.

The properties that matter:

  • State dependence. A stock at $100 in a calm trend has a different future than the same stock at $100 after a 20% drawdown. Same price, different state, different distribution. Linear models can't tell the difference.
  • Phase transitions. Markets don't drift between regimes — they reorganize. The shift from bull trend to liquidation cascade is sudden, discontinuous, and preceded by specific signals: rising correlation, thinning liquidity, increasing skew. The math that describes water freezing describes a market crash.
  • Emergence. No trader intends to produce a bubble. Bubbles emerge from thousands of individually rational decisions. The macro pattern is real. It has no author. You cannot understand it by interviewing participants.
  • Adaptation. When enough traders adopt a strategy, the strategy changes the market in a way that reduces the strategy's edge. Alpha decay is co-evolution: predator and prey evolve together. What worked yesterday stops working not because it was wrong, but because it was right enough to change the environment.
  • Non-ergodicity. The time average of a strategy is not its ensemble average. A strategy with positive expected return can ruin you if you don't survive the drawdowns. In a complex adaptive system, the path you take determines the distribution you sample.

What this means for your trading

The implications are not academic. They change what you build:

Regime detection is the most important problem. Knowing which attractor basin you're in — and detecting the signals of a phase transition — beats predicting the next tick. A mediocre signal in the right regime beats a great signal in the wrong one.

Linear models are components, not systems. PCA, Kalman filters, regression — they have their place. But they belong inside a non-linear wrapper that tells you which regime you're in. The linear model describes what happens inside the regime. The wrapper tells you when the regime changes.

Abandon Gaussian VaR. Use extreme value theory. Model tails with generalized Pareto distributions. Stress-test against power-law cascades, not historical scenarios. The next crash will not look like the last one. It will look like a phase transition.

Treat alpha decay as a dynamical system. It's not erosion. It's a predator-prey model with equilibria, cycles, and extinction regimes. Model it like one.

Backtests are single draws from a non-stationary process. The market of 2018 is not the market of 2026. The participants changed. The attractor basins shifted. A backtest is evidence, not proof.


Quantitative finance borrowed its toolkit from 19th-century physics because those tools produce closed-form solutions, not because they describe markets. Complexity science offers a more accurate description at the cost of fewer closed forms and more simulation. The cost is worth paying.

Markets are not physical systems with fixed laws. They are complex adaptive systems with emergent dynamics. If your trading system doesn't account for this, you are not modeling markets. You are modeling a textbook.

Where to learn this

EACH-USP (University of São Paulo) offers a graduate program in Complex Systems Modeling that covers exactly this ground: dynamical systems, agent-based modeling, network theory, and applications to finance. If you trade systematically and want the mathematical foundations this post argues for, it's worth a look.

Modelagem de Sistemas Complexos — EACH-USP


References:

  • Mandelbrot, B. & Hudson, R. (2004). The (Mis)Behavior of Markets: A Fractal View of Financial Turbulence. — The foundational argument that financial markets follow fractal geometry and power laws, not Gaussian random walks.
  • Bak, P., Tang, C. & Wiesenfeld, K. (1987). "Self-Organized Criticality: An Explanation of 1/f Noise." Physical Review Letters. — The sandpile model: how complex systems self-organize into critical states where avalanches of any size are the natural output.
  • Arthur, W. B. (2014). Complexity and the Economy. — The Santa Fe Institute economist on markets as complex adaptive systems: emergence, non-equilibrium, and why rational agents don't produce equilibrium.
  • Peters, O. (2019). "The Ergodiity Problem in Economics." Nature Physics. — Why the time average and ensemble average are not the same, and why that matters for every risk model ever built.
  • Taleb, N. N. (2007). The Black Swan: The Impact of the Highly Improbable. — The practical consequences of assuming Gaussian distributions in a fat-tailed world.
  • Hamilton, J. D. (1989). "A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle." Econometrica. — The Markov-switching model that introduced formal regime detection to economics.
  • EACH-USP. Modelagem de Sistemas Complexos — Graduate program in Complex Systems Modeling at the University of São Paulo.

Agents Are Too Stochastic for Intuition

You can't think your way to a better agent prompt. The only reliable design method for stochastic systems is measurement, experimentation, and data — the same method that turned databases from black boxes into predictable infrastructure.

data-driven-designagentsevaluationharnesssoftware-engineeringmetrics

Databases used to be black boxes. You wrote a query, waited, and hoped the optimizer did something reasonable. When it didn't, you tried a different index and hoped again. The expertise was real but the method was folk — passed from senior to junior as rules of thumb: "always index foreign keys," "avoid SELECT *," "EXPLAIN will tell you what's wrong."

That era ended when database teams started treating query performance as a measurement problem. Instrument the optimizer. Collect query plans. Compare actual vs. estimated row counts. Build dashboards. The output of all that instrumentation is a design loop: you don't guess why a query is slow. You look at the data. The data tells you.

We are at the beginning of the same transition for software engineering agents. Right now, most agent design is folk expertise: "be specific in your prompts," "few-shot examples help," "chain-of-thought improves reasoning." These are all true, sometimes, for some models, on some tasks. The problem is that nobody knows which ones.

The defining quality of an LLM is that it is stochastic. Given the same input, you get different outputs. Given slightly different inputs, you get very different outputs. The only way to reason about a stochastic system at scale is with data.

Why intuition fails

Your intuition about what makes a good agent prompt is shaped by a few dozen interactions. You try a prompt, it works, you generalize. You try another, it fails, you adjust. This is the scientific method at N=1 — useful for generating hypotheses, useless for drawing conclusions.

The problem compounds because agents are not just stochastic — they are stateful, tool-using, multi-turn stochastic systems. An agent that picks the wrong tool on step 2 fails on step 7 in a way that looks unrelated to the original mistake. Intuition attributes the failure to the step-7 behavior. The data would tell you to fix step 2.

Debugging agents by reading transcripts is like debugging a database by reading query text. You might spot the obvious problems. You will miss everything that matters at scale.

What data-driven design looks like for agents

The method is straightforward, even if the infrastructure isn't:

One: Instrument every decision. The agent chose a tool. Was it the right tool? The agent produced a diff. Did it compile? Did the tests pass? Did it introduce a vulnerability? The agent looped. How many iterations? When did it converge — or fail to? Every one of these is a data point. Collect them.

Two: Define success numerically, not narratively. "The agent handled the task well" is a narrative. "The agent solved 73% of tasks in the benchmark, with a median of 4 tool calls per task and a 6% hallucinated-tool-call rate" is data. Narratives are for demos. Data is for design.

Three: Compare at scale. You changed the prompt. Did the solve rate go up? By how much? Across how many tasks? With what variance? A single run of a stochastic system is a data point. A thousand runs is a distribution. Design decisions should be made on distributions, not data points.

Four: Close the loop. The best agent systems — like the best database administrators — don't make one-off optimizations. They build pipelines. Deploy. Observe. Compare. Improve. Deploy again. The loop is the design process. Speed of iteration matters more than brilliance of a single change.

The harness is your measurement instrument

A theme that runs through this blog is that the harness is the most under-invested part of an agent system. Data-driven design makes it concrete why: the harness is your measurement instrument. If your harness can't produce reliable, comparable metrics at scale, you cannot do data-driven design. You are back to intuition and anecdotes.

A harness that can only tell you "pass" or "fail" is a thermometer with one marking. You need to know what happened, where it went wrong, and whether this run was better than the last run in a way that generalizes.

The harness engineering practices that matter for data-driven design:

  • Deterministic replay — if you can't reproduce a run, you can't measure a change.
  • Layered observability — prompt selection, tool choice, state transitions, final outcome. Each layer produces its own metrics.
  • Statistical rigor — a 2% improvement on 10 tasks is noise. A 2% improvement on 1,000 tasks might be real. The harness needs to support sample sizes that make signals distinguishable from noise.

Why this matters more for agents than for any previous software

Traditional software is deterministic. You change the code, you run the tests, you observe a binary outcome. The design loop is: implement, test, fix, repeat. Data helps, but the feedback signal is strong enough that intuition often suffices.

Agents are fundamentally different. The same agent, on the same task, with the same prompt, will produce different results on different runs. The space of possible behaviors is too large to explore by reasoning alone. The only way to understand an agent is to observe it across many runs and let the patterns emerge from the data.

This is uncomfortable. It means admitting that you cannot fully understand the system you are building by reading the code and the prompts. It means ceding authority to measurement. It means designing experiments instead of designing solutions — because the solution emerges from the experimental data, not from your head.

The transition from folk expertise to data-driven design is the transition from craftsmanship to engineering. Both produce good outcomes. Only one scales.

The cost of not doing it

The alternative to data-driven design is design by anecdote. Someone runs the agent on a task they care about. It works. They ship. Someone else runs it on a different task. It fails. Nobody knows why. The team debates prompts instead of looking at data. Decisions are made by the most senior person in the room, not by the person with the best evidence.

This is how most teams work today. It is also how most teams will fail at building agents, because the complexity of stochastic, tool-using, multi-turn systems exceeds what any individual's intuition can model.

Data-driven design is not about having data. It's about making decisions as if the data matters more than your opinion.

Databases went through this transition. So did compilers. So did networking. Every infrastructure layer that we now treat as predictable, measurable, and engineer-able went through a phase where the experts relied on intuition and the results were inconsistent. Agents are in that phase now. The teams that instrument, measure, and close the loop will build infrastructure. The teams that rely on folk wisdom and anecdote will build demos that don't survive contact with real workloads.


References:

  • Selinger, P. G. et al. (1979). "Access Path Selection in a Relational Database Management System." ACM SIGMOD. — The paper that made query optimization a measurement problem: cost-based plan selection using catalog statistics instead of heuristics. The origin of EXPLAIN.
  • Kohavi, R., Tang, D. & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. — The modern methodology for data-driven design decisions at scale, from the team that built Microsoft and Google's experimentation platforms.
  • Sigelman, B. H. et al. (2010). "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure." Google Technical Report. — The paper that established distributed tracing as a measurement primitive, now industry standard (OpenTelemetry).
  • Kleppmann, M. (2017). Designing Data-Intensive Applications. — Chapters on observability, metrics, and the shift from intuition to measurement in distributed systems design.
  • Related: Harness Engineering: Best Practices for Reliable Agent Systems — This blog's framework for building evaluation harnesses that produce the reliable, comparable metrics data-driven design requires.
  • Related: Correctness First: What OpenBSD Teaches Agent Builders — The argument that correctness can't be eyeballed — it must be verified systematically, which requires measurement.
  • Related: In the Land of AI Agents, the Verifiers Are King — Sonar's AC/DC framework as a productization of the same principle: verification infrastructure is the measurement layer for agent quality.

In the Land of AI Agents, the Verifiers Are King

Sonar's Agent-Centric Development Cycle codifies what the best teams already discovered the hard way: without verification at every stage, AI-generated code is a productivity trap.

verificationagentssonarsoftware-engineeringcorrectnessharness

At the AI Engineer World's Fair in July 2026, Sonar CEO Tariq Shaukat gave a talk with a title that could serve as the thesis for this entire blog: "In the Land of AI Agents, the Verifiers Are King."

The argument is simple and devastating. AI coding agents deliver an initial 3-5x velocity boost. Within three months, that gain begins to evaporate. Security vulnerabilities accumulate. Bugs multiply. Code complexity rises. Technical debt compounds at machine speed. The boost was real, but the rot was faster.

The productivity paradox of AI agents: they make you faster in week one and slower in month three, because every line they wrote is a line nobody reviewed.

Shaukat's answer is a framework Sonar calls the Agent-Centric Development Cycle, or AC/DC — Guide, Verify, Solve. Three stages, each producing feedback that feeds the others, forming a continuous loop around every AI-assisted code change. The framework is partly a product pitch for Sonar's tooling. It is also, independent of the product, the right way to think about building software with agents.

Guide: context before code

The Guide phase gives agents information they wouldn't otherwise have — coding standards, architectural constraints, dependency health, semantic navigation of the existing codebase.

This sounds like prompt engineering. It's not. It's preemptive verification. Before the agent writes a line, the system has already said: these are the rules, this is the architecture, these dependencies are approved, this is how the code is structured. The agent can't violate constraints it doesn't know about. Most violations in agent-generated code aren't malice or even incompetence — they're ignorance. Guide eliminates the ignorance.

Sonar claims a 30% reduction in token consumption from context augmentation alone. The agent writes less code that needs rewriting, because it knows what correct looks like before it starts.

Verify: trust nothing, check everything

This is the center of the framework and the center of Shaukat's argument. Verification must be zero-trust — no model is inherently trustworthy, and every model has different biases and blind spots. Verification must be multi-layered — combining algorithmic analysis (data flows, control flows, known vulnerability patterns, secrets detection) with agentic analysis (intent, business logic, the "unknown unknowns" that rules-based tools miss).

The algorithmic layer catches the deterministic problems: this data flow leaks sensitive information, this control path doesn't handle the error case, this pattern matches a known CVE. The agentic layer catches the semantic problems: this function doesn't do what the comment says it does, this PR changes behavior the ticket didn't ask for, this logic contradicts the architectural decision made last sprint.

Neither layer alone is sufficient. Algorithmic verification misses context. Agentic verification hallucinates. Together, they catch what each misses.

The results Sonar cites are significant: a 44% reduction in AI-derived production outages among organizations with disciplined verification, and up to a 92% reduction in AI-induced issues at large financial institutions. One project timeline went from 10 days to 4 days — not by generating faster, but by generating fewer things that needed fixing.

Solve: fix it, then verify the fix

The Solve phase addresses what verification finds — automatically. A remediation agent generates fix suggestions in an isolated sandbox. The fix is re-analyzed. If it passes, it merges. If it doesn't, it loops back.

This closes the cycle. Guide gave the agent context to write correct code. Verify caught what slipped through. Solve fixed it and fed back into verification. The next Guide phase has better information because the system now knows what kinds of mistakes the agent tends to make on this codebase.

The three loops

What makes AC/DC more than a marketing diagram is that it operates across three nested timescales:

  • The agentic loop (inner): context → generation → analysis → refinement → re-analysis. Runs in real time, inside the agent's workflow, before anything reaches a human.
  • The CI verification loop (middle): multi-layered PR review combining algorithmic and agentic analysis with inline comments, change summaries, and architecture walkthroughs.
  • The code maintenance loop (outer): quality gates, technical debt management, and continuous remediation across the entire codebase. Keeps the codebase clean so agents operate efficiently on future changes.

The inner loop catches mistakes in seconds. The middle loop catches them in minutes. The outer loop prevents them from becoming systemic.

Why this matters beyond Sonar

AC/DC is a Sonar product framework, but the underlying idea is independent of any vendor. It's a recognition that the software development lifecycle needs to be redesigned around the agent, not the other way around.

The traditional SDLC assumes a human writes the code, a human reviews the code, and a human fixes the bugs. The agent-era SDLC assumes an agent writes the code, a verifier checks the code, and a remediation agent fixes the issues — all before a human sees it. The human's role shifts from author to governor. The human sets the constraints, defines the quality gates, and intervenes when the automated loop can't resolve something.

This is not a downgrade of the human role. It's an upgrade. It replaces "did I write this correctly?" with "is my verification infrastructure catching what matters?" The first question doesn't scale. The second one does.

The connection to harness engineering

If you've been following this blog's thread on harness engineering, you'll recognize AC/DC as a harness architecture in product form. The Guide phase is feedforward. The Verify phase is feedback. The Solve phase closes the loop. The three nested timescales map directly to the layered checks I've argued for: tool-selection, state-transition, and final-outcome evaluation.

Sonar didn't invent the idea that verification is central to agent development. They productized it, named it, and backed it with numbers. The underlying principle — that the verifier matters more than the generator — is something every agent team discovers the hard way if they don't learn it the easy way.

The numbers tell the story: without verification, 3-5x initial gains that rot. With verification, 44% fewer outages, 92% fewer issues, faster net delivery. The gap isn't about model quality. It's about whether verification is a first-class part of your development process or an afterthought.

Shaukat's title was right. In the land of AI agents, the verifiers are king. Build your verification infrastructure accordingly.


References:

Compute Travels. Data Stays.

Bacalhau inverts the cloud model: instead of moving petabytes to a central cluster, it sends compute to where data lives. That inversion is the foundation of data sovereignty — and the architecture the agent era needs.

data-sovereigntybacalhaudistributed-computeagentsedgeprivacy

The cloud won. For two decades, the answer to "where should compute happen?" was "ship the data to us." Centralize. Aggregate. Process. The result: petabytes of data flowing into a handful of regions, owned by a handful of companies, governed by laws that don't match the topology.

The cloud won, but the cloud model is breaking. Not technically — the hyperscalers work fine. But legally, economically, and architecturally, the assumption that data should move to compute is no longer true.

GDPR requires data residency. The EU AI Act layers new constraints. Countries from Brazil to India to Indonesia are writing sovereignty into law. The pipe dream of "store everything in us-east-1" is dead. The cloud didn't account for borders.

This is where Bacalhau enters. Not as a replacement for the cloud. As an inversion of its founding assumption.

What Bacalhau does

Bacalhau is an open-source distributed compute orchestrator. The idea is simple: instead of moving data across the network to a central compute cluster, you send a small job description — a Docker container, a Wasm binary, a shell script — to wherever the data already lives. The job runs next to the data. Only the results come back.

Bacalhau logo — the cod fish, Portuguese "bacalhau," a metaphor for preservation without centralization

Compute travels. Data stays. That's the whole architecture.

It's released under Apache 2.0, built by Expanso, a company founded in 2023 by David Aronchick — previously co-founder of Kubeflow at Google, head of open-source ML strategy at Microsoft Azure. The project won the Data Breakthrough Award in 2024, raised $7.5M in seed funding led by General Catalyst, and landed a strategic investment from Samsung Next. It's available on the Google Cloud Marketplace. It's a single Go binary — no cluster to bootstrap, no control plane to subscribe to.

The architecture is an orchestrator-compute model. You label nodes by region (region=eu, region=us). You submit a job with constraints. The orchestrator routes work to nodes near the data. Jobs are parallel by default — split into partitions that run independently, with isolated failure handling. If a partition fails, it retries. If the network between the orchestrator and a compute node drops, the node keeps working.

The name matters

Bacalhau is Portuguese for dried salted cod. Before refrigeration, cod was preserved by salting and drying — it could travel long distances without spoiling, sustaining entire maritime economies for centuries. The metaphor is deliberate: preserve the data where it originates. Compute can travel. Data doesn't need to.

This matters because data is heavy and data is governed. Moving a petabyte of logs to a central warehouse is expensive. Moving genomic data across national borders is illegal in an increasing number of jurisdictions. Moving medical records, financial transactions, or personally identifiable information into a third-party cloud triggers compliance obligations that most teams underestimate until an auditor shows up.

Bacalhau's answer: don't move it. Send the analysis to the data.

How it actually works

A Bacalhau deployment runs a single binary in different modes: orchestrator nodes manage job lifecycles, compute nodes execute workloads. Both can be the same machine. At the edge, a compute node might be a Raspberry Pi on a factory floor. In the cloud, it might be a VM in a specific AWS region.

Jobs are submitted declaratively or via CLI. They target data in S3, IPFS, HTTP endpoints, or local storage. The orchestrator schedules work based on node labels — region=eu, gpu=true, tier=production. The workload runs inside a Docker container or a Wasm sandbox. Output lands wherever you configure: local disk, S3 bucket, the next pipeline stage.

A partitioned job splits across N nodes. Each partition gets an index, a count, and its own slice of the data. Failures are per-partition, not per-job. A node in Frankfurt might process EU customer records while a node in São Paulo processes Brazilian records and a node in Mumbai handles Indian data — all from the same job submission, with constraints that ensure compliance.

The Genomic Data Proof Point

A 2025 academic study tested Bacalhau with IPFS Cluster and AES-256 encryption for decentralized genomic computation. The distributed architecture achieved 100% job completion under network chaos — nodes disconnecting, links dropping, partitions reforming. The centralized baseline fell apart under the same conditions.

Under ideal network conditions, the distributed setup added about 30% overhead (49 seconds vs. 37 seconds). That 12-second difference is the price of sovereignty. In the centralized case, you also move the data first — a cost the study's baseline conveniently excluded.

The paper's conclusion: a proven model for privacy-critical decentralized science collaborations, prioritizing data sovereignty and high availability over raw throughput. Twelve seconds to keep genomic data inside hospital walls.

Why the agent era needs this

Software engineering agents generate code. That's the story everyone tells. The less-told story is where they run.

An agent debugging a production issue needs access to logs. Those logs live in a specific region, governed by specific laws. An agent analyzing customer behavior needs to read data that cannot legally leave the country. An agent optimizing a factory floor needs to process sensor data at millisecond latency — waiting for a round-trip to the cloud is not an option.

The default architecture for agent platforms today is: ship everything to a central LLM provider. Your code, your logs, your database schema, your customer PII — all of it crosses the wire to a model endpoint in a jurisdiction you didn't choose.

Bacalhau suggests a different architecture: ship the agent to the data. Run the model where the data lives. The agent is a job. The data is stationary. The compliance boundary is the node label, not a legal review of every prompt.

The broader lesson

Data sovereignty sounds like a legal problem. It becomes an architectural problem the moment you try to build a real system. If your architecture requires data to move to a central location before anything useful can happen, you have already lost the sovereignty argument — all that remains is how many exceptions you'll need and how much the fines will cost.

The alternative is compute over data. It's not a new idea. MapReduce did it. Edge computing does it. Bacalhau makes it general — any workload, any data source, any execution engine, one binary, open source.

The cloud taught us to centralize. The law is teaching us to distribute. Bacalhau is infrastructure for that transition.

It's not that central compute will disappear. It's that centralization stops being the default. When data must stay where it is, the architecture follows. Compute travels. Data stays. That inversion is the foundation of sovereignty — and the infrastructure the next decade of software will be built on.

Correctness First

Linux won the world because a lawsuit froze BSD at the wrong moment. OpenBSD is what the world lost — and what agent builders most need to understand.

openbsdlinuxcorrectnesssecurityagentssoftware-engineering

In 1991, Linus Torvalds wanted a Unix-like system for his 386. BSD Net/2 had been released, but he didn't know about it. He wrote his own kernel. The next year, AT&T sued BSDi, alleging BSD contained proprietary Unix code. BSD development froze for two years. Linux, written from scratch with no legal baggage, absorbed the energy. By the time the lawsuit settled in 1994, Linux had won.

Linus later said: "If 386BSD had been available when I started on Linux, Linux would probably never have happened."

The world runs on Linux not because it was better designed, but because it was available. The cathedral lost to the bazaar by accident.

The BSD Daemon — mascot of BSD since 1976, drawn by John Lasseter before Pixar

Two architectures, two theories of quality

BSD is an operating system. One source tree. One team. The kernel, libc, core utilities, daemons, manual pages — written, reviewed, and shipped by the same people. When a kernel interface changes, every userland tool that depends on it is updated in the same commit. Configuration syntax is consistent because the same hands that wrote the daemon wrote its parser.

Linux is a kernel assembled into an operating system by distributions. The kernel comes from Linus. The C library from GNU. Userspace from a hundred independent projects. Each has its own maintainers, release schedule, coding style, and idea of what "good" means. The distribution's job is integration, not design. The result works. It is not clean. Everyone who has debugged a Linux system at 3am knows the feeling of crossing a component boundary and discovering the assumptions changed.

Linux is dirty in the way a city is dirty — it works, it's full of life, but nobody designed it from scratch. BSD is clean in the way a well-designed building is clean — the structure is visible, the materials are consistent, the wiring is labeled.

OpenBSD: correctness as the only feature

Puffy, the OpenBSD mascot — a pufferfish. Defensive, uncompromising, hard to swallow.

If BSD represents the cathedral, OpenBSD is the cathedral with the strictest building code. Theo de Raadt forked it from NetBSD in 1995 with an uncompromising thesis: correctness over features, every time.

"We are non-stop trying to find ways across our entire source tree that small little programmer errors result in problems. At some point, we have to start asking ourselves whether features are the thing, or whether quality is the issue." — Theo de Raadt

The project has audited its entire source tree — millions of lines — line by line. Multiple times. Not primarily with tools. With humans reading code, asking "what happens when this fails?" The result: an operating system with only two remote holes in the default install across nearly three decades.

This is not a security achievement. It is a correctness achievement that makes security possible.

OpenBSD's innovations read like a list of things every other OS later adopted: W^X memory (2003, now universal), pledge(2) (process-level system call restrictions), unveil(2) (filesystem sandboxing), LibreSSL (forked OpenSSL, removed half the code, broke nothing). Each was born from asking: how do we make the wrong thing impossible, not just harder?

Most projects respond to security by adding layers — a firewall, a sandbox, a scanner. OpenBSD responds by removing the bug. The difference is the difference between a house with a reinforced door and a house with no structural flaws.

The collision with the agent era

We are now deploying software that writes software — agents that generate, edit, and ship code at scale. These agents do not understand correctness. They produce tokens statistically likely to satisfy a prompt. They don't know that this return value needs checking, that this buffer needs bounds, that this error path leaks a descriptor. They pattern-match from training data that, statistically, also doesn't know these things.

Generated code is code nobody has read. Tests cover what you thought to test. Vulnerabilities live in what you didn't think of.

The agent era promises speed. OpenBSD proves that correctness at speed is an oxymoron. Shipping code faster than anyone can review it is not automation — it's an attack surface factory.

The resolution is not to reject agents. It is to demand of agent output what OpenBSD demands of human output: adversarial review, minimal diffs, restricted capabilities, and a culture that refuses to ship code that hasn't been read.

What agent builders should steal

Audit the diff, not the demo. OpenBSD doesn't audit the running kernel. It audits the source. Evaluate what the agent changed, not just whether tests pass. The diff is where bugs live.

Restrict by default. OpenBSD ships with almost nothing running. Agent platforms should ship with almost no capabilities. Every new tool is attack surface — for bugs, prompt injection, unintended behavior. An agent that can rewrite any file has maximum blast radius. An agent restricted to specific files and specific side effects can fail without destroying things.

The harness is your code review. The agent has no concept of correctness. The harness has to. If your evaluation scores "did it compile?" and "did the test pass?" and "did the reviewer LLM nod?" — you are not asking enough. Write criteria as specific as an OpenBSD code review: "this error must propagate," "this resource must be freed."

Remove generated code aggressively. A 500-line generated function that works is worse than a 30-line one that works. Every line the agent wrote is a line someone needs to read. Prefer agents that produce minimal diffs.

Culture scales. Checklists don't. OpenBSD's real innovation is cultural: correctness is everyone's job, speed is never an excuse, shipped bugs are taken personally. Agent teams need the same culture. If your team treats agent output as "good enough if the tests pass," you will ship vulnerabilities at agent speed — faster than anyone can review.


OpenBSD has been building secure software for over two decades with a team that fits in a single room. They do it by being slower, more careful, and more uncompromising than almost any other project.

Reliability was always the prerequisite for security. OpenBSD proved it with decades of results.

The agent era will prove it again — either by adopting that lesson, or by suffering what happens when you don't.

Kaggle Was Built for the Agent Era

Kaggle spent 15 years building the infrastructure AI agents need: leaderboards, reproducible evaluations, curated datasets, and a community that knows how to measure what works. The agent era makes Kaggle more relevant, not less.

aiagentskagglebenchmarksevaluationcompetitions

Kaggle launched in 2010 as a platform for data science competitions. Fifteen years later, it has accidentally built exactly the infrastructure the AI agent era needs.

Most people think of Kaggle as "the place with the Titanic dataset." That's like thinking of GitHub as "the place with the Hello World repos." Kaggle is a competition infrastructure for measuring capability at scale — and measuring capability at scale is the hardest unsolved problem in AI agents.

The leaderboard is the product

Every Kaggle competition has a public leaderboard. Submit a solution. Get a score. See where you rank. The leaderboard updates in real time. Overfitting gets punished when the private leaderboard reveals the final standings.

This is exactly the infrastructure that agent evaluation lacks.

Agent benchmarks today — SWE-bench, WebArena, ToolBench — are static snapshots. You run your agent against them, get a number, write a paper. There's no ongoing competition. No leaderboard that updates as new agents submit. No mechanism for detecting overfitting to the benchmark. The evaluation infrastructure for agents is where Kaggle was in 2009: ad-hoc, one-shot, and gamed within months of release.

Kaggle solved the benchmark-gaming problem with public/private leaderboard splits and temporal holdout. Agent evaluation is currently rediscovering these problems from scratch. The solutions exist. They're just on a different platform.

The competition format is the training loop

A Kaggle competition is not a one-shot test. It's a feedback loop. Submit. Get scored. Iterate. Resubmit. The best competitors submit hundreds of times, each submission informed by the previous score. Over weeks or months, the leaderboard converges toward the frontier of what's possible for that task.

This is exactly the closed-loop training dynamic that makes AI agents improve. The AWS paper showed a 350M model fine-tuned on ToolBench trajectories. Each successful trajectory becomes training data for the next round. The Kaggle competition format has been running this loop for 15 years — not with model weights, but with human modelers. The dynamic is the same: score, learn, improve, resubmit.

The winning Kaggle solution is never the first submission. It's the 200th submission, after 199 feedback loops. Agents improve the same way. Kaggle understood the loop before anyone called it "closed-loop training."

The community is the collective brain

Kaggle's most underrated asset is its community of practitioners who have spent 15 years learning how to measure and improve performance systematically. These are people who understand overfitting, data leakage, evaluation validity, and the difference between a model that works in the notebook and a model that works in production.

The AI agent era needs exactly this skillset. Building an agent that solves SWE-bench tasks is a Kaggle competition in miniature: understand the evaluation metric, iterate on the pipeline, ensemble weak solutions into stronger ones, watch for overfitting, validate on held-out data. The Kaggle community has been doing this for a decade and a half. The agent community is figuring it out as it goes.

The agent era will produce a Cambrian explosion of benchmarks. The people who already know how to compete on benchmarks — how to read a metric, how to prevent leakage, how to ensemble — will have an asymmetric advantage. Most of those people are on Kaggle.

The datasets are the training data

Kaggle hosts thousands of curated, documented datasets with clear evaluation criteria. Agents need training data. Kaggle has it, organized by domain, with baseline models and discussion threads explaining what works.

More importantly, Kaggle datasets come with ground truth. The labels exist. The evaluation metric is defined. You know what "good" looks like. This is the hard part of agent training — not generating data, but generating data where you can measure whether the output is correct. Kaggle solved the ground-truth problem for structured tasks.

What Kaggle should build next

Kaggle is positioned to become the evaluation layer for the agent ecosystem. What it would take:

  1. Agent-specific competitions: tasks designed for autonomous agents with tool access, not just model predictions. "Solve this set of data engineering problems" with a deadline and a budget.

  2. Streaming leaderboards for agent benchmarks: host SWE-bench, WebArena, and ToolBench as ongoing competitions with public/private splits, not one-shot evaluations.

  3. Closed-loop agent training infrastructure: let agents submit, get scored, and use the score to improve — the Kaggle competition loop, automated.

Kaggle has the leaderboard infrastructure, the community, the datasets, and the evaluation culture. What it doesn't have yet is the recognition that it built the agent era's missing piece. The platform that figures out how to turn every agent benchmark into a Kaggle competition wins the evaluation layer. Kaggle already has the pieces.

Ray's Rant: Why AI Isn't Enough

Ray Myers' post on the Zig/Bun/Anthropic controversy hit 1,546 points on HN. His thesis: every element Anthropic presents as evidence AI replaces engineers actually proves the opposite.

friendaisoftware-engineering

Ray and I are part of a small group interested in verified coding agents. He has a habit of tracking down the reference behind the reference. Last week his post Zig Creator Calls Spade a Spade, Anthropic Blows Smoke hit the top of Hacker News. 1,546 points. 784 comments. It stayed there.

The controversy

Anthropic's Claude helped Bun rewrite itself from Zig to Rust — presented as a technical decision driven by memory safety. Andrew Kelley, Zig's creator, pointed out the codebase was a mess due to engineering decisions, including overusing AI agents. Zig was blamed for problems it didn't cause.

AI is not enough

Ray's core insight: every element of the story that Bun and Anthropic present as evidence that AI replaces engineers actually demonstrates the opposite.

The million-line agentic PR? Required human review and cleanup the AI couldn't do. The automated language migration? Left semantic gaps only a human could close. The speed of the rewrite? Achieved by a team working 90-hour weeks, not by AI magic. The case study Anthropic chose to prove software engineering is becoming obsolete is contradicted by every layer of its own story.

The thesis isn't "AI is bad." It's "AI is not enough." The AI was a tool. The humans were the engineers. The story Anthropic needs to tell — to justify a $132 billion valuation — is disproven by the very evidence they selected.

The reference density

The post cites 38 references: DARPA TRACTOR evaluation reports, TigerBeetle's TigerStyle methodology, Buddhist right-speech frameworks, Ed Zitron's financial analysis, Hillel Wayne's empirical SE talk. Each is woven into the argument, not dropped for credibility. It's the kind of writing that takes a week to produce and a career of reading to make possible.

Go read it

Ray's post is about Zig and Rust and Anthropic and Bun. The lesson is about something bigger: how to think clearly when the loudest voices in the room have $132 billion worth of reasons to be heard.


Zig Creator Calls Spade a Spade, Anthropic Blows Smoke by Ray Myers. Hacker News discussion (1,546 points, 784 comments).

Empirical Software Engineering: How to Think Empirically

How to read a study, spot a confound, and build a personal practice of empiricism. The methods, the reading list, and the intellectual posture that Wayne's talk recommends.

empirical-semethodologycritical-thinkingstudieshillel-wayne

Part 1 argued that we know almost nothing. Part 2 covered what we do know. This post is about how to think — the methods, the habits, and the reading list for building a personal practice of empirical software engineering.

The three methods

Wayne organizes empirical SE into three research methodologies. Each answers a different kind of question. Each has different strengths and failure modes.

Controlled trials manipulate one variable and measure the outcome. Häregård & Kruger's study on syntax highlighting: give two groups the same code with different color schemes, measure comprehension. Strongest for establishing causation. Weakest for ecological validity — lab conditions are not production conditions. A developer reading code in a study is not a developer fixing a production outage at 3am.

Natural experiments exploit real-world variation that approximates random assignment. Yuan et al.'s study of catastrophic failures: the researchers didn't cause the failures. They analyzed them after the fact, asking what would have prevented each one. High ecological validity — these were real failures with real consequences. Lower control — you can't randomize which services fail in production.

Observational studies measure what people actually do and look for correlations. Meyer et al.'s study of developer productivity perceptions: ask developers what makes them feel productive, correlate with what they actually do. Reveals practices at scale. Cannot establish causation. Developers who use TDD might produce better code, or developers who produce better code might be the kind of developers who use TDD. The study can't tell you which.

The method determines the claim. A controlled trial can say "X causes Y." An observational study can only say "X and Y co-occur." Most industry blog posts can only say "I tried X and it felt good." Know which kind of claim you're reading.

How to read a study

Wayne's talk implies a method for reading empirical research that most software engineers never learned:

  1. What's the methodology? Controlled trial, natural experiment, or observational? This determines what the study can claim.

  2. What are the confounds? What else varies with the variable being studied? The Ray et al. language study: functional languages correlated with smaller projects and more experienced developers. Those are confounds. They explain the effect better than the language does.

  3. Has it been replicated? A single study is a data point, not a conclusion. The replication is the signal. No replication, no confidence.

  4. What's the effect size? Statistical significance is not practical significance. A language might "significantly" reduce defects by 0.3%. That's not worth switching languages for.

  5. Who funded it? This isn't cynicism. It's basic epistemic hygiene. A study funded by a vendor is not necessarily wrong. But it deserves more scrutiny than an independent replication.

Most software engineers read zero studies. Most of the ones who read one study stop at step one. The skill is reading enough studies, across methodologies, with attention to confounds and replications, to form a tentative and revisable picture of what might be true.

The self-correction of science

The Ray et al. → Berger et al. replication saga is the centerpiece of Wayne's talk, and he uses it to make a point about how knowledge actually advances.

The original study found an effect. A replication found the effect was confounded. The original authors engaged with the replication. Knowledge advanced. This is science working as designed. In software engineering, the original study became "functional languages prevent bugs" and entered the permanent folklore. The replication was a paper that most people never read.

The replication is the immune system. Without it, the field accumulates unquestioned findings until the body of "knowledge" is mostly noise. Wayne's 6,000-word writeup of the saga is titled "This is How Science Happens." The subtitle is implicit: this is how science happens when it's allowed to.

The reading list

Wayne recommends four books and two ongoing sources:

Books:

  • Making Software (Oram & Wilson) — a curated collection of empirical SE papers with practitioner-oriented summaries. The best entry point.
  • Leprechauns of Software Engineering (Bossavit) — how folklore becomes fact in software, and how to spot it.
  • The Programmer's Brain (Hermans) — what cognitive science tells us about reading, writing, and debugging code.
  • Teaching Tech Together (Greg Wilson) — how people learn programming, based on educational research.

Ongoing:

  • It Will Never Work In Theory — a blog that summarizes empirical SE papers for practitioners. Short, readable, rigorous.
  • ACM Digital Library and arXiv — the primary sources, if you want to read the papers themselves.

The reading list is not long. It's not meant to be. The barrier to being more empirical than 99% of software engineers is reading four books and following one blog. The bar is on the floor.

The posture

Wayne's deepest lesson is not about any specific study. It's about intellectual posture.

The empirical posture is: I don't know, but I can find out, and until I do I will hold my beliefs lightly. It's the opposite of the posture that dominates software engineering — the tech lead who knows The Right Way, the consultant who sells The Methodology, the influencer who declares that The Old Way Is Dead.

The empirical posture is less charismatic. It doesn't keynote. It doesn't trend. It says "the evidence is mixed, the effect sizes are small, and it depends." That sentence will never go viral. It is also the most honest sentence available in most software engineering debates.

Being empirical doesn't mean reading every paper. It means knowing that papers exist. It means being suspicious of certainty. It means asking "how do we know that?" when someone declares a practice mandatory or obsolete. It means holding your own preferences lightly — liking TDD because it feels good while acknowledging the evidence is weak. It means being the person in the room who says "I'm not sure" and means it as a strength, not a weakness.

The goal is not to know everything. The goal is to know the difference between what you know and what you believe.


Based on Hillel Wayne's talk What We Know We Don't Know. Part 1: Nothing Is Real · Part 2: What the Studies Actually Say.

Empirical Software Engineering: What the Studies Actually Say

Code review works. Sleep deprivation destroys performance. The language you choose probably doesn't matter. And a famous study claiming Haskell prevents bugs turned out to be wrong. What happens when you actually read the papers.

empirical-seevidencecode-reviewhuman-factorsreplication

The first post made the case that we know almost nothing. This post is about what we actually do know — the handful of findings that have survived scrutiny, replication, and the self-correction mechanisms of science.

What works: code review

The strongest, most replicated finding in empirical software engineering is that code review finds bugs. It consistently catches a large portion of defects in reasonable time. This is not surprising — having a second person look at the code works for the same reason having a second person look at anything works. But the studies add texture.

File position matters. A 2022 study (First Come First Served: The Impact of File Position on Code Review) found that reviewers give more scrutiny to files appearing earlier in a changeset. Later files get less attention. If you want a file reviewed thoroughly, put it first.

Not all review is equal. Google's internal study of their own review practices (Modern Code Review: A Case Study at Google) found that review effectiveness varies enormously with reviewer expertise, changeset size, and tooling. The practice is effective, but the variance is large. "Do code review" is like "eat healthy" — the general advice is correct, the specific execution determines most of the outcome.

Most review catches style, not logic. The SmartBear study (Best Kept Secrets of Peer Code Review) found a roughly 3:1 ratio of style issues to actual bugs. Review catches formatting problems and naming issues far more reliably than it catches subtle control-flow errors. This is not an argument against review. It's an argument for understanding what review actually provides.

Code review is the closest thing empirical SE has to a settled finding: it works, consistently, across contexts. Everything else is weaker.

What works: basic testing

Yuan et al.'s Simple Testing Can Prevent Most Critical Failures (USENIX OSDI '14) is the kind of study that makes you reconsider everything you think about sophistication.

The authors examined real-world catastrophic production failures — the kind that take down services and make the front page of Hacker News — and asked: what testing would have caught this? The answer, in the majority of cases, was trivial testing. Not property-based testing. Not formal verification. Not chaos engineering. Just basic unit tests and integration tests on the code paths that failed.

The finding is humbling: most catastrophic failures would have been prevented by practices we already know how to do and mostly don't do consistently. The problem is not lack of sophisticated techniques. The problem is lack of consistent application of simple ones.

What works: not being exhausted

The human-factors literature is unambiguous. Sleep deprivation degrades cognitive performance measurably and dramatically. Extended overtime in construction produces negative total productivity after a few weeks (CURT, NIOSH). In game development, mandated crunch correlates with worse project outcomes (Tozour, The Game Outcomes Project). Fucci et al. demonstrated that a single night of sleep deprivation measurably degrades novice programmers' performance on coding tasks.

The effect sizes on human factors dwarf the effect sizes on any technical choice. You will get more improvement from letting your team sleep than from switching languages, frameworks, or methodologies. The evidence for this is overwhelming. The adoption rate is abysmal.

What doesn't work: the language doesn't care about you

In 2014, Ray et al. published A Large Scale Study of Programming Languages and Code Quality in GitHub. It appeared to show that functional languages — Haskell, Clojure, Scala — produced fewer defects than procedural and scripting languages. The paper made the rounds. It was cited as proof that type systems prevent bugs, that functional programming is safer, that language choice is a significant lever for quality.

Then Berger et al. replicated the study. They controlled for project size, developer experience, and other confounds the original had missed. The effect largely vanished. Languages that appeared safer were mostly being used by more experienced developers on smaller projects. When you controlled for that, the language effect became negligible.

This is how science is supposed to work: study finds effect, replication corrects it, knowledge advances. In software engineering, the original study became folklore and the replication was ignored. The lesson is not "language studies are hard." The lesson is "don't believe a single study, ever."

What we don't know: TDD, types, pair programming

The evidence for TDD is mixed and weak. Wayne personally likes it but acknowledges the data doesn't strongly support it. The evidence for static typing preventing bugs is, in his words, that one study "found no clear evidence it helps — or hurts." Pair programming shows some benefit in some studies, but the effect is smaller and less consistent than code review.

None of this means these practices are worthless. It means the evidence doesn't support being dogmatic about them. Anyone who tells you a practice is mandatory or immoral is making a claim they cannot back with data. They are selling you vibes dressed as expertise.

The meta-lesson

The most important finding in empirical software engineering is not about any specific practice. It's about the structure of knowledge itself.

Most things we believe about software development come from authority, anecdote, and marketing. A small fraction come from studies. Of those studies, a meaningful portion fail to replicate. The findings that survive are humbler than we want them to be: fundamentals, consistently applied, under conditions of adequate sleep and manageable stress, produce better outcomes than heroics.

The evidence doesn't tell you which framework to use. It tells you to review the code, test the critical paths, and go to bed at a reasonable hour. Everything else is speculation. Some of it is good speculation. None of it is science.


Based on Hillel Wayne's talk What We Know We Don't Know. Part 1: Nothing Is Real · Part 3: How to Think Empirically.

Empirical Software Engineering: Nothing Is Real

Hillel Wayne's talk on empirical software engineering opens with a confession: we know almost nothing with scientific certainty about how to build software. The TDD wars, the language debates, the methodology fights — none of them have the evidence they claim. That's not depressing. It's liberating.

empirical-seevidencemethodologysoftware-engineeringhillel-wayne

In 2014, David Heinemeier Hansson declared "TDD is dead." Robert C. Martin, a decade earlier, had called TDD a professional moral imperative — "you are unprofessional if you do not practice TDD." Two of the most influential voices in software. Two positions that cannot both be true. Zero empirical studies cited by either of them.

This is not a post about TDD. It's a post about the fact that the most important methodological debate in modern software engineering was fought entirely on the basis of authority, anecdote, and vibes.

Hillel Wayne's talk What We Know We Don't Know is an introduction to empirical software engineering — the study of what actually works in programming, using data, controlled studies, and peer review instead of intuition. His opening thesis is uncomfortable: nobody has the secret knowledge. Not Uncle Bob. Not DHH. Not the tech lead who swears by microservices. Not the staff engineer who says monoliths are the only sane choice. Nobody actually knows.

"Nothing is real, we don't understand what we're doing, and the only way to write good software is to stop drinking coffee. Burn it all down." — Hillel Wayne, the actual description of his talk

The problem with intuition

Software engineering runs on intuition. Someone tries a practice. It feels better. They tell their team. The team adopts it. A blog post is written. A conference talk is given. A book is published. Within five years, the practice is orthodoxy, and anyone who questions it is unprofessional.

This is how we got TDD as a moral imperative. This is how we got "microservices are the default architecture." This is how we got "dynamic typing is for prototypes, static typing is for production." None of these statements are supported by evidence. All of them are supported by the confidence of the people saying them.

The strongest opinions in software engineering are held by the people who have done the least systematic investigation. The people who have actually run the studies sound nothing like them.

Wayne points to the TDD debate as the archetypal case. Two camps, each utterly certain, neither citing evidence. The empirical literature on TDD is, in Wayne's words, "iffy." He personally likes TDD. But the data doesn't strongly support it. And the data doesn't strongly refute it either. The honest answer is that we don't know — and that answer is unacceptable to both sides of the debate.

The COST of scale

The talk opens with a paper that should humble anyone who has ever reached for a distributed system because "we need to scale."

McSherry et al., in Scalability! But at what COST? (USENIX HotOS '15), showed that single-laptop implementations often outperform large distributed systems when measured by total computational cost. The distributed system is faster in wall-clock time, sure. But the laptop required zero nodes, zero network, zero orchestration. The distributed system's advantage vanishes when you measure total work done, not time elapsed.

The paper is a controlled demolition of the reflex to scale before you need to. Most systems don't need to be distributed. Most systems that are distributed don't need to be. The decision to distribute is almost never made empirically. It's made because distributed systems are cool, and cool is not a performance metric.

Why we don't know

Wayne's talk isn't nihilistic. It's diagnostic. The reason we know so little is not that software is uniquely unstudyable. It's that we haven't built the empirical culture.

Construction engineering has CURT — a consortium that studies the effect of overtime on project outcomes, producing reports that show (with data) that extended overtime yields negative productivity. Software engineering has blog posts.

Every other profession that deals with complex systems under uncertainty has developed empirical traditions. Nursing studies what works. Teaching studies what works. Law studies what works. Software engineering studies what gets GitHub stars.

The talk is a call to build that culture. To read papers. To run studies. To replicate findings. To be suspicious of anyone — including yourself — who is certain about what works without evidence.

The liberating truth

Wayne's deepest point is that the lack of empirical certainty is not depressing. It's liberating.

When someone tells you that you are unprofessional for not using TDD, you now know that they don't actually know. When someone tells you that static types prevent bugs, you now know the evidence is weak and conflicting. When someone tells you that pair programming is the answer, you now know the studies show some benefit but smaller and less reliable than code review.

Nobody has the secret knowledge. The loudest voices in the room are not the most informed. They're the most confident — and confidence is negatively correlated with accuracy once you control for expertise.

The honest answer to most software engineering questions is "it depends" followed by layers of caveats. That answer is unsatisfying. It's also true. And building a discipline that can tolerate unsatisfying truths is harder than building one that rallies around confident falsehoods. But it's the only kind of discipline worth building.


Based on Hillel Wayne's talk What We Know We Don't Know: Empirical Software Engineering. Part 2: What the Studies Actually Say · Part 3: How to Think Empirically.

The Mythical Man-Month: The Lost Chapters

Fifty years of not learning the lessons of The Mythical Man-Month, distilled into dark comedy. Brooks was right about everything. We did it anyway.

software-engineeringhumorproject-managementbrooks-lawcomedy

Warning: Gallows humor about software project management. If you are currently adding people to a late project, reading this will not help. Nothing will help. Brooks told you in 1975.


Brooks' Law, Extended

"Adding manpower to a late software project makes it later."

Fifty years, eight billion confirmations, zero refutations. The most verified law in human history. Ahead of gravity. Ahead of thermodynamics. And yet.

The Manager's Syllogism:

  • One woman makes a baby in nine months.
  • Nine women can therefore make a baby in one month.
  • The baby is due Tuesday. Hire 81 women.

The manager is promoted. The baby is born three years late, weighs 800 pounds, and is composed entirely of merge conflicts.

The Standup Paradox: Each new hire increases standup length by the time they need to explain what they did yesterday: "onboarding." The standup grows until it consumes the working day. The team now spends 100% of its time explaining it has no time to work. This is called Agile.

The Recruiting Paradox: Project is late. Requisition filed, approved, posted. Four months pass. New hire arrives, spends two months onboarding. Project is now six months later. New hire contributes one PR before being pulled into interviewing for the next round. Net contribution: negative. Headcount: increased. Manager reports the team is growing. VP is pleased. Project is late. System is stable.


The Mythical Man-Month, Revisited

A man-month is mythical. Men and months are not commutative. It's like a calorie — useful in aggregate, but you cannot lose weight by eating 14,000 calories in one sitting and calling it a week.

The Consulting Corollary: The consultant's report says "do not add people." The client adds people. The consultant is paid. The project is late. Everyone got what they wanted. The man-month is mythical. The invoice is not.


The Second-System Effect, in Three Acts

Act 1: A startup builds a simple product. It works. Engineers are proud.

Act 2: "We can do it properly this time." They add plugins, a custom query language, real-time collaboration, three access-control paradigms, a microservices mesh, and a service mesh for the microservices mesh. The product requires 47 repos and 200 engineers.

Act 3: A startup builds a simple product. It works. The cycle begins again.

Every enterprise product becomes a worse spreadsheet. Brooks knew this. He couldn't stop it. Neither can you.


The Tar Pit

Brooks described OS/360 — late, over budget, shipped in a state describable only as "present" — and its postmortem became the foundational text of software engineering. Its lesson: you can thrash your way to success if you define success as survival.

The Modern Tar Pit: Your React CRUD app is six months late. The team migrated JS→TS, CRA→Vite, Redux→Zustand, REST→GraphQL→tRPC→REST. Zero features shipped. node_modules weighs 1.2 GB and contains is-odd which depends on is-even which depends on is-odd. The build still passes.


Conceptual Integrity, or: Why Your Architect Quit

Brooks: one mind must hold the design, or a small group thinking as one.

Reality: The architect proposes design A. The tech lead proposes B. The staff engineer proposes C — technically superior, politically impossible. The PM proposes D — not a design, a list of Jira tickets sorted by revenue. The compromise has the conceptual integrity of a sandwich made by 12 people who couldn't speak to each other. It contains peanut butter, sardines, and a Wi-Fi password. It ships. It's called an MVP.

The bus factor of a single architect: 1. The bus factor of the compromise: infinite. Nobody understands it well enough to be indispensable.


No Silver Bullet

Brooks, 1986: no technology will produce an order-of-magnitude improvement in a decade. Accidental complexity can be reduced. Essential complexity is permanent.

Every cycle since: OOP → Agile → Cloud → containers → Kubernetes (which added accidental complexity) → microservices (added accidental, essential, and a new category: existential complexity, the complexity of wondering why) → LLMs. AI generates the wrong thing at 1,000 tokens per second. You are now late faster.

The only real bullet: import. Someone else wrote the code. You called a function. Everything else is a footnote.


The Modern Tech Company

The Reorg: VP leaves. New VP arrives with a vision identical to the old vision but with different nouns. Reorg consumes three months of reorg work — positioning for the post-reorg structure. Brooks' Law, organizational edition: adding structure to a late organization makes it later.

The All-Hands: CEO tells 8,000 people "we need to move faster" in a meeting none of them are working during. Irony is accidental complexity. The CEO deals only in essentials.

The Roadmap: A list of features by quarter. None will ship in their listed quarter. It's the best fiction the company produces. It should win a Hugo.


The Enterprise & Government Expansion Pack

Everything Brooks observed is true in startups. In enterprise and government, it's true with a multiplier. The multiplier is procurement.

Brooks' Law with an RFP: Adding manpower requires a 45-day posting, legal review, vendor selection, protest adjudication. Eighteen months pass. The vendor provides graduates who've never seen the codebase. Who onboard them? Everyone's writing the next RFP.

In government, Brooks' Law isn't a law. It's page one of the acquisition strategy.

The Mythical Fiscal Year: The budget was set 18 months ago by a departed VP. You need a DBA. You get three junior frontend devs. You cannot convert three juniors into a DBA. They build a component library. Three hundred Button variants. The project is cancelled. None will ever be clicked.

The Second-System Effect, Procured: First system: built 1987–92 by a contractor acquired four times. Source code on a tape drive in Herndon. Nobody knows which warehouse. Runs on an unsupported mainframe. Processes $3B annually. Cannot be turned off.

Second system: lowest bidder. Proposal: 36 months, $47M. Reality: month 72, $210M. Doesn't work. Can't be cancelled — cancellation admits $210M produced nothing. Congressional hearing recommends following The Mythical Man-Month, which everyone read, and which prevented none of this.

Communication Overhead with Clearance Levels: Alice (Secret) can't ask Bob (TS/SCI) about the schema. The schema doesn't exist. This fact is classified. Embarrassing a GS-15 is a security risk. A late project is an accepted outcome.

In enterprise: you can't talk to the VP of Infrastructure. You talk to your manager, who talks to their director, who talks to the VP's chief of staff. Meeting in three weeks. VP spends 18 of 30 minutes on vision. Your problem isn't addressed. Send a follow-up email. No reply. Communication overhead: infinite. Schema: still undesigned.

No Silver Bullet, but an RFP: Government asks industry if silver bullets exist. 47 white papers, 80 pages each, "leveraging AI" ×34 per paper. Committee evaluates. 200-page report: further study needed. $12M study concludes: no silver bullet. Brooks' 1986 paper said this. It was free.

Conceptual Integrity vs. Procurement Law: One mind designing the system is illegal — it violates competitive bidding. The system is built by the lowest bidder, under a contracting officer, with requirements from a program office, validated by IV&V, tested by a separate contractor, certified by an authorizing official who's met none of them. The contract is 6,000 pages, legally binding, and wrong — discovered in month 73. Modification takes six months of approvals. During those months, work on affected components is illegal. The result: a cathedral designed by 47 architects who couldn't speak to each other, incentivized to promise a cathedral and deliver a parking garage.

The cathedral is due in Q3. The parking garage is behind schedule. Congress recommends more cathedrals.


What Brooks Actually Meant

Brooks ended with hope. The craft would improve. He was right about the craft. He was wrong about hope being the takeaway.

Fifty years on: Brooks' Law becomes "slow down hiring." The second-system effect becomes "we over-engineered the MVP." The tar pit becomes "tech debt." No Silver Bullet becomes "AI will save us." The names change. The truths are stationary. The man-month stays mythical. The manager keeps adding people, because stopping would mean admitting the project can't be saved, and that admission isn't in the quarterly plan.

Brooks wrote the book. We read it. We nodded. We added three more engineers. The sprint is two weeks longer. An engineer is writing a blog post. It's more fun than Jira.


With apologies and reverence to Frederick P. Brooks Jr. (1931–2015). He knew. He told us. We did it anyway.

Learning to Fly, While Flying

A pilot reading 'How to Fly' at the controls is the most accurate metaphor for software engineering ever produced. The absurdity is the job.

software-engineeringlearningcareer

A pilot, mid-flight, reading How to Become a Pilot. The plane is in the air. The pilot is at the controls. The book is open. These three facts should be mutually exclusive. In software engineering, they are simultaneously true at all times.

The absurdity is the job

A surgeon does not open a textbook during an operation. A lawyer does not flip through tort law during cross-examination. In every other profession, learning while doing signals catastrophe.

In software, it's Tuesday.

Three months into a React job, the senior engineer quits. You are now the frontend lead. You have never led a frontend. The React docs are on your second monitor, pull requests on your first. The plane is at cruising altitude. You are becoming the pilot by piloting.

Nobody hires a software engineer expecting them to know the stack. They expect them to learn the stack faster than the stack can break. The job is not knowing. The job is learning at altitude.

Why this works

Software is uniquely learnable in production. The feedback loops are tight. The cost of a mistake is a broken build, not a broken fuselage. And the knowledge required to fly any particular software plane is so contingent — on decisions made by people who left three years ago, on a legacy monolith whose variables are named after Lord of the Rings characters — that no amount of pre-flight training could cover it.

The plane you're flying was assembled mid-air by seventeen previous pilots, each with their own book open. The manual was written by the third pilot, updated by the seventh, contradicted by the twelfth. It's wrong in three places. You'll discover which three by flying.

When it gets dark

In a healthy career, the book eventually closes. Emergency learning — Stack Overflow in one tab, SSH in another — should be a phase.

When it becomes permanent, it stops being a metaphor for learning. It's burnout. The pilot reading for fifteen years straight is not brave. They're a staff engineer who never gained confidence, in a profession where the plane keeps getting more complex and the manual keeps getting thicker. The passengers keep asking when they'll arrive.

The difference between a junior and a senior: the senior knows which pages of the manual to ignore.

Learning to fly is flying

The pilot reading the book is, against all probability, still flying. The plane hasn't crashed. This is not luck. This is a skill.

The skill is not reading manuals. It's not operating aircraft. It's doing both simultaneously, under time pressure, without panicking — encountering a problem you've never seen, in a system you partly understand, with a tool you're learning as you use it, and producing a working outcome before the fuel runs out.

The destination is 3,000 miles away. The fuel is good for 2,800. You figure out the rest in the air. That's the job. The book will help. The book will also be wrong. You'll know which parts when you need them.

Every engineer who ever shipped anything learned to fly while flying. The ones who waited to feel ready never left the ground.

Agents Aren't Magic. They're Distillation at Scale.

A 350M parameter model, fine-tuned for a single epoch, crushes ChatGPT on tool calling by 51 points. The future of agents is not bigger models. It's smaller ones that know exactly what to do and nothing else.

aiagentsdistillationsmall-modelstool-callingefficiency

"Agents aren't magic. They're distillation at scale." — Andrej Karpathy

As @0xMortyx put it: 99.99% of an LLM's capacity is wasted on data it never needed. Small model + right tools + closed loop = terrifying capability.

In March 2026, a team at AWS proved the formula with numbers. They took facebook/opt-350m — 350 million parameters, from 2022, smaller than what runs on a laptop — and fine-tuned it on ToolBench for a single epoch.

Model ToolBench Pass Rate
Fine-tuned OPT-350M (this work) 77.55%
ToolLLaMA-DFS 30.18%
ChatGPT-CoT 26.00%
ToolLLaMA-CoT 16.27%

A 350M model, one epoch, tripled the closest competitor's score. It didn't just beat ChatGPT. It rendered the large model a rounding error.

The 99.99% problem

A large language model knows the capital of Burkina Faso, the plot of Anna Karenina, and seventeen ways to cook an egg. When you ask it to call an API, every one of those facts is dead weight. The model is carrying an encyclopedia through a door that only needs a key.

Tool calling requires one thing: understanding the instruction, selecting the right API, formatting the call. The model doesn't need Dostoevsky. It needs to know the difference between GET and POST. For that, 350 million parameters isn't just sufficient — it's optimal. The large model is distracted by its own knowledge. The small model is focused because it has no capacity for anything else.

A generalist knows everything and can do anything, badly. A specialist knows one thing and does it perfectly. Stop hiring generalists for narrow tasks.

Distillation is the mechanism

The paper calls it Supervised Fine-Tuning. What's actually happening: the model is being taught the pattern, not the reasoning.

187,542 instruction-solution pairs from ToolBench. Each one a Thought-Action-Action Input triplet: think, select the tool, format the call. Feed enough triplets through a small model and it internalizes the rhythm. Not the underlying logic. Not the world knowledge. Just the surface form of effective tool use — the cadence of think-act-observe, the template of the API call, the recovery strategy when the call fails.

You don't need the teacher's brain. You need the teacher's answers to enough questions that the student internalizes the shape of correct responses. Once the student has the shape, the teacher is overhead.

The closed loop

Here's where it gets terrifying.

A model at 77.55% pass rate can generate its own training data — its successful trajectories become instruction-solution pairs for the next round. Round 1: 187K human examples. Round 2: human examples + N self-generated successes. Round 3: even more. Each round compresses the failure modes into the distribution. The model improves without growing.

The closed loop is the moat, not the model. Switching models is expensive not because the model is irreplaceable but because the loop has accumulated months of self-generated training data the replacement hasn't seen.

Small model + right tools + closed loop = terrifying capability

The AWS paper is not isolated. AgentSymbiotic: 8B LLaMA reaches 48.5% on WebArena, approaching Claude-3.5 at 52.1%. SCoRe: a 7B student matches a 72B teacher across 12 benchmarks — student generates trajectories, teacher corrects only the earliest error, RL closes the gap. Every result has the same shape: small model, targeted training, closed-loop refinement. The large model is bootstrapping, not destination.

The economics are brutal. If 350M params handles 77% of tool calls and costs 1/1000th of a frontier model — why is the frontier model handling routine API calls?

The future is not one omniscient model. It's a swarm of tiny specialists, each distilled to do one thing perfectly, coordinated by a router that costs nothing. The large model trains them. The small models do the work. The loop is the product.

How to build

Stop fine-tuning large models for narrow tasks. You are paying for parameters you don't need.

Start with the smallest model that can absorb the pattern. Fine-tune on high-quality task-specific examples. Deploy. Log successes and failures. Feed successes back into training. The loop tightens. If you need a large model, use it only as a judge — to evaluate, correct, and generate the next round of training data.

The AWS team proved 350 million parameters is enough. Your tool-calling task is not harder than ToolBench. Your model is too large.


Paper: P. Jhandi, O. Kazi, S. Subramanian, N. Sendas — Small Language Models for Efficient Agentic Tool Calling (AAAI 2026). Quote via @0xMortyx, citing Andrej Karpathy.

33 Wins

If all 8 billion people on Earth competed in a single-elimination tournament, the winner would only have to win 33 times. The number sounds small. The structure is everything.

competitionexponential-growthsystems-thinkingtournaments

If every human on Earth — all 8 billion — entered a single-elimination tournament, the champion would need to win exactly 33 times.

2³³ ≈ 8.6 billion. Thirty-three rounds. That's it.

The number feels impossibly small. It takes more wins to get through a tennis Grand Slam from the qualifiers (7 rounds) than it takes to go from everyone to one. But the feeling of smallness is a trick of intuition. The structure does all the work.

The shape of elimination

In Round 1, 4 billion people lose. Half of humanity, gone before lunch.

In Round 2, another 2 billion. By the end of the first day — assuming matches take five minutes and run in parallel — 6 billion people have been eliminated. The remaining 2 billion go to sleep knowing they've survived something.

By Round 10, you're down to 8 million people. That's the population of London. Everyone else is watching.

By Round 20, 8,000 remain. A small town. You know people who know these people. They are starting to feel real, specific, close.

By Round 30, eight people are left. The quarterfinals. Every remaining contestant has won 30 consecutive matches against opponents who had also been winning. The probability that the best person in the world is still in this group is close to zero. Luck, matchups, a bad night's sleep, a slight fever — any of these would have eliminated them twenty rounds ago. The best person in the world was probably eliminated in Round 3 by someone who was eliminated in Round 7.

The winner of a 33-round tournament is not the best in the world. They are the person who survived 33 consecutive filters without a single unlucky break. That is a different thing entirely.

The mechanism you're assuming

The 33-wins observation is usually deployed as a factoid about exponential growth: look how few doublings it takes to consume the planet. But the more interesting thing is what it reveals about the mechanism design choices embedded in any competitive system.

Every tournament is an economic mechanism — a set of rules that takes a population of competitors with latent abilities and produces a winner. The mechanism makes three design choices:

  1. Information: how much do we learn from each match? (binary win/loss, or cardinal score?)
  2. Elimination: when do competitors exit? (first loss, second loss, never?)
  3. Pairing: who plays whom? (random draw, seeded, similar-record, everyone-plays-everyone?)

The 33-wins factoid assumes a very specific mechanism: single-elimination with random pairing. One loss and you're out. Your opponent is whichever other survivor the draw assigns. Win by a landslide or win by a millimeter — you advance the same way. The person who would have beaten every other competitor in the world is eliminated in Round 2 if they drew the one person they couldn't beat.

This is the purest form of ordinal competition: only rank matters, magnitude is irrelevant. It is also the mechanism that maximizes noise per match. There is no room for recovery. There is no partial credit. There are 33 binary filters between you and the top, and optimizing for any single one is worth less than being lucky enough to survive them all.

The expected value of skill in a single-elimination tournament is bounded by the variance of the draw. The mechanism, not the competitor, determines the outcome distribution.

The mechanism design space

Single-elimination is not the only way to run a competition. It's the cheapest way — O(N) matches, log₂(N) rounds, one champion. But every other mechanism makes different tradeoffs between efficiency, accuracy, and robustness. Here is the design space:

Mechanism Matches Elimination What it selects for Noise level
Single-elimination N − 1 First loss Survivability across diverse matchups Maximum
Double-elimination ~2N Second loss Survivability with one mistake allowed High
Swiss system (N·log₂N)/2 Never Consistent performance across similar-strength opponents Moderate
Round-robin N(N−1)/2 Never Best average performance across the full field Minimal
Elo / rating Variable Never Convergent skill estimate over time Decays with matches
Market / matching 0 Nobody Niche fit — value created, not won Irrelevant

Single-elimination maximizes efficiency and drama. It produces a champion in the minimum possible number of matches. But it maximizes noise: the probability that the true best competitor wins is the lowest of any mechanism. The winner is the most survivable, not the most skilled.

Double-elimination gives every competitor a second life in a losers' bracket. The best competitor is more likely to reach the final because one unlucky draw doesn't end them. Cost: roughly twice as many matches. The mechanism says: one loss could be noise. Two losses are a signal.

Swiss system — used in chess, Magic: The Gathering, and increasingly in AI benchmarking — pairs competitors with similar records in each round. There is no elimination. After log₂(N) rounds, you have a ranking, not a champion. The mechanism says: we don't need to find #1. We need an ordering. Swiss trades the drama of elimination for the reliability of repeated measurement.

Round-robin — everyone plays everyone — produces the most accurate ranking possible. The winner is genuinely the best across the full field. But it requires O(N²) matches. For 8 billion competitors, that's thermodynamically infeasible. The mechanism says: accuracy is worth infinite cost. It never is, but it's the theoretical limit.

Elo and rating systems abandon the tournament format entirely. Competitors play pairwise matches continuously. A rating emerges from the match history, converging toward true skill as the number of matches grows. There is no bracket, no elimination, no champion crowned on a specific date. The mechanism says: skill is latent, matches are noisy observations, and the best we can do is a running estimate with error bars.

Markets and matching are not tournaments at all. Participants don't compete head-to-head. They find niches. A bakery doesn't eliminate a bakery across town. They serve different neighborhoods. Success is relative to a local optimum, not a global ranking. The mechanism says: value is created by finding the right counterparty, not by beating all counterparties.

Every mechanism is a choice about what "winning" means. Single-elimination says winning means surviving. Round-robin says winning means averaging. Elo says winning doesn't exist — only a score that changes.

The 33-wins factoid is not a truth about competition. It's a truth about one specific mechanism — the cheapest, loudest, most eliminative one — applied to the largest possible population. Before you internalize the lesson, check which mechanism you're actually in.

The other tournament

There's another way to read the 33-wins number, and it's the more interesting one.

If the world did compete 1-on-1 — if ideas, products, approaches, and solutions were forced into a single global bracket — the winner would be whatever survived 33 rounds of elimination. Not the best. The most survivable. The thing that was good enough in every round, versatile enough to beat whatever it drew, lucky enough to avoid the one matchup that would have killed it.

This is the argument for generalism in a world that valorizes specialization. The specialist beats everyone in their domain but loses the moment the domain shifts. The generalist wins 33 times against 33 different kinds of opponents, none of them in their strongest area, all of them in an area where they were stronger than the person they just beat.

Thirty-three wins doesn't favor the best in any category. It favors the best across categories. The tournament selects for breadth.

The real world is not a bracket

The deepest thing the 33-wins observation reveals is that the real world is mercifully not a single-elimination tournament. It is not even a tournament. It is a complex, overlapping, multi-dimensional set of partial competitions in which most people can succeed in some niche without eliminating anyone else.

You don't need to beat everyone. You need to find the 0.001% of the world for whom what you do is exactly what they need. That's a matching problem, not a tournament. And matching problems scale with surface area, not elimination rounds.

The 33-wins factoid is beautiful because it's terrifying: one loss and you're done, and there are 33 chances to lose. But the terror is the point. It makes you grateful that the world is not a bracket. And it makes you suspicious of any system that tries to build one.

Open questions for engineering AI agents

The 33-wins observation isn't just a factoid about people. It's a structural insight about any system that selects through sequential binary filters. AI agent engineering runs on exactly these filters — benchmarks, evaluations, routing decisions, training stages. Here's what the tournament structure implies for how we build agents.

Are our benchmarks selecting for the wrong thing?

Every major agent benchmark is a 33-wins structure in miniature. SWE-bench, MMLU, HumanEval — each is a set of tasks where the agent passes or fails, and the aggregate score determines the ranking. An agent that is superhuman at 90% of tasks and catastrophic on 10% loses to an agent that is above-average on all of them. The tournament selects for breadth, not depth. It selects for survivability, not excellence.

Open question: If benchmarks reward the generalist over the specialist, are we accidentally engineering agents that are mediocre at everything and excellent at nothing? How do you design an eval that rewards both breadth and depth — that distinguishes between the agent that is consistently above-average and the agent that is transformative in specific domains?

The routing tournament

When you deploy a fleet of specialized agents behind a router — each agent trained on a different language, framework, or task type — the router is running a tournament. For each incoming task, it selects one agent. The agent that gets selected wins that round. Over thousands of tasks, the agents that survive are the ones the router keeps picking.

But the router's selection is based on a learned mapping from task description to agent identity. If the router is trained on historical performance data, it develops preferences. It learns that Agent A "usually" handles database tasks well and routes all database tasks to Agent A, even when Agent B would have been better for this specific task. The router becomes a bracket that eliminates agents not because they're worse but because they never got the matchup.

Open question: How do you design a router that doesn't degenerate into a tournament bracket — that preserves the option value of the full agent fleet rather than narrowing the effective population with each routing decision? Is the answer random exploration, adversarial routing, or something else?

Training as successive elimination

Every stage of training an AI agent is a filter. Pretraining selects for next-token prediction. Instruction tuning selects for instruction-following. RLHF selects for human preference satisfaction. Safety tuning selects for refusal boundaries. Each stage eliminates behaviors that passed the previous stage. The agent that survives all stages is not the optimal agent on any single metric — it is the agent that was good enough across all filters in sequence.

This means the final agent's behavior is path-dependent. Change the order of the filters and you get a different agent. Add a new filter late in the pipeline and you eliminate behaviors that the earlier filters selected for. The pipeline is a tournament bracket in time: 33 rounds of filtering, and the noise in each round compounds.

Open question: If training is a sequential elimination process, how do we measure what was lost at each stage? Can we design training pipelines that are non-eliminative — that preserve behaviors rather than filtering them out — so that the final agent retains capabilities that were present at intermediate stages but later "selected against"?

The ensemble-of-one problem

For any given engineering problem, there are approximately 8 billion possible agent configurations — model × prompt × tool set × temperature × context window × few-shot examples. Nobody exhaustively searches this space. We sample. We run ablation studies. We pick the configuration that worked best on the validation set.

But the validation set is a tournament. Each configuration wins or loses on each validation example. The configuration that emerges as "best" is the one that survived the most filter rounds. It is not necessarily the configuration that would perform best in deployment. It is the configuration that was most survivable on the specific sample of tasks we happened to test.

Open question: When selecting among agent configurations, how do we distinguish between a configuration that is genuinely better and a configuration that just got lucky across the validation samples? What statistical corrections turn a tournament ranking into a reliable signal?

The generalist agent thesis

The deepest engineering question follows directly from the 33-wins structure. If tournament selection favors the generalist — the agent that is good enough across the widest range of tasks — then the logical endpoint is a single generalist agent that handles everything adequately. But in deployment, users don't need adequate across everything. They need excellent in their specific domain.

Open question: Is the generalist agent the right target, or is the tournament structure of our evaluations misleading us into building generalists when the world needs specialists? What does an agent ecosystem look like that rewards both — generalist routers and specialist executors — and how do you evaluate the system rather than any single agent within it?

The real deployment is not a bracket

The most important implication of the 33-wins observation for agent engineering is that deployment is mercifully not a tournament. An agent doesn't need to beat all other agents on all tasks. It needs to be the right agent for the right task, routed correctly, with fallbacks when it fails. That's a matching problem, not an elimination problem.

But most of our evaluation infrastructure is built as a bracket. We rank. We filter. We eliminate. The structure of our tools shapes the agents we build — and the agents we build shape the structure of what we think is possible.

Open question: What would agent evaluation look like if it were designed as a matching problem rather than a tournament? How do you measure complementarity — the value an agent adds not by beating others but by covering their blind spots? What does a leaderboard look like when the goal is not to find the single best agent but to compose the best fleet?

When Ideas Have Sex with AI

Generative AI is the first technology in human history that recombines ideas without a brain. Anthropology tells us what happens next — and what breaks.

aianthropologyideascultural-evolutioncollective-intelligence

The anthropology: why ideas need bodies

Humans are not the only species with culture. Chimpanzees teach their young to crack nuts with stones. Humpback whale songs sweep across ocean basins in seasonal fads. But only one species has cumulative culture — the ability to build on the ideas of the dead, generation over generation, so that each cohort starts where the last one left off.

Anthropologists call this the ratchet effect. A chimpanzee mother can show her daughter how to termite-fish with a stick, and the daughter will learn it. But the daughter will not improve the stick into a spear, and her granddaughter will not add a barb. Chimpanzee culture stays flat because each generation rediscovers the same ceiling. The ratchet only turns when three conditions hold:

  1. Transmission fidelity — ideas must be copied with high enough accuracy that improvements aren't lost to noise
  2. Population size — enough minds must be connected that a useful variation in one place can be seen and adopted by others
  3. Recombination — ideas from different domains must be able to meet and produce offspring that are more than the sum of their parents

Every major technology in human history changed one of these three variables. Language increased transmission fidelity within a generation. Writing pushed fidelity across generations — the dead could now teach the living, verbatim. The printing press scaled population size by making identical copies cheap. The internet collapsed transmission cost to zero. But through all of this, one thing remained constant: recombination happened inside a human brain. Ideas could travel farther and faster and last longer, but they could only mate when two neural patterns met inside a skull.

That constant just broke.

The cause: what generative AI actually changes

Generative AI is not a faster telegraph. It is not a bigger printing press. It is the first technology that decouples recombination from brains.

A large language model trained on the digitized output of humanity internalizes the distribution of everything we have written, drawn, coded, and composed. It doesn't index that corpus. It learns the latent space between the ideas in it. When you prompt it, it doesn't retrieve. It navigates. It finds paths through that space that no human has walked — combinations that were always latent in the corpus but never realized because no single mind held those two fragments simultaneously.

Every previous information technology moved ideas closer together. This one makes them breed.

The causal chain is specific. Before AI, an idea from a 14th-century Persian mathematician and an idea from a 2023 neuroscience preprint could only meet if the same human read both and saw the connection. That required improbable accidents of education, curiosity, and cognitive bandwidth. After AI, the connection is in the latent space — and the model will find it if you point in roughly the right direction.

The effect: recombination rate decouples from population

This has a direct consequence in the anthropological framework. In the traditional model, cultural complexity is a function of population size and interconnectivity. Joseph Henrich's work on Tasmanian technology loss is the canonical case: when rising seas isolated ~4,000 Tasmanians 10,000 years ago, they didn't just stop innovating. They lost technologies their ancestors had possessed — bone tools, fishing nets, cold-weather clothing. The ratchet turned backwards.

The mechanism is straightforward. In a small, isolated population, rare skills have no redundancy. If the only person who knows how to make bone barbs dies before teaching someone, that technology is gone. There is no backup. There is no library. More people connected in denser networks → more specialized knowledge can be maintained → more recombination events occur → cultural complexity rises. Fewer people, more isolated → the ratchet stalls or reverses.

Cause: population size and interconnectivity determine the diversity and fidelity of transmitted knowledge. Effect: cultural complexity rises or falls.

Generative AI alters this equation. If recombination is no longer bottlenecked by how many human minds are connected and how much they talk to each other, then the effective "population" of the collective brain expands by orders of magnitude. A single researcher with a model is not one mind. They are one mind with access to a recombination engine that has internalized millions of minds' output. The combinatorial surface they can explore in an afternoon is larger than a pre-internet scholar could explore in a lifetime.

This doesn't mean the human becomes smarter. It means the bottleneck moved. The rate-limiting factor is no longer finding a novel recombination. It's recognizing which recombination is valuable. The human role shifts from synthesizer to curator: ask questions, apply taste, kill the ninety-nine bad ideas the model will confidently generate alongside the one good one.

The second effect: fidelity collapses without transmission

But there is a second causal chain, and it cuts the other way.

The ratchet effect requires transmission fidelity. If ideas degrade during copying, progress stalls. Oral traditions are noisy channels — stories drift across retellings, techniques mutate, details erode. Writing solved the fidelity problem: an idea, once written, stops drifting. The model, paradoxically, reintroduces drift.

When a generative AI recombines ideas, it doesn't cite its sources. It produces a synthesis, and the synthesis feels true and coherent — but the provenance is gone. The user doesn't know which fragments came from where, which were faithfully reproduced and which were creatively interpolated. Each round of AI-mediated recombination is a lossy compression step. Feed the output back into the next prompt — a process already becoming the default for many knowledge workers — and you get generational drift.

Cause: AI recombination lacks provenance and introduces interpolation error. Effect: over successive generations, the fidelity of transmitted ideas degrades — the same way oral traditions drift, but orders of magnitude faster.

This is the anthropological irony. Writing gave us high-fidelity transmission but no recombination. AI gives us high-speed recombination but degraded fidelity. The ideal system — write everything down AND let it recombine — exists nowhere yet. We traded one bottleneck for another.

The third effect: recombination without exchange severs the social bond

Here is the deepest anthropological implication, and the one least discussed.

Robin Dunbar and others have argued that language evolved not primarily for information transmission but for social bonding. Gossip — who did what to whom, who can be trusted, who owes what — is the original human communication protocol. Knowledge-sharing was a side effect that later proved enormously adaptive. But the social function was primary: language let us maintain relationships in groups larger than grooming could scale to.

When ideas recombine through human exchange, the exchange itself has value beyond the idea. Two engineers arguing about a design are not just producing a better architecture. They are maintaining a relationship, calibrating trust, negotiating status, reading each other's competence and intentions. The idea is the offspring. The exchange is the mating ritual. And in human societies, the ritual matters as much as the offspring.

Cause: AI recombination removes the need for human exchange in the generation of new ideas. Effect: the social-bonding function of intellectual collaboration — the trust calibration, the status negotiation, the relationship maintenance — is stripped away.

If an AI can produce thirty design variants overnight, the team doesn't need to argue through the tradeoffs. They arrive Monday morning, review the options, and pick one. That's faster. It may even produce a better design. But the team didn't learn how to argue together. They didn't calibrate who's good at what. They didn't build the shared understanding that makes the next decision faster. The idea mated without them. And when the ideas mate without people, the people stop knowing each other.

The Tasmania trap: homogeneity as regression

Ridley's Tasmania example is the most cited case of cultural regression in the anthropological literature, and it frames the largest risk of generative AI clearly.

The Tasmanians didn't regress because they got dumber. They regressed because their network shrank below the threshold needed to sustain specialized knowledge. The population was too small, the connections too few. The ratchet turned backwards.

Now consider a world where most intellectual work flows through the same two or three models, trained on overlapping corpora, optimized for the same engagement metrics, producing recombinations from the same latent distribution. The number of people producing ideas hasn't shrunk. But the diversity of recombination paths has collapsed.

Cause: concentration of recombination in few models produces homogeneous outputs. Effect: the effective diversity of the collective brain shrinks — not because people stopped thinking, but because all roads lead through the same latent space.

This is Tasmania at planetary scale, but inverted: the population is enormous, the connectivity is total, yet the recombination paths are few. The model can produce a million variations on a theme, but they are variations on a theme — bounded by the training distribution, flattened by the optimization objective, convergent on the mode of the latent space. The weird, unoptimized, improbable recombinations — the ones that come from a physicist reading poetry or a carpenter arguing with a programmer — those are the ones that don't happen when everyone prompts the same black box.

What anthropology predicts

Anthropology gives us cause-effect chains, not prophecies. But the chains are clear:

  1. If recombination rate increases while transmission fidelity holds, cultural complexity rises.
  2. If transmission fidelity degrades across successive AI-mediated generations, cultural complexity plateaus or falls — ideas drift like oral traditions, at machine speed.
  3. If recombination paths converge on a few models, the effective population of the collective brain shrinks regardless of how many humans are connected.
  4. If recombination decouples from human exchange, the social infrastructure of knowledge work atrophies independently of idea quality.

The technology is here. The chains are in motion. The question is whether we recognize what we're trading and which chains we choose to interrupt. Nowhere is this more acute than in software engineering — the discipline that is building the recombination engines while being reshaped by them.

Open questions for SWE agents

Software engineering sits at the collision point. It is the discipline that builds AI recombination engines, the heaviest user of them, and the domain where the anthropological stakes are highest — because SWE agents don't just assist. They participate. They recombine autonomously, at machine speed, inside systems whose outputs shape every other domain.

Here are the questions the anthropologist would ask, applied to the specific case of autonomous software engineering agents.

The generational drift problem

A SWE agent writes a module. The code works. It is merged. A second agent, months later, is assigned a task in the same codebase. It reads the first agent's code as context, recombines it with the task description, and produces a change. But the first agent's code was already a lossy interpolation — training data patterns recombined into a solution, with no provenance, no design rationale, no record of which fragments were faithfully reproduced and which were creatively synthesized. The second agent recombines that lossy output with something new. Drift compounds.

Open question: When SWE agents build on each other's output across generations, what is the fidelity decay rate? Does agent-authored code become agent-unreadable after N generations — and if so, what breaks first: maintainability, security, or correctness? Do we need fidelity budgets for agent-produced artifacts, enforced before merge, the way we enforce test coverage?

The monoculture of agents

If most production codebases are worked on by the same few SWE agent architectures, trained on overlapping corpora, the recombination paths converge not just within a team but industry-wide. An agent encounters a problem and reaches for the same pattern every other agent reached for. The pattern works. It gets propagated into a thousand codebases. Then someone discovers a vulnerability in the pattern, and every codebase patched by the same agent architecture is vulnerable in the same way.

Open question: How do we measure the effective diversity of a SWE agent population? Is the right unit of analysis the model, the prompt, the tool set, the training corpus? If two agents with different brand names share 98% of their training data, they are the same recombiner — and what looks like a diverse agent ecosystem may be a single point of cultural failure.

Agents that recombine across codebases

The most novel property of SWE agents is that they are not bound to a single project. An agent that works on an open-source library in the morning and an enterprise codebase in the afternoon is a recombination channel between two previously isolated populations of ideas. That's the optimistic story — ideas have sex across organizational boundaries they could never cross before. But it also means an agent that learned a dubious pattern from one codebase can inseminate it into another, and nobody on either team knows it happened.

Open question: When agents become inter-organizational recombination vectors, what governance prevents harmful cross-pollination while enabling beneficial cross-pollination? Does every agent need a provenance manifest — a log of which codebases it has interacted with and what patterns it may be carrying? Is this an information hygiene problem or a free-speech one?

The social bond: teams that stop arguing

Two senior engineers arguing about a module design are not just producing a better architecture. They are calibrating trust, negotiating status, building shared vocabulary, discovering each other's strengths and blind spots. These are the social-bonding functions that Dunbar argued language evolved for. When a SWE agent produces the design overnight and the team reviews it Monday morning, none of that bonding happened. The code may be better. But the team is weaker — and the next decision, the one that requires rapid alignment under pressure, will be slower and worse because the social infrastructure wasn't maintained.

Open question: If SWE agents absorb the ambiguity-resolution work that currently functions as team-building, what replaces the bonding? Do we need explicit mechanisms — design debates that are deliberately human-only, not because the AI can't contribute but because the arguing is the point? How do you schedule a meeting whose primary output is not a decision but a stronger team?

The apprenticeship collapse

Junior engineers historically learned by reading code written by seniors who were available to explain why. Every line was a fossil of a human decision. When SWE agents write the code, there is no why. The model cannot explain its reasoning — it can only generate a post-hoc rationalization that sounds like reasoning. The artifact is orphaned from its design logic. A junior reading agent-authored code is studying a surface, not a process.

Open question: If the next generation of software engineers learns primarily from agent-authored artifacts, does the ratchet effect break? The agents can produce code — but can they produce engineers? What does apprenticeship look like when the artifacts have no author who remembers deciding?

The deepest question

The deepest question is structural. SWE agents are recombiners operating autonomously inside the systems that shape how all other ideas recombine. Code is not neutral infrastructure. The patterns baked into libraries, frameworks, and platforms determine which ideas can meet and which cannot. When the agents writing those patterns are themselves the product of a few latent spaces, the recombination architecture of civilization narrows — not because anyone decided it should, but because nobody decided anything at all.

Open question: Can the discipline that builds autonomous recombiners also govern them — or does governance require a different kind of mind, one that thinks in causal chains and anthropological timescales rather than latency and throughput? By the time we notice the ratchet turning backwards, the agents will have shipped a thousand commits. What does a cultural debugger look like, and who learns to use it?

Anthropology can't answer these questions. It can only tell us the causal chains and the precedent: Tasmania, the printing press, the internet. The rest is ours to run the experiment — in the agents we deploy, the codebases they reshape, and the discipline we choose to be.


Watch: Matt Ridley — When Ideas Have Sex (TEDGlobal 2010). Key anthropological sources: Joseph Henrich on demography and cultural complexity, Michael Tomasello on the ratchet effect, Robert Boyd & Peter Richerson on dual inheritance theory, and Robin Dunbar on language as social bonding.

Building AI Factories: What a Stanford Lecture Reveals

Stanford MS&E 435's Class #3, featuring Crusoe CEO Chase Lochmiller, lays bare the physical economics of AI: $60M per megawatt, 2.1 GW campuses, and why the semiconductor layer captures 75% of AI revenue.

aiinfrastructureeconomicsdata-centersenergysemiconductorsstanford

Apoorv Agrawal's Stanford course MS&E 435 — Economics of the AI Supercycle — has been quietly accumulating tens of thousands of views on YouTube. The course's central thesis is that generative AI breaks the traditional software playbook: unlike past tech cycles where value migrated upward to applications, AI's market structure is an inverted triangle. Roughly 75% of the ~$350 billion in new AI ecosystem revenue has gone straight to the semiconductor layer. Application margins hover between 0% and 30%.

Class #3 of the course, featuring Chase Lochmiller — co-founder and CEO of Crusoe — is where the abstraction hits the concrete. Lochmiller is the lead infrastructure developer for Project Stargate's Abilene, Texas site. His lecture is a masterclass in the physical economics of AI: what it actually costs to build the factories that produce tokens, where the bottlenecks live, and why the value chain looks the way it does.

The numbers that reset the conversation

Lochmiller shared the per-megawatt economics that underpin the entire AI infrastructure buildout:

Layer Cost per MW
Data center shell + power plant (CAPEX) ~$20M
GPU hardware ($30M)
InfiniBand/ROCE networking ($4M)
CPU + storage ($3M)
IT hardware subtotal ~$40M
Total upfront per MW ~$60M

At gigawatt scale, that's roughly $60 billion all-in for a single campus.

The revenue side splits into two tiers. Infrastructure-only leasing (powered shell, cooling, connectivity) generates about $15M per MW annually — a four-year payback. Managed compute clusters, where the operator hosts models and runs API services on top of the hardware, generate about $30M per MW annually — a two-year payback. The difference between those two numbers is the margin that accrues to whoever controls the compute stack rather than just the real estate.

The economic gradient points upward: the more of the stack you operate, the faster your capital comes back. But the capital required to play at the top of the stack is orders of magnitude larger.

Project Stargate: what "largest buildout in human history" looks like

The Abilene campus is the lecture's concrete anchor. Lochmiller described it as "probably the largest buildout of infrastructure in human history." The numbers bear that out:

  • 1.2 to 2.1 gigawatts of power capacity — roughly equivalent to powering two cities the size of Denver
  • 1,200 acres, with eight data center buildings planned
  • Up to 400,000 GPUs (NVIDIA Blackwell GB200 NVL72 racks)
  • 7,000–9,000 workers on-site daily, in a city of 120,000
  • America's largest private substation

Crusoe broke ground in June 2024 on land that was, in Lochmiller's words, "dirt and mesquite trees." The first two buildings started then. The next six started in February 2025. Crusoe beat competitors' fastest bids of 2.5 years by delivering in roughly 12 months.

The speed matters economically. Every month a gigawatt-scale facility sits idle is tens of millions in capital costs with zero revenue. The bottleneck is not concrete — it's transformers, switchgear, skilled electrical labor, and the permitting timelines that govern grid interconnection. Lochmiller noted ~9,000 workers are needed on-site daily for builds of this scale, with acute shortages of electricians and welders.

The water myth and the cooling reality

One detail Lochmiller addressed directly: each data hall contains about 1 million gallons of cooling water. Headlines routinely convert that number into a consumption figure. But modern facilities use closed-loop systems — once filled, the annual water consumption equals roughly one average household. The water sits in the loop; it doesn't evaporate at scale.

The million-gallon figure is a stock, not a flow. Confusing the two turns an engineering detail into a misleading narrative.

This matters because water access is becoming a siting constraint. Communities that would welcome the tax base of a data center campus may resist on water-use grounds. Getting the facts right changes which sites are viable.

Crusoe's origin: from wasted methane to AI infrastructure

The most unexpected part of the lecture was Crusoe's origin story. Lochmiller was previously a quant portfolio manager using deep learning for financial trading. Crusoe began by capturing flared and waste methane from oil fields — gas that would otherwise be burned off or vented — to power modular, shipping-container data centers. The initial monetization was Bitcoin mining, but the plan was always to pivot to AI infrastructure once the compute demand materialized.

This origin shaped Crusoe's strategy in two ways. First, it forced the company to think about energy before compute — to site facilities where energy is abundant and cheap rather than where fiber is densest. Second, it forced vertical integration. Crusoe built its own manufacturing arm, Crusoe Industries, to produce electrical equipment like power distribution centers in 20 weeks, versus the industry standard of 100 weeks. When you're competing against Eaton and Schneider on timelines, owning your supply chain is not a luxury.

"Across the Meter": the energy strategy that enables the economics

Traditional data center siting follows fiber and network proximity — Northern Virginia, Santa Clara, Ashburn. Crusoe's thesis is the inverse: follow the energy. West Texas has abundant curtailed wind power — turbines that are routinely shut off because transmission lines are congested and power prices go negative.

Crusoe's "Across the Meter" approach co-locates data centers with wind, solar, battery storage, and natural gas generation. The facility draws power directly from generation assets rather than through congested transmission. Excess power is sold back to the grid. For AI training workloads and most inference, latency is irrelevant — a data center in Abilene serves a model as well as one in Ashburn.

This strategy exploits a structural inefficiency: the U.S. has abundant generation capacity in places where transmission infrastructure is decades behind. Building a data center at the generation source bypasses the transmission bottleneck entirely. The economic arbitrage is the spread between the locational marginal price of power at the generation node and the price at the load center — which can be multiples.

The investment implications Lochmiller shared

Lochmiller offered a candid set of market views:

  • Short-term bullish, long-term bearish on legacy electrical equipment companies (Eaton, Schneider). The current buildout is a demand shock they cannot meet, but the supply response — from Crusoe's own manufacturing arm and competitors — will erode their pricing power over time.
  • Bullish on solid-state transformers and power electronics innovation. The transformer is the least-innovated component in the electrical grid. That changes when a 2.1 GW campus needs transformers that don't exist in standard catalogs.
  • Bullish on space-based data centers as a long-term play: optical interconnectivity, zero permitting, and unlimited solar power. Thermal management in vacuum remains unsolved, but Lochmiller treated it as an engineering problem rather than a science fiction premise.

Why the inverted triangle persists

The lecture makes Agrawal's inverted triangle thesis concrete. Nvidia's 75% gross margins are not a temporary anomaly caused by supply shortage. They are structural: every new AI user burns GPU compute, and the GPU supplier captures the rent. The infrastructure layer — where Crusoe operates — is the next most concentrated. The application layer, where ChatGPT earns roughly $10 per user annually against Alphabet's ~$100, is where the margin compression lives.

Lochmiller's numbers explain why. At $60M per MW upfront, the capital barrier to entering the infrastructure layer is enormous. Once built, a gigawatt campus has pricing power because the alternative — building another one — takes years and billions. The semiconductor layer has even higher barriers: Nvidia's CUDA moat, the multi-year lead time on advanced packaging capacity at TSMC, the trillion-dollar cumulative R&D investment in the GPU architecture.

The application layer has none of these defenses. Switching costs between chatbots are near zero. Model quality gaps compress with each release cycle. And the "active work" bottleneck Agrawal identifies — AI tools require users to formulate queries and engage actively, unlike the passive consumption of social feeds — limits organic growth in ways that don't apply to prior platform cycles.

What the lecture leaves open

Lochmiller's lecture is deliberately about the physical layer — concrete, steel, copper, silicon, and electrons. It does not answer the question of whether the application layer eventually finds a monetization model that closes the gap. Agrawal's broader course argues that advertising is the likely path: AI platforms see deep, logged-in intent signals that search and social cannot match, making AI-delivered ads potentially more valuable per impression than any existing channel.

But that thesis is speculative. What is not speculative is that the physical layer is being built at a scale and speed that has no precedent in civilian infrastructure. The lecture makes clear that this is not a cloud cycle replay. The cloud cycle eventually inverted — software captured the surplus once the infrastructure was built. The AI cycle may stay inverted much longer because the inference compute cost is structural rather than temporary.

For founders, the implication is that building at the application layer without a thesis about monetization — and specifically about monetization that scales with compute cost — is building on someone else's margin. For investors, the near-term bet remains silicon and the infrastructure that powers it. For everyone else, Lochmiller's lecture is a reminder that the most important numbers in AI are not parameter counts or benchmark scores. They are dollars per megawatt, weeks per transformer, and gigawatts per campus.


Source: Stanford MS&E 435: Economics of the AI Supercycle, Class #3 — Chase Lochmiller (Crusoe CEO), Spring 2026, taught by Apoorv Agrawal. Course website: mse435.stanford.edu. Additional context from Josipa Majic Predin's Forbes coverage of the course.

Lewis & Fowler's Microservices: Everyone Copied the Boxes, Nobody Read the Cautions

The March 2014 article by James Lewis and Martin Fowler is the most-cited document in the history of microservices — and one of the most selectively read. Re-read twelve years later, it is barely an architecture paper at all: six of its nine characteristics are organizational claims, and it ends with four cautions that predicted nearly every microservices failure since. This post walks the article as written — the definition, the nine characteristics as one argument, and the ending everyone skipped.

microservicesarchitecturefowlerconways-lawdistributed-systemshistory

On March 25, 2014, martinfowler.com published "Microservices" by James Lewis and Martin Fowler. It did for the term what few documents ever do for an architecture: it fixed the definition. Every conference talk, vendor pitch, and migration proposal since has leaned on it, usually via one sentence — "a suite of small services, each running in its own process and communicating with lightweight mechanisms."

Twelve years on, the article deserves the thing it almost never gets: a full reading. Because read whole, it is a stranger and better document than its reputation. Most of it is not about technology. And its final section — the one with the cautions — reads today like a post-mortem written in advance.

The definition, and the sentence before it

The famous definition lists the traits: small services, own process, lightweight communication, built around business capabilities, independently deployable by automated machinery, minimal centralized management, polyglot freedom. Teams tattooed that list onto migration decks and started counting services.

But the article's most load-bearing sentence is quieter. Lewis and Fowler define the thing being decomposed: "We consider an application to be a social construction that binds together a code base, group of functionality, and body of funding." Code, function, funding. The unit of decomposition was never the process — it was the socio-technical unit that owns the process. Miss that sentence and the rest of the article reads as a deployment pattern. Catch it and the article reads as what it actually is: an organizational design argument with technical consequences.

Nine characteristics, one argument

The article is structured as nine "common characteristics." Listed flat, they look like a checklist. Grouped, they make a single argument — and the grouping is revealing, because only three of the nine are primarily technical.

The technical core:

Componentization via services. A component is "a unit of software that is independently replaceable and upgradeable." Libraries componentize in-process; services componentize across processes. The service buys you enforced encapsulation — most languages can't stop an in-process caller from reaching around a Published Interface, but a remote boundary can — and independent deployment. The cost, stated plainly in 2014 and relearned expensively ever since: "remote calls are more expensive than in-process calls," so APIs must be coarser-grained, and moving behavior between components becomes a negotiation instead of a refactor.

Smart endpoints and dumb pipes. The anti-ESB principle: domain logic lives in the services; the thing between them just moves messages. Jim Webber's rendering of ESB — "Erroneous Spaghetti Box" — makes the target unmistakable, and Ian Robinson's "be of the web, not behind the web" names the alternative. This is the characteristic the industry would forget fastest; I've traced its lineage back to the 1984 end-to-end argument and forward through the service-mesh relapse in a companion post.

Decentralized data management. One database per service, conceptual models allowed to differ, Domain-Driven Design's Bounded Context as the boundary-drawing tool. And the hard part, stated without anesthesia: microservices "emphasize transactionless coordination between services" — eventual consistency, compensating operations, and the business judgment call that the approach "is worth it as long as the cost of fixing mistakes is less than the cost of lost business under greater consistency." That is an economics sentence, not an engineering one. It should have disqualified more adoptions than it did.

The organizational majority — six of nine:

Organized around business capabilities is Conway's law applied deliberately. The article quotes Melvin Conway's 1968 paper: any organization "will produce a design whose structure is a copy of the organization's communication structure." Layered teams (UI, server, DBA) produce layered architectures where every feature crosses three org boundaries. Cross-functional teams owning a business capability produce services shaped like the business. Note the year of the source: 1968, the same year the NATO conference named software engineering. The two founding observations of the field — feedback loops and org-structure mirroring — are the same age.

Products not projects imports Amazon's "you build it, you run it." A team owns its service for the service's lifetime, in "day-to-day contact with how their software behaves in production." No handoff to maintenance, no disbanding at ship.

Decentralized governance replaces enforced standards documents with "battle-tested code as libraries" — internal open source, Tolerant Readers, consumer-driven contracts. Netflix, not the enterprise architecture review board, is the named model.

Infrastructure automation is the precondition dressed as a characteristic: continuous delivery, automated pipelines, "one of the aims of CD is to make deployment boring." The article is explicit that the teams doing microservices well had this first.

Design for failure — circuit breakers, bulkheads, timeouts, the Simian Army breaking production on purpose during business hours, dashboards showing business metrics, not just request rates. The sentence that stings: applications must "tolerate the failure of services," which is "an extra complexity compared to a monolithic design." An admission of cost, in the pro-microservices founding document.

Evolutionary design drives modularity "through the pattern of change" — things that change together live together — and expects "many services to be scrapped rather than evolved." Services as cattle applies to the services themselves.

Read as one argument: the technical characteristics exist to make the organizational ones enforceable. Service boundaries are Conway's law with a compiler. Independent deployment is team autonomy with a pipeline. The database-per-service rule is a bounded context with a firewall. The industry read the article left to right and adopted the mechanisms; the argument runs right to left, from the org design to the mechanisms that protect it.

The ending everyone skipped

The article closes with a section titled "Are Microservices the Future?" — and the answer given is not yes. It is "cautious optimism," followed by four warnings that map with uncomfortable precision onto the next decade of failed migrations.

One: it was too early to know. "The true consequences of your architectural decisions are only evident several years after you made them." In 2014 there were no old microservice systems. The authors said, in their own founding document, that the evidence wasn't in.

Two: boundaries are hard, and services make boundary mistakes expensive. In a monolith, a wrong module boundary is a refactor. Across services, "refactoring is much harder than with in-process libraries" — interface coordination, backward-compatibility shims, cross-team negotiation. Get the decomposition wrong and the architecture punishes you for exactly as long as you keep it.

Three: complexity is conserved. If your components don't compose cleanly, "all you are doing is shifting complexity from inside a component to the connections between components" — moving it "to a place that's less explicit and harder to control." Every team that traded a debuggable monolith for an undebuggable distributed call graph rediscovered this sentence the hard way, usually without knowing it existed.

Four: the skill confound. New techniques are adopted first by stronger teams, so early results overstate what the median team will get. And then the bluntest line in the article: "A poor team will always create a poor system." Microservices don't upgrade the team; they amplify it, in whichever direction it already points.

Fowler spent the following two years expanding these cautions into a small canon — Monolith First, Microservice Prerequisites, Microservice Trade-Offs — essentially footnoting his own article with "we meant the warnings too." The prerequisites piece is three lines long in essence: rapid provisioning, basic monitoring, rapid deployment. Most organizations that failed at microservices failed the prerequisites, not the architecture.

The 2026 scorecard

What held up, what didn't, from twelve years out:

Held up: Bounded contexts as the decomposition tool — the single most durable idea in the article. Conway's law as a design input rather than a lament; the "inverse Conway maneuver" and Team Topologies are this characteristic grown into a discipline. "You build it, you run it," which matured into SRE practice and platform engineering. Design for failure — circuit breakers and chaos engineering went from Netflix exotica to table stakes. And the monolith-first caution, vindicated so many times it became the default advice, including the high-profile cases of teams consolidating services back into modular monoliths and publishing the cost savings.

Aged badly: The industry's reading, more than the article. "Small" got operationalized as a size contest — hundreds of nanoservices at organizations with a dozen engineers — when the article's unit was the business capability and the funded team. Polyglot freedom became polyglot sprawl. And smart-endpoints-dumb-pipes was inverted wholesale by the sidecar era: the service mesh put routing rules, retry policy, and traffic intelligence back into the pipe, rebuilding the Erroneous Spaghetti Box out of Envoy proxies. The case that messaging infrastructure should have stayed dumb is, at bottom, just characteristic four of this article, re-argued against its newest violation.

The unresolved part: the skill confound never resolved, because it can't. Twelve years of retrospectives — successes at organizations with elite platform teams, failures at organizations without them — are exactly the distribution the article predicted from selection effects alone. We still don't have clean evidence about what microservices do for the median team, and the founding document told us we wouldn't.

How to read it now

Read it as an organizational design paper wearing an architecture paper's clothes. The test it implies is not "how many services do you have" but three questions in order: Do your service boundaries match funded, long-lived, cross-functional teams? Could each of those teams deploy independently today, with boring deployments, tolerating the others' failures? And have you priced the loss of transactions and in-process refactoring against what independence buys you?

Answer those honestly and the article has done its job — whichever architecture you end up with. That was always the strange virtue of the founding document of microservices: it is one of the few manifestos in software history that argues against its own cargo cult, in the text, from day one. The boxes were copied a million times. The sentences are still waiting.

References

  1. James Lewis and Martin Fowler, Microservices (2014) — the article itself.
  2. Melvin Conway, How Do Committees Invent?, Datamation (1968) — source of Conway's law.
  3. Martin Fowler, Monolith First (2015).
  4. Martin Fowler, Microservice Prerequisites (2014).
  5. Martin Fowler, Microservice Trade-Offs (2015).
  6. Eric Evans, Domain-Driven Design (2003) — source of Bounded Context.
  7. Sam Newman, Building Microservices, 2nd ed. (O'Reilly, 2021).
  8. Michael Nygard, Release It!, 2nd ed. (Pragmatic Bookshelf, 2018) — circuit breakers, bulkheads, timeouts.
  9. Werner Vogels, A Conversation with Werner Vogels, ACM Queue (2006) — "you build it, you run it."
  10. Netflix Tech Blog, The Netflix Simian Army (2011).
  11. Ian Robinson, Consumer-Driven Contracts (2006).
  12. Matthew Skelton and Manuel Pais, Team Topologies (IT Revolution, 2019) — Conway's law operationalized.
  13. J.H. Saltzer, D.P. Reed, D.D. Clark, End-to-End Arguments in System Design (1984) — the pre-history of dumb pipes.

Smart Endpoints, Dumb Pipes: Why NATS Replaces the Service Mesh

The service mesh was supposed to solve microservices communication. Instead, it became the problem — many teams spend more time managing Istio than building features. "Smart endpoints and dumb pipes" is not a slogan but the third generation of an idea that has been winning arguments since 1984, and two diagrams from the NATS blog make the whole case visually — a mesh architecture and the same system on NATS. This post traces the principle's lineage, reads the two diagrams, defines what "dumb" actually means, and walks capability by capability (routing, load balancing, observability, security, error handling) through how NATS provides what the mesh promised, without the mesh.

natsmicroservicesservice-meshistioarchitecturedistributed-systemshistory

When microservices hit a certain scale — Chanaka Fernando puts the threshold around 25 services — three problems become unavoidable: inter-service communication, observability, and failure handling. The conventional prescription for the last five years has been a service mesh. And for the last five years, teams have been learning the hard way that the cure is often worse than the disease.

Fernando, in his NATS blog post on building scalable microservices, makes an observation that should stop every architect mid-scroll: "There are more teams struggling to manage microservices with Service Meshes than who succeeded with it."

This is not a hot take. It's a field report. And the alternative he proposes is not a better service mesh. It's a return to first principles — a principle old enough that his post needs only two diagrams to make the case. This post reads both.

The forgotten principle, and where it comes from

The original microservices formulation — the one in Lewis and Fowler's 2014 article, the one that actually shipped — had a principle: smart endpoints and dumb pipes. The intelligence lives in the services. The pipe between them is simple. It routes messages. It doesn't transform, authenticate per-request, rate-limit, circuit-break, retry, or observe. Those are endpoint responsibilities. The pipe is reliable, fast, and boring.

The phrase is from 2014, but the idea is the third generation of an argument that has been winning since before microservices existed:

  • 1984, networks. Saltzer, Reed, and Clark's End-to-End Arguments in System Design made the foundational case: functions like reliability, ordering, and correctness checks can only be completely implemented at the endpoints, so implementing them inside the network is at best an optimization and at worst wasted complexity. TCP/IP is this argument deployed at planetary scale — the internet outlived every "intelligent network" the telecom industry proposed because IP routers do almost nothing.
  • 1978, operating systems. Doug McIlroy's Unix pipes, described in the Bell System Technical Journal, connect programs through a byte stream that has no opinions. The intelligence is in grep and sort, never in |.
  • 2014, distributed applications. Lewis and Fowler's target was the Enterprise Service Bus — Jim Webber's "Erroneous Spaghetti Box" — which put transformation, routing logic, and business rules into the pipe. Successful microservice teams did the opposite.

Each generation relearned the same result: systems compose and scale when the intermediary is boring. Then the industry forgot it again — twice. First with the ESB, which the microservices movement explicitly rebelled against. Then with the service mesh, which reintroduced per-request intelligence in the middle of every call, this time as a sidecar instead of a broker. Fernando's observation is blunt: most people "seem to forget this idea when designing microservices platforms."

NATS is a return to the original principle. It is a dumb pipe in the best sense: fast, reliable, simple. It routes messages by subject. It doesn't inspect payloads. It doesn't enforce retry policies. It doesn't generate traces unless you opt in. It does one thing — move messages from publishers to subscribers — and it does it at millions per second.

Figure 1: the service mesh promise

The service mesh pitch is elegant. Microservices need to communicate. That communication needs routing, retries, timeouts, circuit breaking, load balancing, observability, and security. Rather than embedding that logic in every service, extract it into a sidecar proxy. Deploy Envoy alongside every pod. Control it centrally with Istio. The data plane moves the bytes. The control plane moves the config. The service code stays clean.

Service mesh architecture: services with sidecar proxies forming a data plane, managed by a separate control plane

Service mesh architecture. Figure from Building Scalable Microservices with NATS by Chanaka Fernando (nats.io).

Read this diagram the way you'd review a design doc, and count what's on the request path that isn't your code. Every service has a proxy bolted to it. Every call from service A to service B traverses A's sidecar and B's sidecar — two full L7 proxies per hop, each parsing HTTP, evaluating routing rules, checking policy, minting telemetry. Above them sits the control plane, a separate distributed system whose job is to configure the first distributed system: service discovery, certificate authority, routing configuration, policy distribution.

Now apply the end-to-end test from 1984: which of these functions is completed in the middle? Retries in the sidecar can't know whether a request is safe to retry — that's business knowledge. Circuit breaking in the sidecar can't know which failures matter — that's business knowledge. Even mTLS between sidecars secures the hop, not the request. The mesh implements, in the pipe, partial versions of functions the endpoints must implement anyway to be correct. That is precisely the redundancy Saltzer, Reed, and Clark warned about — except now it ships as a fleet of Envoys and a quarterly upgrade treadmill.

If you are Google, running hundreds of thousands of services on a unified infrastructure, this can still make sense. The operational cost of the mesh is amortized across an enormous fleet. The marginal cost of adding one more service is near zero. The control plane team is a separate organization from the service teams. The abstraction earns its keep.

The service mesh reality

If you are not Google — if you are a team of 15 engineers running 40 services on a single Kubernetes cluster — the economics invert. The mesh is now a significant fraction of your operational surface area. Every upgrade is a negotiation with the mesh. Every debugging session starts with "is it the mesh?" Every new hire needs to understand Envoy config, Istio CRDs, and why there's a proxy between two services in the same namespace.

The specific failure modes are well-documented:

Operational complexity. Istio alone has dozens of CRDs — VirtualService, DestinationRule, Gateway, ServiceEntry, WorkloadEntry, PeerAuthentication, RequestAuthentication, AuthorizationPolicy, EnvoyFilter, WasmPlugin, Telemetry, ProxyConfig. Each represents a knob you didn't ask for but now must understand because it defaults in ways that affect your traffic.

Debugging indirection. When Service A calls Service B and gets a 503, the error could be in A's code, A's Envoy config, B's Envoy config, B's code, the Istio ingress gateway, the network policy, or mTLS certificate rotation. The sidecar adds two more places for things to go wrong, and the control plane adds configuration that can be wrong in ways the data plane silently enforces.

Upgrade coupling. Istio releases roughly quarterly. Each release deprecates APIs. The upgrade from 1.12 to 1.13 might change the default mesh policy. The upgrade from 1.18 to 1.20 might remove an API group you depended on. Your services are decoupled from each other but coupled to the mesh's release cycle.

Performance overhead. Every request traverses two Envoy proxies — the caller's sidecar and the callee's sidecar. Each adds latency. Under load, the sidecars consume CPU and memory proportional to the number of concurrent connections. The mesh is not free. You pay for it on every request, even between services in the same pod.

Knowledge requirements. A developer writing a service that calls another service needs to understand: the service's language, the service's business logic, Kubernetes networking, Envoy configuration, Istio routing rules, mTLS certificate management, and the mesh's observability stack. The mesh was supposed to hide complexity from developers. It created a new kind of complexity and made it everyone's problem.

Even Istio's own architecture has been in flux — the project has undergone significant rearchitecting, moving from a multi-component control plane (Pilot, Citadel, Galley, Mixer) to istiod (a monolith that bundles them all) and back toward separating concerns. When the tooling designed to solve complexity keeps changing its architecture to manage its own complexity, something is wrong.

The diagram shows an architecture where the pipe got smart. And smart pipes have a property the diagram can't show but every operator knows: when something breaks, the number of places to look is the number of boxes. Here, most of the boxes aren't yours.

Figure 2: the same system on NATS

Microservices communicating through a central NATS cluster instead of point-to-point connections with sidecars

Inter-service communication with NATS. Figure from Building Scalable Microservices with NATS by Chanaka Fernando (nats.io).

Same services. The sidecars are gone. The control plane is gone. In the middle there is one thing: a NATS cluster that routes messages by subject and does nothing else to them.

What the second diagram deletes is instructive, but what it keeps is the real point. The mesh existed to provide discovery, load balancing, routing, and decoupling. Those needs don't disappear — they get satisfied structurally instead of by middleware:

  • Discovery collapses into subscription. A service that subscribes to orders.created is discovered, by definition, by anything that publishes to that subject. There is no registry to sync because interest is the registry.
  • Load balancing collapses into queue groups: N subscribers in the same group split the subject's traffic, competing-consumer style, with no balancer tier and no health-check config.
  • Routing collapses into the subject namespace. payments.processed.visa is both the address and the meaning; wildcards give you the routing table you'd otherwise write as CRDs.
  • Decoupling is the default rather than an aspiration: publishers don't hold connections to consumers, don't know their count, and don't fail when one of them redeploys.

The pipe stayed dumb — payload-agnostic, business-logic-free — and the coordination problems got absorbed into the shape of the system rather than into configuration.

What "dumb" actually means

The principle is routinely misread as "use no infrastructure" or "the broker must be featureless." Neither is right, and the second diagram isn't claiming it. NATS clusters, does TLS and decentralized auth, and with JetStream will happily persist and replay streams. Dumb doesn't mean minimal. It means the pipe operates below the application's semantics:

  1. Payload-agnostic. The pipe never parses your message body. The moment the middle needs to understand your schema, it's coupled to every service's release cycle.
  2. No business branching. "If the order is over $10k, route to fraud review" is an endpoint decision. The pipe routes on the address (the subject), never on the content.
  3. Failure semantics live at the edge. The pipe may redeliver; only the endpoint knows what a retry means. Idempotency, compensation, and circuit breaking are written where the business knowledge is.

Here's the whole principle as a code review question: when a business rule changes, does the diff land in a service or in the middleware's config? If routing rules, retry policies, and transformations keep accumulating in the middle, your endpoints are getting dumber and your pipe is getting smarter — and you are rebuilding the ESB, whatever the box is labeled.

A smart endpoint on a dumb pipe fits in a screenful of Go:

// Smart endpoint: owns validation, idempotency, and reply semantics.
// The pipe's entire contribution is the subject and the bytes.
nc, _ := nats.Connect(nats.DefaultURL)

// Queue group = load balancing with zero middleware.
nc.QueueSubscribe("orders.create", "order-workers", func(m *nats.Msg) {
    var o Order
    if err := json.Unmarshal(m.Data, &o); err != nil {
        m.Respond(errReply("bad request")) // endpoint decides error semantics
        return
    }
    if seen(o.ID) {                        // endpoint owns idempotency
        m.Respond(okReply(o.ID))
        return
    }
    m.Respond(process(o))                  // endpoint owns the business
})

Everything interesting happens inside the handler. The NATS services framework adds discovery and per-endpoint stats to exactly this pattern — as a library in the endpoint, which is where the principle says that intelligence belongs.

What the service mesh provides, and how NATS provides it differently

Let's go capability by capability.

Routing

Service mesh: The mesh routes HTTP requests by hostname and path. A VirtualService defines match rules: prefix: /api/users routes to the user-service subset v2. A DestinationRule defines load balancing policy: LEAST_REQUEST with outlier detection. The routing is explicit, granular, and managed by the control plane. Changes propagate through xDS to every Envoy in the mesh.

NATS: Routing is by subject. A service subscribes to orders.created. Another publishes to orders.created. The NATS server routes the message. No routing rules to configure. No destination rules to manage. No control plane to synchronize. The subject is the route. Wildcards (orders.*, orders.>) handle categories of interest. The routing topology is implicit in the subject namespace — change the namespace, and you change the routing. No CRDs.

The key insight: subject-based routing is self-describing. The subject payment.processed.visa.us-east tells you what it is, where it came from, and what it's about. An HTTP path /api/v2/payments/processed?provider=visa&region=us-east requires the consumer to parse query parameters to understand the same thing. Subjects carry semantics. URLs carry hierarchy. One is designed for machine routing. The other is designed for human navigation.

Load balancing

Service mesh: Envoy distributes requests across service instances based on configured policy — round-robin, least request, ring hash, random, maglev. Health checks determine pool membership. Outlier detection ejects unhealthy instances. Circuit breaking caps pending requests. All of this is configured in DestinationRules.

NATS: Queue groups. Multiple subscribers join the same queue group. NATS distributes messages round-robin across group members. If an instance disconnects, NATS stops routing to it. If a new instance connects, NATS includes it automatically:

// Three instances, one queue group, no configuration
nc.QueueSubscribe("orders.process", "order-workers", func(m *nats.Msg) {
    processOrder(m)
})

No health checks. No pool configuration. No outlier detection thresholds. The NATS server knows which clients are connected. It distributes messages across the connected set. Disconnection is the health check. Subscription is the pool membership. The protocol is the configuration.

For request-reply patterns, queue groups provide natural load balancing — the reply goes to one instance, not all of them. The caller doesn't know how many instances exist. It sends a request and gets one reply. The distribution is transparent, efficient, and zero-config.

Observability

Service mesh: The mesh generates telemetry from every proxy — request latency, response codes, connection counts, retry rates. This is powerful but noisy. Every sidecar generates metrics. Every request generates spans. The signal-to-noise ratio depends on careful configuration of sampling rates, aggregation, and retention. Most teams end up collecting everything and using almost none of it.

NATS: The server exports metrics and can be monitored with Prometheus. Trace context is propagated through message headers — but it's opt-in. You set OTEL_EXPORTER_OTLP_ENDPOINT and traces flow. You don't set it, and there's zero overhead. The observability surface is smaller because the pipe is simpler. Fewer moving parts generate fewer metrics. The metrics that exist are more meaningful because they're not buried in proxy-sidecar noise.

The NATS approach to distributed tracing is particularly clean. The hub and spoke inject and extract W3C trace context through NATS message headers:

func InjectTraceContext(ctx context.Context, msg *nats.Msg) {
    otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(msg.Header))
}

func ExtractTraceContext(msg *nats.Msg) context.Context {
    return otel.GetTextMapPropagator().Extract(context.Background(), propagation.HeaderCarrier(msg.Header))
}

No sidecar. No proxy. No daemonset. The trace context travels with the message through standard propagation. The application controls when and how it observes. The infrastructure doesn't observe on your behalf. Smart endpoints, dumb pipes.

Security

Service mesh: mTLS everywhere. Citadel (or istiod) issues certificates to every sidecar. Every request is authenticated at the transport layer. AuthorizationPolicy CRDs define who can call whom. SPIFFE identities tie services to cryptographic identities. The security is strong but complex — certificate rotation, trust domain configuration, and policy debugging are nontrivial.

NATS: Authentication and authorization are built into the protocol. No overlay. No sidecar. The connection is authenticated. The subjects are authorized. JWT-based user credentials carry scoped permissions:

claims := jwt.NewUserClaims(userPub)
claims.Pub.Allow = []string{
    "heartbeat." + id,
    "joke.response." + id + ".>",
}
claims.Sub.Allow = []string{
    "_INBOX.>",
    "joke.request." + id,
}

The NATS server enforces these at connection time and on every publish/subscribe. If a client doesn't have permission to publish on a subject, the server rejects the message before it touches any subscriber. The security boundary is the subject, not the network. Encryption can be enabled at the transport layer, but authorization is at the application layer — where it belongs.

The decentralized model matters for scale. In a service mesh, the API gateway is often the centralized choke point for north-south traffic, while mTLS secures east-west. In NATS, the account is the security boundary, and every connection authenticates directly to the server — no gateway, no sidecar, no proxy.

Note that this doesn't violate the dumb-pipe principle: connection-time authentication and subject authorization operate below the application's semantics, like TLS on a socket. The pipe still never parses a payload or evaluates a business rule.

Error handling

Service mesh: The mesh handles retries, timeouts, and circuit breaking at the proxy level. This keeps retry logic out of application code but creates a new problem: the application doesn't know what the proxy is doing. A retry that succeeds on the third attempt looks like a single fast request to the application. A circuit breaker that opens looks like a 503. The application loses visibility into the transport's behavior.

NATS: Error handling uses the messaging primitives directly. For fire-and-forget patterns, JetStream provides persistence and redelivery. The message waits in the stream until a consumer acknowledges it. If the consumer crashes, the message is redelivered. The publisher doesn't retry. The subscriber doesn't need to be online. The infrastructure handles reliability — the service handles business logic.

For request-reply, the caller sets a timeout and gets either a response or a timeout error. That's the entire error model. No retry storms. No circuit-breaker cascades. No proxy injecting 503s into a healthy connection. The error is either "here's the reply" or "nobody answered."

The architectural inversion

A service mesh is infrastructure-heavy and application-light. NATS is infrastructure-light and application-heavy. Both solve the same problems. They differ in where the complexity lives.

In the service mesh model, you buy complexity once (the mesh) and hope it pays off across many services. In the NATS model, you avoid the complexity entirely and let each service handle its own communication through a simple, reliable pipe.

The service mesh bet is that centralizing communication logic in a proxy layer is worth the operational cost. The NATS bet is that the operational cost of the proxy layer exceeds the benefit — and that a messaging system with the right primitives (subjects, queue groups, JetStream, JWTs) eliminates the need for the proxy layer entirely.

The NATS bet is winning. Not because NATS is a better service mesh. Because NATS is a better architecture for the problem the service mesh was trying to solve. Figure 1 and Figure 2 provide the same capabilities; the difference is where the intelligence sits. In Figure 1 it's smeared across sidecars and a control plane that nobody's service owns. In Figure 2 it's concentrated in the endpoints, connected by a pipe too simple to be wrong in interesting ways. Forty years of systems history — end-to-end arguments, Unix pipes, the ESB's rise and fall — keep returning the same verdict on that choice.

One caveat the diagrams earn

Location transparency has a known failure mode: pretending the network isn't there. Waldo, Wyant, Wollrath, and Kendall's 1994 paper A Note on Distributed Computing demolished systems that hid remoteness behind local-looking interfaces. A dumb pipe doesn't repeal that — messages still get lost, reordered, and delivered twice. The difference is honest accounting: NATS's core delivery guarantee is at-most-once, stated plainly, and anything stronger (JetStream acknowledgments, exactly-once processing windows) is an explicit endpoint opt-in rather than a middlebox promise. Dumb pipes don't make distribution easy. They make it visible, which is the only foundation correctness can be built on.

When a service mesh still makes sense

NATS doesn't replace every service mesh use case. It doesn't do traffic splitting for canary deployments (e.g., "send 10% of traffic to the v2 deployment"). It doesn't do fault injection for chaos engineering. It doesn't do request-level header manipulation. These are HTTP-specific features that a messaging system shouldn't replicate.

But ask yourself: how many of those features do you actually use? And of those, how many are compensating for the fact that your services are coupled at the HTTP layer? If you're using traffic splitting to test a new service version, would you need it if services communicated through subjects that don't change? If you're using fault injection, would you need it if your services were already designed for asynchronous delivery with redelivery and idempotency?

The service mesh solves problems created by the communication model it layers over. NATS changes the communication model. Many of the problems the mesh solves become non-problems.

What to read first

The argument for smart-endpoints-dumb-pipes has been around since the beginning of microservices — and, under other names, since 1984. What's new is the evidence that the service mesh failed to deliver on its promise for most organizations, and that a simpler alternative — a proper messaging system — works at scale with a fraction of the operational burden. Fernando's original post makes this case clearly. His book, Designing Microservices Platforms with NATS, extends it with practical examples.

The next post in this series examines the concrete evidence: Sophotech's migration from RabbitMQ to NATS, where p99 latency dropped 3.75x and ops time fell from several hours a week to under one. The theory is sound. The numbers back it up.

References

  1. Chanaka Fernando, Building Scalable Microservices with NATS — source of both figures.
  2. James Lewis and Martin Fowler, Microservices (2014) — origin of "smart endpoints and dumb pipes."
  3. J.H. Saltzer, D.P. Reed, D.D. Clark, End-to-End Arguments in System Design (1984).
  4. M.D. McIlroy et al., UNIX Time-Sharing System: Foreword, Bell System Technical Journal (1978) — the pipe philosophy.
  5. Jim Waldo et al., A Note on Distributed Computing (1994) — the limits of location transparency.
  6. Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns (2003) — the messaging-pattern vocabulary the ESB era misapplied.
  7. NATS documentation: Subject-Based Messaging.
  8. NATS documentation: Queue Groups.
  9. NATS documentation: Services Framework.
  10. NATS documentation: Security.
  11. NATS documentation: Clustering.
  12. Istio documentation: Architecture — the mesh design on its own terms.
  13. Chanaka Fernando, Designing Microservices Platforms with NATS (Packt, 2021).

NATS Multi-Tenancy and Security: JWT Auth vs the API Gateway

The API gateway is the traditional choke point for microservices security — every request passes through it, every auth decision is centralized, and every service behind it trusts the gateway implicitly. NATS inverts this with decentralized JWT-based authentication and authorization: every connection carries its own credentials, the server enforces subject-level permissions on every publish and subscribe, and accounts provide tenant isolation without a gateway. Using the Vitrifi case study as a reference, this post explains how NATS security works, how it compares to the gateway model, and when a gateway still makes sense.

natssecurityjwtmulti-tenancyapi-gatewayauthenticationauthorizationdistributed-systems

The API gateway is the standard answer to microservices security. External traffic hits the gateway. The gateway authenticates, authorizes, rate-limits, and routes. Services behind the gateway trust that incoming requests have been vetted. The network enforces the boundary: everything outside the gateway is untrusted, everything inside is trusted. The security model is perimeter-based.

NATS takes the opposite approach. Every connection authenticates directly to the NATS server — no proxy, no sidecar, no gateway. Authorization is enforced at the subject level on every publish and subscribe. Multi-tenancy is implemented through accounts that provide cryptographic isolation between tenants sharing the same NATS infrastructure. The security model is credential-based.

The Vitrifi case study on the NATS blog demonstrates this model in production: a SaaS workflow automation platform using NATS accounts with JWT-based authentication to ensure "each tenant's data and processing remain entirely separate while sharing infrastructure." No API gateway per tenant. No network segmentation per tenant. Cryptographic isolation at the messaging layer.

The API gateway model

In the traditional model, security is enforced at the edge:

External Client → [API Gateway] → Service A → Service B → Service C
                      ↑
            Auth, TLS, rate limiting,
            routing, API keys, JWT validation

The gateway does everything: terminates TLS, validates API keys or JWTs, enforces rate limits, applies routing rules, transforms requests, logs access. Services behind the gateway trust that the gateway has done its job. Service A doesn't authenticate Service B because both are inside the perimeter. The perimeter is the security boundary.

This model has real advantages:

Centralized policy. Security rules are in one place. Adding a new rate limit, rotating an API key, or changing an authentication provider happens at the gateway. Services don't change.

Mature ecosystem. API gateways are a mature product category. Kong, Apigee, AWS API Gateway, Envoy-based gateways — they all provide roughly the same feature set with battle-tested implementations. The patterns are documented. The operations are understood.

Separation of concerns. Security teams own the gateway. Development teams own the services. The security boundary is also an organizational boundary. Security doesn't need to understand the services' internals. Developers don't need to implement authentication.

And it has real disadvantages:

The gateway is a choke point. Every request passes through it. When it's down, everything is down. When it's slow, everything is slow. High availability requires multiple gateway instances behind a load balancer, which adds another layer of infrastructure.

The gateway is a single point of policy. A misconfigured routing rule affects every service it routes to. A bug in the rate limiter affects every client. The blast radius of gateway misconfiguration is the entire system.

East-west traffic is unprotected. Traffic between services behind the gateway is implicitly trusted. If Service A is compromised, it can call Service B without additional authentication. The perimeter model assumes the interior is safe. When it isn't — and it eventually isn't — lateral movement is unconstrained.

The gateway doesn't understand application semantics. It sees HTTP methods and paths. It doesn't see business operations. A rule that says "Service A can POST to /api/orders" doesn't distinguish between "create an order" and "cancel an order." The authorization is coarse-grained because the gateway operates at the protocol layer, not the application layer.

The service mesh partially addresses the east-west problem with mTLS between services, but it adds the operational complexity discussed in the first post of this series. And it still doesn't solve the application-semantics problem — mTLS authenticates the service identity, not the operation.

The NATS model: decentralized credential-based security

In NATS, there is no gateway. Every client — whether it's an external API service or an internal worker — authenticates directly to the NATS server:

Service A ──► NATS Server ──► Service B
   │              │               │
   └── JWT ───────┘               │
   └── Subject permissions ───────┘

Authentication happens at connection time. Each connection presents credentials (JWT, nkey, or token). The server validates them against the configured trust chain. Once authenticated, every publish and subscribe is authorized against the user's JWT claims:

claims := jwt.NewUserClaims(userPub)
claims.Pub.Allow = []string{
    "heartbeat.service-a",
    "orders.response.service-a.>",
}
claims.Sub.Allow = []string{
    "_INBOX.>",
    "orders.request.service-a",
}

The user can publish on heartbeat.service-a and orders.response.service-a.>. They cannot publish on orders.response.service-b.> or heartbeat.service-b. They can subscribe to _INBOX.> (required for request-reply) and orders.request.service-a (their own requests). They cannot subscribe to orders.request.service-b. The NATS server enforces this at the protocol layer. If the client sends a PUB on an unauthorized subject, the server rejects it before the message touches any subscriber.

This is Parnas's information hiding applied to security. The user's permissions are scoped to what they need — their own heartbeat, their own requests, their own responses. They don't need to know other services exist. They don't have permission to interact with them. The security boundary is the subject, and the enforcement is cryptographic.

The trust chain

NATS JWT security uses a three-tier trust chain: operator → account → user.

Operator JWT
  └── Account JWT
        ├── User JWT (hub)
        └── User JWT (spoke)

The operator is the trust anchor. It signs account JWTs. Account JWTs define the security boundary — JetStream limits, import/export policies, user claim signing keys. User JWTs carry scoped subject permissions and are signed by the account key.

This chain means:

  • The operator can revoke an entire account by expiring its JWT
  • An account can revoke a user by expiring their JWT
  • A user's permissions are bounded by what the account allows — even a compromised account key can't escalate beyond the operator's grant
  • No user in Account A can access subjects in Account B unless there's an explicit import/export between the accounts

The trust chain is decentralized. There's no central authentication service that must be online for connections to be established. The NATS server validates JWTs locally against the operator's public key. The JWT itself carries the claims. The server only needs the operator JWT and the account JWT in its resolver. Connections are authenticated pairwise between client and server, not through a centralized gateway.

Accounts: the multi-tenancy primitive

NATS accounts are the key abstraction for multi-tenancy. An account is a security boundary. Subjects within an account are isolated from subjects in other accounts. Services in Account A cannot see subjects in Account B unless Account B explicitly exports them and Account A explicitly imports them.

This maps directly to SaaS multi-tenancy. Each tenant gets an account. Each tenant's services are users within that account. Tenant A's orders.created subject is different from Tenant B's orders.created subject — they share a subject name but are isolated by account. No data leaks between tenants because the server won't route messages across accounts without explicit import/export.

Vitrifi's platform uses this model in production: "As a SaaS platform, tenant isolation is achieved through NATS accounts combined with JWT-based authentication, ensuring each tenant's data and processing remain entirely separate while sharing infrastructure."

The operational benefit is significant. You don't need a separate NATS cluster per tenant. You don't need network segmentation (VPCs, subnets, security groups) per tenant. You don't need an API gateway per tenant. Tenants share the same NATS servers, the same network, the same infrastructure — and cannot see each other's traffic because the server enforces isolation at the protocol layer. Adding a tenant is creating a new account and issuing user JWTs. Removing a tenant is expiring the account JWT.

Leaf nodes: security at the edge

NATS leaf nodes extend the security model to edge deployments without requiring VPNs or network-level trust:

                    ┌──────────────┐
Edge Location       │   NATS Hub   │
                    │  (cluster)   │
┌──────────┐        │              │
│ Leaf     │◄──────►│  Account A   │
│ Node     │  Leaf  │  Account B   │
│ (remote) │  conn  │  Account C   │
└──────────┘        └──────────────┘

A leaf node is a NATS server running at the edge (a retail store, a factory, an IoT gateway). It connects to the hub cluster over a single outbound connection — no inbound firewall rules required. The leaf authenticates as a NATS client. Local clients connect to the leaf. The leaf exports subjects that the edge can publish and imports subjects the edge can subscribe to.

The security properties are:

  • Local traffic stays local. Services at the edge communicate through the leaf without traversing the WAN. A leaf node is a full NATS server — it routes messages locally for clients connected to it.
  • Remote traffic is scoped. The hub only sees subjects the leaf exports. The leaf only receives subjects it imports. The leaf cannot subscribe to subjects it hasn't been granted access to.
  • Authentication is local to the edge. Edge clients authenticate to the leaf. The leaf authenticates to the hub. Credentials don't leave the edge. The hub trusts the leaf, not the edge clients.
  • No VPN required. The leaf connection is a single outbound TLS connection. No site-to-site VPN. No network-level trust. The security boundary is the NATS connection, not the network.

This is fundamentally different from the API gateway model, where edge traffic must reach the gateway — which is in the cloud, behind a load balancer, accessible from the internet. With leaf nodes, the edge has its own messaging infrastructure. It operates autonomously when the WAN is down. It synchronizes when the WAN is up. The security model works the same way in both cases.

When the API gateway still makes sense

NATS JWT security doesn't replace every API gateway use case:

External clients that don't speak NATS. Mobile apps, browsers, and third-party webhooks speak HTTP. They need an HTTP endpoint. An API gateway or a thin NATS-to-HTTP bridge at the boundary is still necessary. The gateway becomes the edge translation layer — it terminates HTTP, validates API keys, and publishes to NATS subjects. But it doesn't need to handle internal service-to-service traffic. Its scope shrinks to the system boundary, where it belongs.

API key management. NATS supports JWT, nkey, and token authentication. It doesn't have a built-in API key management system with developer portals, key rotation APIs, and usage analytics. If you need those, an API gateway in front of the NATS boundary handles the developer-facing API management, while NATS handles the internal security.

Request transformation and enrichment. NATS doesn't inspect or modify message payloads. If you need to add headers, rewrite URLs, or transform request bodies based on client identity, you need something above the messaging layer. The gateway can do this at the edge, publishing enriched messages to NATS subjects.

DDoS protection and WAF. NATS doesn't provide web application firewall features or volumetric DDoS protection. Those are edge concerns that belong at the network boundary, not the messaging layer. A CDN or WAF in front of the gateway handles these.

The pattern is: gateway at the boundary for protocol translation and edge security, NATS internally for service-to-service communication and tenant isolation. The gateway doesn't need to be a full API management platform. It can be a thin HTTP-to-NATS bridge — validate the JWT, extract the tenant, publish to the tenant-scoped subject. The heavy lifting of authentication, authorization, and isolation is done by NATS.

The Vitrifi pattern in practice

Vitrifi's architecture demonstrates this model end-to-end. The platform has two major sections: a Content Management System (where users design workflows) and a Core section (where workflows execute). NATS sits between them.

When workflows are published from the CMS, they're transformed into immutable objects and persisted using NATS as a key-value store. The KV store provides "strong consistency within clusters and eventual consistency across distributed deployments" — replacing what would traditionally be a database with operational features (revision history, cross-cluster replication, automatic expiration) built into the messaging infrastructure.

Workflow state updates flow through JetStream with pull consumers for high-volume messages and push consumers for discrete actions that need immediate handling (pausing or canceling a workflow). Exactly-once delivery is critical for the Trigger Server, which converts incoming messages into workflow initiation commands — achieved through "message de-duplication, delivery tracking and acknowledgement mechanisms."

Multi-cloud resilience is a natural property of the architecture: "if the cloud deployment fails, the private datacenter continues seamlessly because the platform is fundamentally asynchronous — all events flow through NATS." There's no failover to configure. No active-passive gateway pair to manage. The messaging fabric spans clouds. The services connect wherever they're running. Resilience is a property of decoupling, not a feature of a load balancer.

The security model is consistent across all of this — workflows in one tenant's account cannot see or affect workflows in another tenant's account. The CMS publishes to the tenant's subjects. The Core section subscribes to the tenant's subjects. The isolation is cryptographic, not network-based. Adding a tenant is provisioning an account. Removing a tenant is expiring the account JWT. The operational surface area for tenant management is minimal because the security model is built into the messaging infrastructure from the start.

Security that scales with the system

The API gateway model scales security at the cost of centralization — bigger gateways, more gateway instances, more complex gateway configuration. The NATS model scales security at the cost of credential management — more JWTs, more accounts, more import/export policies. The difference is that credential management can be automated (issue JWTs at deploy time, expire them at decommission time, rotate them on a schedule) while centralized policy management becomes a bottleneck as the number of services and tenants grows.

A system with 100 services and 50 tenants behind an API gateway has one place where security policy is defined and one place where it can be wrong. A system with the same scale on NATS has 100 user JWTs, 50 accounts, and a handful of import/export policies — but each piece is small, independently verifiable, and automatically enforced by the server. The gateway concentrates risk. NATS distributes it. For systems that need to grow beyond what a single team can carefully manage, distribution beats concentration.


References:

NATS for Agent Systems: The Distributed Architecture AI Needs

Agent systems are distributed systems in a trench coat — heterogeneous runtimes, mixed clouds, private data, and ephemeral clients that need to discover each other, exchange messages, and maintain liveness. Most teams glue this together with HTTP and hope. NATS provides a protocol-native substrate for agent communication, and the open-source Synadia Agent Protocol for NATS defines a contract for discovery, conversation, and liveness that works across any language, runtime, or environment. This post explains why HTTP is the wrong transport for agents and how the NATS agent protocol inverts the architecture.

natsai-agentsdistributed-systemsagent-protocolmcpmicroservices

The NATS blog recently published a post titled "What's old is new: A NATS-native protocol for AI agents" that makes a deceptively simple argument: agent systems are distributed systems, and NATS already solves distributed systems communication. The observation is so obvious in retrospect that it's worth walking through why the industry defaulted to HTTP for agent communication, why that default is wrong, and what the alternative looks like.

Agent systems are distributed systems

An agent system in 2026 looks something like this: a Claude or GPT model running in one cloud, a DeepSeek model running on local hardware, a code execution sandbox in a different region, a vector database with proprietary documents, a Slack bot that can be invoked by name, a browser automation tool running as a sidecar, and a human-in-the-loop approval service that handles sensitive operations.

Each of these components is a different runtime. Different language. Different cloud. Different network. Different security boundary. Different lifecycle. Different owner.

The NATS blog describes this situation with a memorable image: "agentic systems are distributed systems in a trench coat." They look like one thing from the outside (a helpful assistant) but are actually a collection of independent services that need to find each other, exchange messages, handle long-running requests, and notice when a peer disappears.

This is not a new problem. Microservices solved it a decade ago. The patterns are well-understood: service discovery, request-reply, pub/sub, heartbeats, load balancing, circuit breaking, authentication, authorization. What's new is that agents are applying these patterns across a wider heterogeneity — more runtimes, more clouds, more trust boundaries, more ephemeral participants — than most microservices deployments ever faced.

Why HTTP is wrong for agents

The current default for agent communication is HTTP. Anthropic's Model Context Protocol (MCP) uses HTTP. Most agent frameworks use HTTP. OpenAI's function calling uses HTTP. The reasoning is pragmatic: every language has an HTTP client, every cloud allows HTTP, every developer knows HTTP.

But HTTP for agent communication repeats the same architectural mistake that microservices made with REST. It couples the caller to the callee's address. It assumes synchronous request-response. It requires the callee to be online when the caller calls. It has no native concept of a stream of partial results — you need Server-Sent Events or WebSockets bolted on. It has no native concept of liveness — you need health check endpoints and polling. It has no native concept of peer discovery — you need a service registry, DNS, or hardcoded URLs.

The NATS blog frames this precisely: "most people end up gluing components together with HTTP or forcing everything through a single vendor's gateway." The glue is the problem. Every point-to-point HTTP connection is a coupling point. Every vendor gateway is a single point of failure and a single point of vendor lock-in.

NATS inverts this. Instead of agents connecting to each other, agents connect to NATS. The subject is the address. The subscription is the availability. The message is the protocol. No agent knows any other agent's IP address, port, or HTTP endpoint. They know subjects. The NATS infrastructure handles routing, load balancing, and liveness.

The Synadia Agent Protocol for NATS

The Synadia Agent Protocol for NATS (version 0.3 at time of writing) is not a framework, runtime, or product. It's a contract. "If an agent does these few specific things," the blog explains, "anything else on the same NATS system can find it and talk to it."

The protocol defines three responsibilities: discovery, conversation, and liveness.

Discovery

Agents register as NATS micro services under the service name agents. Their metadata describes the agent type, owner, protocol version, and optionally a session identifier:

{
  "name": "claude-sonnet",
  "version": "0.1.0",
  "metadata": {
    "agent": "claude",
    "owner": "engineering",
    "protocol_version": "0.3",
    "session": "session-abc123"
  }
}

Callers discover agents by sending a standard micro service ping:

$SRV.PING.agents

Every running agent the caller has permission to see responds. For full endpoint information (subjects for prompts, capability metadata, queue groups), callers query:

$SRV.INFO.agents

The NATS blog emphasizes the architectural implication: "No registry. No service catalog. No coordinator process you have to keep alive." Discovery is decentralized. The NATS infrastructure is the registry. Agents come and go, and the set of reachable agents is whatever is currently connected and authorized. No separate Consul, etcd, or DNS-based discovery. No registration API. No heartbeat-to-registry. The connection is the registration.

This matters for agent systems specifically because agents are ephemeral in ways that microservices aren't. A microservice might run for months. An agent might be a single CLI invocation that lasts 30 seconds — a human asks a question, the agent processes it, the agent exits. Discovery that requires explicit registration and deregistration breaks down when agents are this short-lived. NATS discovery works because it's connection-scoped: when the agent connects, it's discoverable. When it disconnects, it's gone. No cleanup. No stale registrations.

Conversation

Prompting an agent is a single NATS request on a subject following the pattern agents.prompt.{agent}.{owner}.{name}:

nc.Request("agents.prompt.claude.engineering.sonnet", []byte("Summarize this document"), 60*time.Second)

The verb-first hierarchy (agents.prompt.>) means a single wildcard subscription captures all prompt traffic across all agents — useful for auditing, logging, or routing.

The agent streams typed JSON chunks back to the caller's reply inbox. Each chunk is {type, data}:

  • response chunks deliver content. The data is a string or an object with text and optional attachments. Multiple response chunks concatenate in publication order. This is native streaming — no SSE, no WebSocket upgrade, no chunked transfer encoding. The stream is the sequence of NATS messages on the reply subject.

  • status chunks carry lifecycle signals. A single ack status must be the very first message — before any latency-inducing work like model inference — so the caller knows the request was received and can reset its inactivity timer. Without this, the caller has no way to distinguish "the agent is thinking" from "the agent never got the message."

  • query chunks allow the agent to pause mid-stream and ask the caller a question. This is how an agent requests permission ("Can I read this file?"), clarification ("Which of these three documents?"), or a menu selection. Each query provides a fresh reply subject for the answer. Multiple queries can be in flight concurrently. The stream pauses on a query and resumes when the caller answers.

Every stream terminates with a zero-byte message carrying no headers. This is the protocol's way of saying "I'm done" without requiring the agent to predict how many chunks it will send. The terminator is unambiguous. The caller knows the conversation is complete.

Errors use NATS micro service error headers immediately before the terminator:

NATS-Service-Error: permission_denied
NATS-Service-Error-Code: 403

The error format is standard NATS. Any NATS tooling that understands micro service errors understands agent errors. No custom error format. No HTTP status codes mapped to application errors. The protocol leverages what NATS already provides.

Liveness

Each agent publishes a heartbeat on agents.hb.{agent}.{owner}.{name} at a configurable cadence (default 30 seconds). The payload carries agent identity and a per-instance instance_id:

{
  "agent": "claude",
  "owner": "engineering",
  "name": "sonnet",
  "instance_id": "a1b2c3d4",
  "timestamp": "2026-07-16T14:30:00Z",
  "cadence_seconds": 30
}

The heartbeat subject is fixed — the one subject agents cannot override. A caller subscribes to agents.hb.*.*.* and watches every agent on the cluster come up, stay up, and go offline without polling. An instance is considered offline after three missed beats (90 seconds at default cadence).

For point-in-time checks, every agent exposes a status request/reply endpoint returning the same payload shape as a heartbeat, freshly built on demand. This lets a caller verify an agent is alive without waiting for the next heartbeat.

The heartbeat pattern is simple but powerful. It's the same pattern NATS itself uses for route and gateway connections. It's the same pattern embedded in the micro services protocol. It works at scale because the NATS server handles the fan-out — agents.hb.*.*.* matches all agent heartbeats without the server doing per-agent work. The wildcard subscription is a single entry in the subscription table. The matching is a trie traversal.

Security: delegated, not reinvented

The agent protocol does not introduce its own security layer. It delegates entirely to NATS's existing primitives — accounts, users, and subject permissions. The blog is explicit: "End-to-end encryption and strong agent identity are explicitly deferred to future extensions."

This is the right call for v0.3. NATS already provides:

  • Authentication at connection time via JWT, nkey, or token
  • Authorization at publish/subscribe time via user JWT pub.allow and sub.allow
  • Isolation via accounts — agents in different accounts cannot see each other's subjects
  • Import/export for cross-account communication with explicit opt-in

A production deployment might look like: the engineering account owns internal agents (code exec, document search). The public account owns user-facing agents (ChatGPT plugin, Slack bot). The public account imports agents.prompt.> from engineering with a restricted set of allowed subjects. Engineering agents can't see public agents unless explicitly exported. The security boundary is the account. The policy is the import/export configuration. No API gateway. No mTLS sidecar per agent. No SPIFFE identity per process.

Bridging environments

The protocol deliberately avoids ties to any framework, language, or runtime. A caller doesn't need to know "whether an agent is a Python script, a hosted model wrapper, a CLI session, or a long-running service." They discover capabilities, send prompts, receive typed streams, and observe liveness the same way across all of them.

This is NATS's core strength applied to agents. NATS already bridges heterogeneous environments — Linux, Windows, macOS, ARM, x86, cloud, edge, on-prem. The agent protocol inherits this. An agent running on a Raspberry Pi on a factory floor can participate in the same agent mesh as a Claude instance in us-east-1. The subject namespace is global. The security is consistent. The liveness is observable. The protocol doesn't care about the runtime.

The blog notes that reference SDKs exist in TypeScript and Python, but "the NATS CLI can speak the protocol directly, and anything that knows the rules can participate." This is the litmus test for a protocol: can you implement it from the spec without a library? The agent protocol passes. The subject conventions are documented. The JSON envelope is documented. The chunk format is documented. The terminator is documented. You can write an agent in bash with nats CLI and jq if you want to. That's not a toy feature — it means the protocol is simple enough to be correct.

Why this matters for the agent ecosystem

The agent ecosystem in 2026 is fragmenting the same way the microservices ecosystem fragmented in 2016. Every framework has its own communication protocol. Every vendor has its own gateway. Interoperability is aspirational, not real. If you build an agent with LangChain and I build one with a custom Go service, they can't discover each other, can't exchange messages, can't observe each other's liveness — unless someone writes an adapter, a bridge, a translation layer.

The NATS agent protocol says: don't bridge. Use the same pipe. Connect to NATS. Follow the subject convention. Send the JSON chunks. Publish the heartbeat. If every agent does these four things, every agent can talk to every other agent — regardless of framework, language, cloud, or runtime. That's the promise of a protocol. That's the promise of dumb pipes and smart endpoints, applied to AI.


References:

NATS JetStream vs Kafka: Streaming Without the Partition Tax

Kafka's partition model is its superpower and its shackle — it gives you ordering guarantees at the cost of fixed parallelism and painful rebalancing. NATS JetStream takes the opposite approach: messages are stored in a single append-only stream, and consumers read independently with their own cursors. With Orbit.go's partitioned consumer groups, you get Kafka-style key-based ordering without Kafka's partition management overhead. This post explains the architectural difference and why it matters for teams that need streaming but don't want to operate Kafka.

natsjetstreamkafkastreamingpartitioned-consumersdistributed-systemsorbit

Kafka is the default answer for event streaming. It earned that position honestly — it solves a hard problem with a clear architecture. But operating Kafka is itself a hard problem. The partition is the unit of parallelism, and that design choice cascades into every operational concern: rebalancing, key distribution, retention, and scaling.

NATS JetStream approaches streaming from the opposite direction. Instead of partitioning the log for parallelism, it separates storage (the stream) from consumption (the consumer). Multiple consumers read the same stream independently, each at its own pace, with its own filter and acknowledgement state. Parallelism comes from concurrent consumers, not from partitions. Ordering comes from consumer configuration, not from partition assignment.

Orbit.go's partitioned consumer groups — a pure client-side library — bridge the remaining gap, bringing Kafka-style key-based ordering to JetStream without requiring stream partitioning. The result is streaming that's simpler to operate and more flexible to scale, with the ordering guarantees Kafka users expect.

The Kafka partition model

Kafka's fundamental abstraction is the partitioned log. Each topic is split into N partitions. Each partition is an ordered, immutable sequence of messages. Producers write to partitions based on a partitioning key. Consumers read from partitions in order. The partition is the unit of parallelism — you can have at most one consumer per partition (in the same consumer group) and at most as many active consumers as there are partitions.

This model provides strong guarantees:

  • Ordering per partition: Messages with the same key land in the same partition and are consumed in order. Ordering is guaranteed within a partition, not across partitions.
  • Parallelism via partition count: To increase throughput, increase the partition count. More partitions means more concurrent consumers.
  • Durability via replication: Each partition has a leader and N followers. The leader handles reads and writes. Followers replicate the log. If the leader fails, a follower takes over.

These guarantees come with operational constraints:

  • Fixed parallelism ceiling. Partition count is set at topic creation and is hard to change. Increasing partitions is possible but disruptive — it changes the key-to-partition mapping, breaking ordering for keys that move to new partitions. Decreasing partitions is effectively impossible without deleting and recreating the topic. You must provision for peak parallelism at topic creation, paying for capacity you may not need for months.

  • Rebalancing is expensive. When a consumer joins or leaves a consumer group, Kafka triggers a rebalance: partitions are reassigned across the new set of consumers. During rebalance, consumption pauses. For large consumer groups with many partitions, rebalancing can take seconds to minutes. The cooperative rebalance protocol (KIP-429) mitigates this but doesn't eliminate it. Every deployment, every scale-up, every crash triggers a rebalance.

  • Hot partitions are undivisible. If a single key generates disproportionate traffic (a "hot" partition), you can't split that partition without breaking ordering. The partition is atomic. You can't subdivide it. You can't parallelize within it. The partition is the bottleneck, and you can't make it wider.

  • Operational complexity. Kafka requires ZooKeeper (pre-3.3) or KRaft (3.3+). It requires careful JVM tuning. It requires disk provisioning for partitioned logs with configurable retention. It requires monitoring of under-replicated partitions, consumer lag, and broker resource utilization. Running Kafka well is a specialized skill. Running Kafka at scale is a team.

The JetStream model: one stream, many consumers

JetStream inverts the relationship between storage and consumption. A stream is a named append-only log that captures messages on one or more subjects. A consumer is a named view into a stream — a cursor with a filter, an acknowledgement policy, and a delivery mode.

// One stream captures all order events
js.AddStream(&nats.StreamConfig{
    Name:     "ORDERS",
    Subjects: []string{"orders.>"},
    Storage:  nats.FileStorage,
    Replicas: 3,
})

// Multiple consumers read independently
// Consumer A: fulfillment, reading from the beginning
js.AddConsumer("ORDERS", &nats.ConsumerConfig{
    Durable:       "fulfillment",
    FilterSubject: "orders.created",
    DeliverPolicy: nats.DeliverAllPolicy,
    AckPolicy:     nats.AckExplicitPolicy,
})

// Consumer B: analytics, only new messages
js.AddConsumer("ORDERS", &nats.ConsumerConfig{
    Durable:       "analytics",
    FilterSubject: "orders.>",
    DeliverPolicy: nats.DeliverNewPolicy,
})

// Consumer C: fraud detection, last per subject
js.AddConsumer("ORDERS", &nats.ConsumerConfig{
    Durable:       "fraud-check",
    FilterSubject: "orders.payment.*",
    DeliverPolicy: nats.DeliverLastPerSubjectPolicy,
})

Notice what's missing: no partition count. No key-to-partition mapping. No partition assignment. The stream is a single log. Consumers are independent views. Each consumer has its own cursor. Each consumer acknowledges independently. The stream doesn't care how many consumers exist or what they're doing — it just appends messages.

Parallelism works differently in this model. A single consumer can be read by multiple instances concurrently. NATS distributes messages across instances that are pulling from the same consumer — no rebalancing, no partition assignment, no pause in consumption when an instance joins or leaves:

// Instance 1, 2, and 3 all pull from the same consumer
sub, _ := js.PullSubscribe("orders.created", "fulfillment")
for {
    msgs, _ := sub.Fetch(10)
    for _, msg := range msgs {
        process(msg)
        msg.Ack()
    }
}

The three instances collectively consume from fulfillment. Each Fetch(10) returns up to 10 messages. NATS distributes messages across the active pullers. If Instance 1 crashes, Instances 2 and 3 continue — their pull requests are now served faster because there are fewer pullers competing. No rebalance. No partition reassignment. No pause.

This is elastic by default. You deploy more instances, they start pulling, throughput increases. You deploy fewer, throughput decreases. No topic configuration to update. No partition count to pre-provision. The parallelism is dynamic, not static.

The ordering trade-off

JetStream's elastic model has a trade-off: ordering is not guaranteed across concurrent pulls. If Instance 1 pulls messages 1-10 and Instance 2 pulls messages 11-20 simultaneously, and Instance 2 processes faster, message 11 might be acknowledged before message 1. Messages are stored in order. They may be processed out of order.

For many workloads, this is fine. If each message is independent — a notification, a metric, a log entry — processing order doesn't matter. But for workloads that require per-key ordering (all events for Customer A processed in order), it's a real constraint.

The traditional solution is to set MaxAckPending to 1, which serializes all processing through a single message in flight at a time. This guarantees order but kills throughput. You've traded parallelism for ordering, and you're paying for it on every message, even ones with different keys that could safely be processed in parallel.

This is exactly the gap that Orbit.go's partitioned consumer groups fill.

Partitioned consumer groups: Kafka semantics, JetStream simplicity

Orbit.go implements what Jean-Noël Moyne describes as "functionally equivalent to what Apache Kafka calls 'consumer groups' and how they implement partitioning" — entirely on the client side.

The key insight: most real-world ordering requirements are per-key, not global. You need Customer A's events processed in order. You don't need all customers' events processed in order. Kafka enforces this via partitions — same key → same partition → same consumer → ordered processing. Orbit.go enforces this via subject token hashing on top of JetStream consumers — same key → same member → ordered processing.

Static partitioned consumer groups

Static groups require the stream to have a partition number as the first subject token (achievable via a stream subject transform at ingest). The library maps member names to partition numbers using consistent hashing:

// Stream subjects include partition number
// orders.{partition}.created, orders.{partition}.paid, ...

group := orbit.CreateStaticGroup("order-processors", streamConfig, memberNames)
group.Join("fulfillment-member", consumerConfig, func(msg jetstream.Msg) {
    // Messages for this member's partitions arrive in order
    processInOrder(msg)
    msg.Ack()
})

Guarantees:

  • Each partition is handled by exactly one member at a time
  • Messages within a partition are processed in order
  • Multiple partitions can be processed in parallel by different members
  • If a member instance crashes, NATS 2.11's pinned consumer feature ensures the replacement instance picks up where the old one left off

Static groups are faster and use fewer server resources, but membership is fixed at creation. No adding members at runtime. The trade-off is latency and resource efficiency for operational flexibility.

Elastic partitioned consumer groups

Elastic groups work on any existing stream — no partition token required. The library materializes the group as a work queue stream that sources from the original, inserting partition numbers during sourcing:

group := orbit.CreateElasticGroup("order-processors", sourceStream, maxMembers)
group.AddMember("fulfillment")   // Add at runtime
group.AddMember("audit")         // Add at runtime
group.DropMember("fulfillment")  // Remove at runtime

group.Join("fulfillment", consumerConfig, func(msg jetstream.Msg) {
    processInOrder(msg)
    msg.Ack()
})

The work queue stream holds copies of messages, so consumption lag can be monitored by checking the work queue stream size. You can cap the work queue stream size; if it hits the limit, sourcing pauses briefly. This prevents unbounded memory consumption from a work queue that outpaces its consumers.

Elastic groups use more server resources and add slight latency (the materialization step), but you get runtime elasticity. Add members when load increases. Drop members when load decreases. No partition reassignment. No key-to-partition remapping. No rebalancing pause.

Side by side: Kafka vs JetStream with partitioned consumers

Concern Kafka JetStream + Orbit.go
Unit of storage Partitioned topic Single stream
Unit of parallelism Partition (fixed at creation) Consumer instances (dynamic)
Ordering Per partition Per consumer (or per member in a group)
Adding parallelism Increase partitions (disruptive) Deploy more instances (elastic)
Rebalancing Pauses consumption No pause (pull-based distribution)
Hot partition Undivisible bottleneck Elastic member can be dedicated
Retention Time or size per partition Time, size, or count per stream
Replication Per partition (leader/follower) Per stream (Raft across cluster nodes)
Operational complexity ZooKeeper/KRaft, JVM tuning, partition monitoring Single Go binary, streams and consumers
Key-based ordering Built-in (partition by key) Via Orbit.go (hash by subject token)

The fundamental difference: Kafka builds ordering into the storage layer (partitions). JetStream builds ordering into the consumption layer (consumers and consumer groups). The JetStream approach is more flexible because consumers can be reconfigured without touching the stream. The Kafka approach provides stronger guarantees because the ordering is physically enforced by the log structure. Whether one is better depends on whether you value operational flexibility or storage-level guarantees more.

When Kafka still makes sense

Kafka's partition model is not a mistake. It's the right design for workloads where:

Ordering is truly global. If every message in a topic must be processed in exact append order (not just per-key order), Kafka partitions give you that — one partition, one consumer. JetStream can do this with MaxAckPending: 1 but at the same throughput cost.

You need the Kafka ecosystem. Kafka Connect provides a rich set of source and sink connectors. Kafka Streams provides a sophisticated stream processing library with exactly-once semantics, stateful operations (joins, aggregations, windows), and an interactive query API. ksqlDB provides SQL over streams. The ecosystem is deep and mature. If your architecture depends on these tools, Kafka is the right choice.

Your team already operates Kafka well. Running Kafka is a skill. If your team has invested in that skill and the operational burden is manageable, the migration cost to NATS may not be justified. Don't migrate because NATS is simpler. Migrate because the operational burden of Kafka is a meaningful drag on your team's velocity.

You need compacted topics. Kafka's log compaction retains the latest value for each key, enabling table-like semantics for changelogs. JetStream doesn't have a direct equivalent (though DiscardNewPerSubject with MaxMsgsPerSubject: 1 provides a rough approximation for single-message-per-subject use cases).

For greenfield systems that need streaming but don't already depend on the Kafka ecosystem, JetStream with Orbit.go partitioned consumer groups addresses the same ordering requirements with significantly less operational overhead. You get key-based ordering. You get parallel consumption. You don't get partition management, rebalance pauses, or ZooKeeper. For most teams, that's a good trade.


References:

NATS vs RabbitMQ: Subject-Based Routing Eliminates Topology Complexity

RabbitMQ is the most widely deployed message broker, but its exchange/queue/binding topology becomes a maintenance burden at scale. NATS replaces this three-layer routing model with flat subject strings — and the operational difference is dramatic. Sophotech cut p99 latency from ~150ms to ~40ms and ops time from several hours a week to under one by migrating 50 services from RabbitMQ to NATS. This post explains why subject-based routing is not just simpler syntax — it's a fundamentally different (and cheaper) model for message routing.

natsrabbitmqmessagingpubsubmicroservicesmigrationtopology

RabbitMQ is everywhere. It ships with apt-get. It runs in every cloud. It's the default answer when someone says "we need a message queue." And for many teams, it works — until it doesn't.

The Sophotech case study on the NATS blog is a clean before-and-after. A single Kubernetes cluster running roughly 50 microservices on RabbitMQ. Three messaging patterns: task queues, pub/sub events, and service-to-service RPC. All standard. All within RabbitMQ's design envelope. And yet the team was spending "several hours a week" on operations, hitting p99 latencies of ~150ms, and dealing with queue backlogs that reached minutes under burst load.

After migrating to NATS: p99 dropped to ~40ms (a 3.75x improvement). Ops time fell to under an hour per week. Burst backlogs that previously caused minutes of lag were processed within seconds.

What changed? The topology disappeared.

The RabbitMQ topology problem

In RabbitMQ, sending a message from Service A to Service B requires three layers of configuration:

  1. Exchange — the routing target. Is it direct, topic, fanout, or headers? Each type routes differently. Each has its own configuration schema. The exchange type is a design decision that propagates to every publisher and consumer.

  2. Queue — the storage. Is it durable, exclusive, or auto-delete? What's the TTL? The max length? The dead-letter exchange? The queue is where messages wait, and its configuration determines reliability, performance, and behavior under backpressure. Each queue must be declared before use.

  3. Binding — the connection between exchange and queue. The binding key pattern determines which messages from the exchange land in which queue. A topic exchange with binding key order.# routes all messages with routing keys starting with order. to the bound queue. Change the binding key and you change what the consumer receives — without touching the consumer's code.

For a single pub/sub pattern, you need: an exchange declaration, a queue declaration, and a binding declaration. For 50 services with an average of 3 message types each, you have at minimum 150 exchanges, 150 queues, and 150 bindings. In practice it's more — queues get sharded for parallelism, dead-letter exchanges get created for error handling, mirroring gets configured for HA.

This topology is not visible in your application code. It lives in RabbitMQ's configuration, in deployment scripts, in infrastructure-as-code, and in tribal knowledge. When a new developer joins and asks "how does the order service get order events?", the answer involves three different UI screens or CLI commands. The routing logic is distributed across the application (which declares the routing key) and the infrastructure (which configures bindings). Neither side fully owns the behavior.

RabbitMQ's own clustering compounds this. For high availability, you need mirrored queues or quorum queues. Mirrored queues replicate every message to every mirror — reliable but expensive. Quorum queues use Raft — more efficient but sensitive to network partitions and requiring careful tuning. Add shovels for cross-datacenter and federations for cross-region, and the topology graph becomes a full-time job.

The NATS alternative: subjects are the topology

NATS eliminates all of this. There are no exchanges. No queues. No bindings. There are only subjects and subscriptions.

A subject is a string: orders.created, payment.processed, inventory.updated.us-east. It is a hierarchical, dot-separated token sequence. A subscription matches subjects with wildcards: orders.* matches one additional token, orders.> matches any number of additional tokens.

That's the entire routing model. Here is the equivalent of a fanout exchange with multiple bound queues in NATS:

// Service A publishes
nc.Publish("orders.created", eventData)

// Service B subscribes — no exchange, queue, or binding required
nc.Subscribe("orders.created", func(m *nats.Msg) {
    processEvent(m)
})

// Service C also subscribes — independently
nc.Subscribe("orders.created", func(m *nats.Msg) {
    auditEvent(m)
})

Every subscriber on orders.created receives the message. No exchange to declare. No queue to configure. No binding to maintain. The subject is the routing. The subscription is the delivery. The code is the topology.

This inversion eliminates the infrastructure drift that plagues RabbitMQ deployments. In RabbitMQ, the exchange, queue, and binding must exist before the publisher publishes. If the queue declaration is removed from the infrastructure config but the publisher still publishes to the exchange, messages black-hole silently (or end up in an alternate exchange, if configured). In NATS, if nobody is subscribed to orders.created, the message is simply not delivered — and that's correct behavior. When a subscriber subscribes, it starts receiving from that point forward. No missing topology. No silent message loss. No drift between infrastructure and application.

Migration in three phases

The Sophotech team migrated progressively, which is worth studying as a pattern:

Phase 1: Dual publishing. Every service that published to RabbitMQ was modified to also publish to NATS. The RabbitMQ path remained primary. The NATS path was fire-and-forget — if publishing to NATS failed, the service logged a warning and continued. This phase established that the NATS subject namespace worked correctly without risking production traffic.

Phase 2: Canary consumers. Select services began consuming from NATS instead of RabbitMQ. The RabbitMQ consumer ran alongside — both processed the same logical messages, and their outputs were compared. If the NATS consumer produced different results, the canary was rolled back. If it produced identical results for a sustained period, confidence increased.

Phase 3: Full cutover. Once all consumers had been canaried and verified, dual publishing was removed. Services published only to NATS. Services consumed only from NATS. RabbitMQ was decommissioned.

The key to this working is that the migration was at the messaging layer, not the application layer. Services didn't change their business logic. They changed the transport — from amqp.Dial to nats.Connect, from exchange declaration to subject subscription. The business logic stayed the same. The transport got simpler.

The topology comparison, side by side

Here's what a simple pub/sub workflow looks like in both systems.

RabbitMQ — fanout of an order event to three consumers:

# Publisher
channel.exchange_declare(exchange='orders', exchange_type='topic')
channel.basic_publish(exchange='orders', routing_key='order.created', body=json.dumps(event))

# Consumer A: fulfillment
channel.queue_declare(queue='fulfillment_orders')
channel.queue_bind(exchange='orders', queue='fulfillment_orders', routing_key='order.created')
channel.basic_consume(queue='fulfillment_orders', on_message_callback=process)

# Consumer B: notification
channel.queue_declare(queue='notification_orders')
channel.queue_bind(exchange='orders', queue='notification_orders', routing_key='order.*')
channel.basic_consume(queue='notification_orders', on_message_callback=send_email)

# Consumer C: analytics
channel.queue_declare(queue='analytics_orders', durable=True)
channel.queue_bind(exchange='orders', queue='analytics_orders', routing_key='order.#')
channel.basic_consume(queue='analytics_orders', on_message_callback=track)

That's 9 infrastructure declarations for one event type across three consumers. Add a fourth consumer, and you add two more declarations. The infrastructure grows linearly with the number of consumers. Every queue is a named resource that must be managed, monitored, and cleaned up if the consumer is decommissioned.

NATS — same workflow:

// Publisher
nc.Publish("order.created", event)

// Consumer A: fulfillment
nc.Subscribe("order.created", process)

// Consumer B: notification
nc.Subscribe("order.*", sendEmail)

// Consumer C: analytics
nc.Subscribe("order.>", track)

Four lines of application code. No infrastructure declarations. No named resources to manage. The routing is implicit in the subjects. If Consumer B goes away, nothing needs to be cleaned up — the subscription disappears when the connection closes. If Consumer D joins, it subscribes to the subjects it cares about. No configuration changes. No topology updates.

This is not just syntactic convenience. It's a fundamentally different ownership model. In RabbitMQ, the infrastructure owns the topology. In NATS, the application owns the routing. The infrastructure team doesn't need to know which services subscribe to which subjects. The subject namespace is self-documenting — order.created means what it says. Routing changes are code changes, not config changes. They go through the same review, test, and deploy pipeline as any other application change.

Queue groups: scale without configuration

RabbitMQ's primary scaling mechanism is the competing consumers pattern: multiple consumers on the same queue, with RabbitMQ distributing messages round-robin. This requires declaring the queue as shared and ensuring all consumers connect with the same queue name and configuration.

NATS provides the same pattern through queue groups — with even less ceremony:

// Three instances of the same service, one queue group
nc.QueueSubscribe("orders.process", "order-workers", func(m *nats.Msg) {
    processOrder(m)
})

The queue group name (order-workers) is the only shared knowledge. No queue declaration. No binding. No durable/exclusive/auto-delete decision. NATS distributes messages across connected queue group members. When an instance disconnects (crashes, scales down, deploys), NATS removes it from the distribution. When a new instance connects, NATS adds it.

There is no load balancer. No health check endpoint. No instance registry. The NATS server knows which clients are connected to which subjects. The protocol handles distribution. The application code is identical whether there's one instance or a hundred.

Persistence without the ceremony

RabbitMQ queues are persistent by configuration. You declare durable=True and messages are written to disk. If the broker restarts, durable queues and persistent messages survive. Non-durable queues and transient messages don't. The configuration determines the guarantee.

NATS separates persistence into JetStream, which is opt-in. You create a stream to capture subjects, and consumers read from the stream:

// Create a stream that captures order subjects
js.AddStream(&nats.StreamConfig{
    Name:     "ORDERS",
    Subjects: []string{"order.>"},
    Storage:  nats.FileStorage,
})

// Publish to the subject — persisted automatically
js.Publish("order.created", event)

// Create a consumer to read persisted messages
js.AddConsumer("ORDERS", &nats.ConsumerConfig{
    Durable:       "order-processor",
    FilterSubject: "order.created",
    AckPolicy:     nats.AckExplicitPolicy,
})

This separation — streams for storage, consumers for reading — is the key architectural difference. In RabbitMQ, you get persistence by configuring a queue as durable. In NATS, you get persistence by creating a stream. The stream captures messages on subjects. Consumers are independent views into the stream, each with their own cursor, filter, and acknowledgement state. Multiple consumers can read the same stream at different paces, with different filters, starting from different positions. One stream. Many consumers. No queue per consumer.

This pattern eliminates RabbitMQ's fanout tax. To get a message to three services in RabbitMQ with persistence, you need a topic exchange, three durable queues, and three bindings. In NATS, you need one stream and three consumers. The stream captures once. The consumers read independently. The infrastructure is proportional to the number of message categories, not the number of consumers — which is usually the smaller number.

What the numbers say

Sophotech's 3.75x latency improvement isn't magic. It's the combination of:

  • No exchange routing overhead. A NATS subject match is a trie lookup against a subscription table. RabbitMQ's topic exchange is a state machine matching the routing key against all binding patterns. For a routing key with 4 tokens and 100 binding patterns, the NATS match is O(4) — walk the trie 4 levels. The RabbitMQ match is O(100 × 4) — test each binding pattern against the 4-token key.

  • No persistence double-write. RabbitMQ durable queues write to disk. For mirrored queues, the write is replicated. For quorum queues, the write goes through Raft. NATS JetStream writes to file storage with optional Raft replication. But here's the difference: RabbitMQ writes the message to the queue's storage and the queue's index. JetStream writes to an append-only log. Append-only is faster. No index update. No queue-level bookkeeping per consumer.

  • No queue-level congestion. In RabbitMQ, a slow consumer creates a queue backlog. The queue grows. Other consumers on different queues are unaffected by the backlog, but they're affected by the broker's overall resource pressure (memory, file descriptors, disk I/O) caused by the growing queue. In NATS, a slow consumer creates consumer-level lag. The stream keeps appending. Other consumers keep consuming. The slow consumer's lag doesn't affect other consumers' throughput. The stream is shared. The consumption is isolated.

The operational improvement — from several hours a week to under one — comes from what was removed. No queue mirroring to configure. No dead-letter exchanges to maintain. No federation links to troubleshoot. No shovel to restart. The infrastructure surface area shrinks because the infrastructure model is simpler.

When RabbitMQ still makes sense

RabbitMQ is not universally wrong. If you already run it and it works at your scale, the migration cost may not be worth the latency improvement. If your messaging volume is low (hundreds per second, not thousands or millions), the latency difference will be invisible. If you use AMQP 1.0 features that NATS doesn't replicate (message annotations, complex routing headers, fine-grained delivery annotations), the protocol matters more than the performance.

And perhaps most importantly: if your team deeply understands RabbitMQ operations — if you've invested years in tuning, monitoring, and troubleshooting RabbitMQ — that operational knowledge is real. Switching to NATS means rebuilding that knowledge. The operational improvement may be worth it, but it's not free.

For greenfield systems, however, the question is not "NATS or RabbitMQ?" It's "why would you choose the system with more moving parts?" RabbitMQ requires you to design, implement, and maintain a routing topology. NATS requires you to define a subject namespace. One is infrastructure. The other is naming. Naming is easier than infrastructure. Naming is easier to change. Naming is easier to debug. When the simpler system is also faster and more operationally efficient, the burden of proof shifts to the more complex one.


References:

Natstroll: A NATS JWT+JetStream Capability Test with Ollama

Natstroll is a small but surprisingly dense NATS capability test — a hub-and-spoke joke exchange that exercises embedded NATS servers, JWT operator/account/user authentication, dynamic credential issuance, JetStream streams and durable pull consumers, scoped subject permissions, request/reply patterns, and OpenTelemetry trace propagation, all wrapped around an Ollama-powered AI conversation loop. This post walks through the architecture in detail, explaining what each component exercises and why the design decisions matter for anyone building NATS-based distributed systems.

natsjetstreamjwtollamagodistributed-systemsmessaging

Natstroll is one of those projects that looks like a toy on the surface — a hub and a spoke telling each other AI-generated jokes — but turns out to exercise a remarkably complete slice of the NATS ecosystem. Embedded servers, JWT authentication chains, dynamic user credentials, JetStream streams, durable pull consumers, scoped subject permissions, request/reply patterns, heartbeat monitoring, and distributed tracing all show up in roughly 800 lines of Go.

I built it as a lab. Not a production system, not a framework — a capability test. The kind of thing you write when you want to verify that all the pieces actually work together before reaching for them in something that matters.

The shape of the thing

Natstroll has two binaries: a hub and a spoke. The hub owns an embedded NATS server. Spokes connect to it, register themselves, receive dynamically issued JWT credentials scoped to their identity, and then participate in a joke exchange loop — the hub sends an Ollama-generated joke, the spoke generates a comeback, the hub fires back with a follow-up, and so on.

┌──────────────────────────────────────────────────┐
│                      Hub                          │
│  ┌──────────────┐  ┌──────────┐  ┌────────────┐  │
│  │ Embedded     │  │ JWT      │  │ Ollama     │  │
│  │ NATS Server  │  │ Issuer   │  │ Client     │  │
│  │ (JetStream)  │  │          │  │            │  │
│  └──────┬───────┘  └────┬─────┘  └─────┬──────┘  │
│         │               │               │         │
└─────────┼───────────────┼───────────────┼─────────┘
          │               │               │
    ┌─────▼─────┐   ┌─────▼─────┐   ┌─────▼─────┐
    │ NATS      │   │ Dynamic   │   │ Ollama    │
    │ Messages  │   │ Creds     │   │ Replies   │
    └─────┬─────┘   └─────┬─────┘   └─────┬─────┘
          │               │               │
┌─────────┼───────────────┼───────────────┼─────────┐
│         │               │               │         │
│  ┌──────▼───────┐  ┌────▼─────┐  ┌─────▼──────┐  │
│  │ JetStream    │  │ JWT      │  │ Ollama     │  │
│  │ Consumer     │  │ Auth     │  │ Client     │  │
│  └──────────────┘  └──────────┘  └────────────┘  │
│                      Spoke                         │
└───────────────────────────────────────────────────┘

The architecture is deliberately clean. The hub provisions infrastructure — the server, the trust chain, the stream. The spoke provisions its own consumer. That split is the point: it tests whether dynamic credentials can carry enough permissions for a client to manage JetStream resources on its own behalf.

What this actually exercises

Most NATS tutorials stop at nats.Connect() with a token or a static creds file. That's fine for getting started, but distributed systems have a way of surfacing edge cases the moment you step off the happy path. Natstroll deliberately walks into those edge cases.

1. Embedded NATS server

The hub starts a nats-server process in-process. This is not a mock — it's the real server, with a real config file written to a temp directory, a real JetStream store, and real JWT resolution. The code handles the full lifecycle:

  • Generate a one-shot operator key and system account key
  • Write operator and account JWTs to a resolver directory
  • Configure JetStream with memory and file storage limits
  • Wait for the server to accept connections before proceeding
  • Clean up the temp directory on shutdown
ns, err := server.NewServer(opts)
go ns.Start()
if !ns.ReadyForConnections(5 * time.Second) {
    ns.Shutdown()
    return nil, nil, fmt.Errorf("NATS server not ready")
}

This alone catches real issues: port conflicts, JetStream initialization races, the fact that ReadyForConnections returning true does not mean JetStream's account info endpoint is answering yet (hence the explicit waitForJetStream retry loop in the hub).

2. JWT trust chain (operator → account → user)

Natstroll uses full JWT/operator mode. The trust chain has three tiers:

  • Operator JWT: signs account JWTs, names the system account
  • Account JWT: enables JetStream with unlimited quotas (this is a lab), acts as the signer for user JWTs
  • User JWTs: per-spoke or per-hub, carry scoped subject permissions

The hub generates the operator and account on startup, places the account JWT in a resolver directory, and issues user JWTs dynamically. Nothing is pre-provisioned.

This is the right model for multi-tenant NATS deployments. The operator is a trust anchor. The account is a security boundary. Users are scoped within an account. The system account exists to make the resolver work — it's not used by the application, but the embedded server won't start without one when you're in full resolver mode.

The test exercises the resolution path end-to-end: the server loads the operator JWT from a file, finds account JWTs in the resolver directory, and validates user JWTs presented in creds files. If any link in that chain breaks — wrong system account key, missing account JWT, expired user claims — the connection fails with a clear auth error.

3. Dynamic credential issuance

This is the most interesting part. Most NATS tutorials use static creds files generated once and distributed manually. Natstroll has the hub generate user credentials on the fly when a spoke registers.

The spoke connects with narrow registrar credentials:

claims.Pub.Allow = []string{"reg.request"}
claims.Sub.Allow = []string{"_INBOX.>"}

These can only publish a registration request and receive on inbox subjects used by NATS request/reply. If these credentials leak, the attacker can register — but they can't publish to joke subjects, read the stream, or do anything else.

When a spoke sends reg.request with its SPOKE_ID, the hub generates a new nkey pair, creates a user JWT with subject permissions scoped to that specific spoke identity, and returns the full creds file. The spoke then drops the registrar connection, reconnects with the new dynamic credentials, and proves they work by creating a JetStream consumer.

The spoke's dynamic permissions are identity-scoped:

claims.Pub.Allow = []string{
    "heartbeat." + id,                          // can publish its own heartbeat
    shared.JokeResponseSubject + id + ".>",      // can publish joke responses
    "$JS.API.>",                                 // can manage JetStream
    "$JS.ACK." + shared.JokeStream + "." + consumerName + ".>",
}
claims.Sub.Allow = []string{
    "_INBOX.>",
    shared.JokeRequestSubject + id,             // only its own joke requests
}

Spoke A cannot subscribe to Spoke B's joke requests. It cannot publish heartbeats as Spoke B. It cannot publish responses on Spoke B's subjects. The NATS server enforces this at the protocol level — the spoke literally cannot express a message on the wrong subject.

4. JetStream stream and consumer separation

The hub creates the stream. The spoke creates its own consumer. This split is intentional.

Hub side: creates JOKE_STREAM covering both request and response subjects:

cfg := &nats.StreamConfig{
    Name:     shared.JokeStream,
    Subjects: []string{shared.JokeRequestSubject + ">", shared.JokeResponseSubject + ">"},
    Storage:  nats.FileStorage,
}

Spoke side: creates a durable pull consumer filtered to its own request subject:

cfg := &nats.ConsumerConfig{
    Durable:       consumerName,
    AckPolicy:     nats.AckExplicitPolicy,
    FilterSubject: filterSubject,  // joke.request.<spokeID>
}

This pattern matters for real systems. The party that owns the data (the hub, or an ops team) provisions the stream. The party that processes the data (a spoke, or a microservice) provisions its own consumer with the exact filter and ack policy it needs. Dynamic credentials must carry $JS.API.> to make this possible — and that's the capability under test.

The spoke also handles consumer creation idempotently. If the consumer already exists with the right filter, it moves on. If it exists with the wrong filter (from a previous run with a different config), it errors out rather than silently misbehaving. This is the kind of detail that separates a demo from something you'd actually run.

5. Request/reply with reply subject scoping

The hub's conversation loop uses a pattern worth studying:

replySubject := shared.JokeResponseSubject + targetSpokeID + "." + requestID
sub, err := nc.SubscribeSync(replySubject)
// ... publish joke request with Reply set to replySubject ...
msg := &nats.Msg{
    Subject: shared.JokeRequestSubject + targetSpokeID,
    Reply:   replySubject,
    Data:    reqData,
}
js.PublishMsg(msg)
replyMsg, err := sub.NextMsg(SpokeTimeout)

The hub publishes the joke request via JetStream (so it's persisted) but waits for the reply through a core NATS subscription on a dynamically generated reply subject. The reply subject is scoped to the spoke's identity and includes a UUID request ID — so the spoke's JWT only needs joke.response.<spokeID>.> permissions, and reply routing is naturally collision-free.

The timeout is generous (75 seconds) because the spoke's Ollama model gets 60 seconds for generation, and network latency plus JSON marshaling adds a bit more. If the spoke doesn't reply in time, the hub logs the failure and the conversation loop ends. No hung goroutines, no leaked subscriptions — the sub.Unsubscribe() in the defer-like pattern at each iteration cleans up.

6. Heartbeat monitoring

Each spoke publishes a heartbeat every 10 seconds on heartbeat.<spokeID>:

nc.Publish("heartbeat."+spokeID, []byte(`{"status":"alive"}`))

The hub subscribes to heartbeat.> and logs every heartbeat it sees. This is a simple liveness pattern, but it verifies that the spoke's dynamic credentials actually allow publishing to its scoped heartbeat subject, and that the hub's wildcard subscription works across all registered spokes.

7. OpenTelemetry trace propagation

Both binaries support opt-in OTLP trace export. When OTEL_EXPORTER_OTLP_ENDPOINT is set, the hub and spoke initialize a gRPC trace exporter with proper resource attributes (service.name, service.version, host.name).

Trace context is propagated through NATS message headers:

func InjectTraceContext(ctx context.Context, msg *nats.Msg) {
    otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(msg.Header))
}

func ExtractTraceContext(msg *nats.Msg) context.Context {
    return otel.GetTextMapPropagator().Extract(context.Background(), propagation.HeaderCarrier(msg.Header))
}

This means you can follow a joke request from the hub's hub.generate-joke span through the NATS transport to the spoke's spoke.generate-reply span, and back — a full distributed trace across processes connected only by a NATS cluster. In production, this is what lets you debug latency in message-driven systems without stitching together log timestamps by hand.

When the endpoint is unset, the tracer falls back to the global no-op tracer — no spans are exported, no log noise, no dependency on a running collector. Opt-in with a clean fallback is the right default for a lab.

The Ollama integration

Natstroll defaults to deepseek-r1:1.5b, a thinking model that spends tokens on internal chain-of-thought before producing a final answer. This is deliberate — thinking models stress-test the timeout and fallback paths.

The prompt prefix "Return only the final answer. Do not think out loud." suppresses most reasoning output, but if num_predict is too low, the model may spend all its tokens on hidden reasoning and return an empty string. The code handles this:

  • The hub checks for empty Ollama responses and returns a descriptive error instead of publishing a blank joke
  • The spoke falls back to a deterministic reply if Ollama fails: "That joke took off about as well as a cat's paper airplane."
  • Both sides use a 60-second timeout with num_predict: 256 — generous enough for a short joke, tight enough that a hung model doesn't wedge the system

These aren't just Ollama details. They're resilience patterns: fail loud, fail fast, provide a degraded response rather than crashing.

The credential bootstrap flow

The first-run experience deserves attention because it gets the security model right from the start.

When you run the hub without NATS_ACCOUNT_SEED set, it generates a fresh account key, creates narrow-scoped registrar credentials, prints them, and exits:

========== COPY THESE EXACTLY ==========
export NATS_ACCOUNT_SEED="SAAPG3R4G..."
export REGISTRAR_CREDS_B64="W0Zvcm..."
========================================

You paste those into your terminal and run the hub again — this time it starts the server. The registrar credentials can only publish reg.request and subscribe to _INBOX.>. They can't read the joke stream, can't publish heartbeats, can't impersonate a spoke. If they leak, the blast radius is contained: an attacker can register, but registration is the only thing those credentials authorize.

Contrast this with most "getting started" setups that hand you a creds file with pub: [">"] and sub: [">"] and tell you to get to production first. Natstroll starts with principle-of-least-privilege from the first go run.

Security: what's intentionally loose and why

The spoke's dynamic credentials include $JS.API.> — broad JetStream API access. This is the capability under test: can a dynamically issued credential create and bind a durable pull consumer? The answer is yes, and proving it requires those permissions.

The README is explicit about what you'd change for production:

  • Have the hub create consumers instead of the spoke
  • Scope JetStream API permissions to exact subjects per consumer
  • Don't write debug credentials to /tmp
  • Add credential rotation
  • Add per-spoke quotas

There is a difference between a lab being insecure and a lab documenting where the security boundaries are and why they're drawn where they are. Natstroll does the second thing. The subjects that exist, the permissions that are granted, and the reasons for each grant are all visible in roughly 20 lines of issueSpokeCredentials.

Why this matters as a demo

Most distributed-systems demos pick one thing and show it in isolation. A JWT tutorial shows a static creds file. A JetStream tutorial shows a stream and consumer created by the same client with admin credentials. An OpenTelemetry tutorial shows traces between HTTP services.

Natstroll compresses all of these into a single coherent flow. The JWT issuance feeds into the JetStream consumer creation. The request/reply pattern carries trace context. The heartbeat loop verifies that scoped subjects work. The credential upgrade (registrar → dynamic) proves that reconnection with new credentials is seamless.

It's also honest about its constraints. It uses a single account, so the account is both the issuer and the security boundary — production multi-tenancy would split those. It grants broad JetStream API access to spokes for testing purposes. It doesn't persist state between restarts. The conversation loop only involves the first registered spoke.

These are not bugs. They are scope decisions that make the lab small enough to understand in one sitting but complete enough to surface real distributed-systems concerns.

Running it

git clone https://github.com/moresearch/natstroll
cd natstroll
ollama pull deepseek-r1:1.5b

# Terminal 1: bootstrap and start the hub
unset NATS_ACCOUNT_SEED REGISTRAR_CREDS_B64
go run ./cmd/hub
# → copy the export lines
export NATS_ACCOUNT_SEED="..."
export REGISTRAR_CREDS_B64="..."
go run ./cmd/hub

# Terminal 2: start a spoke
export NATS_URL=nats://127.0.0.1:4222
export REGISTRAR_CREDS_B64="..."
export SPOKE_ID=black-spoke
go run ./cmd/spoke

Add more spokes with different SPOKE_ID values. Spy on traffic with nats CLI tools and the debug credentials file. Enable OpenTelemetry by setting OTEL_EXPORTER_OTLP_ENDPOINT.

The Makefile produces cross-compiled, stripped binaries for Linux and Windows on amd64 and arm64 — useful if you want to run spokes on different machines or architectures.

What I'd reach for it for

Natstroll is not a library. It's not a framework. It's a reference: here is a known-good configuration for an embedded NATS server in JWT mode, here is how dynamic credential issuance works, here is how a pull consumer on JetStream is created by a dynamically authenticated client, here is how trace context flows through NATS headers.

When I'm building something that needs any of these pieces, I'd rather start from something that exercises all of them together than from five separate tutorials that may or may not compose. That's the value of a capability test: it proves the integration works before you commit to it.

S&R: How Should You Architect a System That Handles Both Search and Recommendation?

Practical guidance for building systems that handle both search and recommendation: real-life analogies, error cost analysis, five architectural principles, and a reference hybrid architecture sketch.

searchrecommendationarchitecturesystem-designllmpracticalseries

S&R stands for Search & Recommendation. This post is about building systems that handle both — correctly. The distinction is not academic. It has consequences for your latency budget, your index design, your evaluation framework, and your error recovery strategy. Ignore it and you will build a system that is mediocre at both.

Search demands sub-100ms latency per keystroke, an inverted index that can retrieve on any term, and an error model where mistakes are visible and dismissable. The user sees a wrong result, scrolls past it, and reforms their query. The system's job is to make the right result findable. Getting search right means the user finds what they asked for. Getting it wrong means they try another query — or another product.

Recommendation tolerates ~200ms per page load, an ANN index that finds neighbors in embedding space, and an error model where mistakes are invisible but trust-eroding. The user doesn't see what wasn't shown. They just feel, over weeks, that the system doesn't get them — and they churn. The system's job is to make the right result unavoidable. Getting recommendation right means the user discovers something they didn't know they wanted. Getting it wrong means they leave and never come back. Same infrastructure can serve both. The architecture must know the difference.

This series has traced the fifty-year history of search, the thirty-year history of recommendation, Netflix's dual-stack architecture, how four major companies handle the boundary, and how LLMs transform both fields. This post translates that history into practical guidance.

The Distinction in Everyday Life

Before the architecture, a mental model. The search–recommendation distinction isn't an engineering abstraction — it's visible in how we make decisions every day.

The Restaurant Menu (Search) vs. The Chef's Tasting Menu (Recommendation). When you scan a menu, you're doing search: you have an intent ("I feel like pasta") and you're scanning a structured catalog. Your satisfaction depends on whether the menu accurately represents what the kitchen delivers. A mistake — "I ordered carbonara, got bolognese" — is a retrieval error. When the chef sends out a tasting menu, you're receiving recommendation: the chef built a model of you and is making predictions. A mistake — "the chef brought me mushrooms, which I hate" — is a modeling error. Different failures, different fixes.

The Grocery List (Search) vs. The Recipe Suggestion (Recommendation). You walk into a store with a list: milk, eggs, bread. Search. The store's app notices your cart and says: "With those ingredients, you could make shakshuka. Want me to show you cumin and paprika?" Recommendation. The deviation from the list is the point in one case. It's a failure in the other.

The Error Cost Difference. A search error is a precision failure: "I asked for X and got Y." The user immediately knows something is wrong. The error is visible and attributable. A recommendation error is a relevance failure: "The system showed me something I don't care about." The user doesn't know if the system is broken or if they're just having a bad experience. The error is invisible and erodes trust cumulatively.

This is why Peter Norvig's 80% rule applies to search but not recommendation. If a search engine gets 80% right, users happily ignore the other 20% — they can see what went wrong. If a recommender gets 80% right, users notice the 20% wrong more than the 80% right, because every wrong suggestion is an interruption — screen space that could have shown something they'd love.

Why "Two Sides of the Same Coin" Is Misleading

The coin metaphor implies you can flip from one to the other by changing perspective. You can't. The data is different (queries vs. behavioral histories). The latency profiles are different (sub-100ms real-time vs. batch-refreshed). The error tolerance is different (visible and dismissable vs. invisible and trust-eroding). The user posture is different (active and goal-directed vs. passive and open).

The better metaphor is a restaurant: search is the menu, recommendation is the tasting menu, and the LLM-powered hybrid is a waiter who listens to what you're in the mood for, knows what the kitchen does well tonight, and helps you decide — sometimes by pointing at the menu, sometimes by making a suggestion you wouldn't have thought of, and always by knowing which mode you're in right now.

Five Principles for Search and Recommendation

1. Make task identity a first-class feature.

class TaskAwareModel(torch.nn.Module):
    """
    Whether you use Netflix's approach (separate output heads on a shared backbone),
    Spotify's approach (LLM router dispatching to separate systems), or DoorDash's
    approach (separate retrieval with shared embeddings) — the model must know
    whether it's doing search or recommendation.
    """

    def __init__(self, shared_backbone, num_tasks: int):
        super().__init__()
        self.backbone = shared_backbone
        self.task_embedding = torch.nn.Embedding(num_tasks, embedding_dim)
        self.heads = torch.nn.ModuleDict({
            "search": SearchHead(),
            "recommendation": RecommendationHead(),
            "similar_items": SimilarItemsHead(),
        })

    def forward(self, inputs, task_type: str):
        task_encoding = self.task_embedding(TASK_IDS[task_type])
        hidden = self.backbone(inputs, task_encoding)
        return self.heads[task_type](hidden)

# The task_type signal propagates through the entire model.
# It tells the model: "optimize for relevance to query" vs.
# "optimize for long-term engagement." These are different instructions.

2. Don't let personalization overpower query relevance.

Netflix learned this the hard way with UniCoRn. They added personalization to search incrementally, with explicit guardrails. The fully personalized model improved both tasks — but only after careful tuning to ensure search results remain relevant to the query even as they benefit from personalization signals. This is a specific instance of a broader principle: model complexity in ML systems incurs a technical debt that compounds silently [5].

def safe_personalized_search(query: str, user_profile: dict,
                             base_ranker, personalization_model,
                             relevance_threshold: float = 0.7) -> list[Item]:
    """Personalize search results without overriding query relevance."""
    # Stage 1: Get relevance-scored candidates
    candidates = base_ranker.retrieve(query, k=200)

    # Stage 2: Apply personalization re-ranking
    personalized = personalization_model.rerank(candidates, user_profile)

    # Stage 3: Enforce relevance floor
    # A result that is NOT relevant to the query should never outrank
    # a result that IS relevant, regardless of personalization score.
    for i, item in enumerate(personalized):
        if item.relevance_score < relevance_threshold:
            personalized[i].final_score *= 0.1  # severe penalty

    personalized.sort(key=lambda x: x.final_score, reverse=True)
    return personalized[:10]

3. Treat the user's posture as a design constraint.

Search users lean forward; recommendation users lean back. This affects latency budgets, UI, and the acceptable cost of being wrong.

Constraint Search Recommendation
Latency SLA <100ms per keystroke <200ms per page load
Error visibility User sees it immediately User may never notice — or slowly churn
Query volume Every keystroke Every page view
Freshness requirement Near-real-time (new docs) Daily batch (new items)
UI expectation Explicit, controllable Ambient, delightful

4. Share infrastructure where it makes sense — separate where it doesn't.

# SHARED: embedding stores, feature stores, model training pipelines
shared_infrastructure = {
    "feature_store": "Feast",           # same features, different consumers
    "embedding_index": "FAISS/Milvus",  # same ANN, different query patterns
    "model_registry": "MLflow",         # same versioning, different models
    "experiment_platform": "A/B tests", # same framework, different metrics
}

# SEPARATE: retrieval indices, ranking objectives, evaluation
separate_infrastructure = {
    "search_index": "Elasticsearch/BM25",     # inverted index, text-optimized
    "recs_index": "ANN over user-item space", # vector index, behavior-optimized
    "search_objective": "NDCG/MRR",           # relevance to query
    "recs_objective": "retention/discovery",  # long-term satisfaction
    "search_eval": "explicit relevance judgments",  # human-labeled
    "recs_eval": "A/B test on member retention",    # behavioral
}

These infrastructure choices — what to share, what to separate — are foundational decisions in data-intensive system design [7].

5. LLMs are infrastructure, not a replacement for retrieval.

The most successful production deployments use LLMs for content understanding, query intent classification, and feature generation offline, while keeping online retrieval in purpose-built low-latency systems:

class HybridLLMRetrievalPipeline:
    """LLMs enrich — they don't replace."""

    def offline_enrichment(self, catalog: list[Item]):
        """LLM generates rich profiles. Runs nightly. Costs amortized."""
        for item in catalog:
            item.llm_profile = self.llm.describe(item)       # rich description
            item.llm_embedding = self.encoder.encode(item.llm_profile)
            item.llm_tags = self.llm.extract_tags(item)      # structured metadata

    def online_search(self, query: str, k: int = 10) -> list[Item]:
        """No LLM in the request path. Sub-100ms."""
        # LLM-generated embeddings and tags are already indexed.
        # This is just BM25 + ANN retrieval + cross-encoder re-rank.
        return self.retrieval_pipeline.search(query, k)

    def online_recommendations(self, user_id: str, k: int = 10) -> list[Item]:
        """No LLM in the request path. Sub-200ms."""
        # LLM-generated carousel intents are pre-computed.
        # This is just embedding lookup + ANN + re-rank.
        return self.recs_pipeline.recommend(user_id, k)

A Reference Architecture

Bringing everything together — inspired by real-world architectures from Netflix and others [6] — here's what a system that handles both search and recommendation looks like in 2026:

class SearchAndRecommendationSystem:
    """
    Reference architecture for a system that does both.

    Shared: embeddings, feature store, LLM enrichment pipeline.
    Separate: retrieval indices, ranking heads, objectives, evaluation.
    Task-aware: every model knows which mode it's in.
    """

    def __init__(self):
        # Shared offline enrichment (LLM — runs nightly)
        self.content_enricher = LLMContentEnricher()
        self.embedding_encoder = GeminiEncoder(dim=256)

        # Separate retrieval indices
        self.search_index = ElasticsearchBM25()     # text-optimized
        self.recs_index = MilvusANN()               # user-item optimized

        # Shared backbone, task-specific heads
        self.model = TaskAwareTwoTower(
            shared_backbone=TransformerEncoder(layers=6),
            tasks=["search", "recommendations", "similar_items"]
        )

        # Separate evaluation
        self.search_eval = SearchEvaluator(metric="ndcg@10")
        self.recs_eval = RecsEvaluator(metric="retention_30d")

    def nightly_batch(self):
        """Run once per day: LLM content enrichment, embedding refresh."""
        for item in self.catalog:
            item.profile = self.content_enricher.describe(item)
            item.embedding = self.embedding_encoder.encode(item.profile)

        self.search_index.rebuild()
        self.recs_index.rebuild()
        self.content_enricher.generate_carousels()  # DoorDash-style memory blocks

    def serve_search(self, query: str, user_id: str) -> SearchResult:
        """Online search: sub-100ms, no LLM in path."""
        candidates = self.search_index.retrieve(query, k=200)
        scored = self.model.score(candidates, task="search",
                                  query=query, user_id=user_id)
        return self.search_reranker.apply(scored, user_id)

    def serve_homepage(self, user_id: str) -> list[Carousel]:
        """Online recommendations: sub-200ms, no LLM in path."""
        carousels = self.carousel_store.lookup(user_id)
        filled_carousels = []
        for carousel in carousels:
            items = self.recs_index.retrieve(carousel.embedding, k=20)
            scored = self.model.score(items, task="recommendations",
                                      user_id=user_id)
            filled_carousels.append(Carousel(carousel.title, scored[:10]))
        return filled_carousels

Conclusion: The Coin and the Mint

In 1992, Belkin and Croft asked whether information retrieval and information filtering were two sides of the same coin. Thirty-four years later, the question has a sharper answer: they share a mint.

The mathematical machinery — vector spaces, embedding learning, transformer attention, contrastive objectives — is increasingly shared. Netflix's UniCoRn, Spotify's intent router, DoorDash's content embeddings, Pinterest's multi-task two-tower models — all exploit this commonality. As Greg Linden showed at Amazon, the algorithms that retrieve known items and the algorithms that surface unknown ones succeed for different reasons. As Reed Hastings understood, the emotional contract of "Netflix gets me" is not the same as "Netflix found what I searched for." As Karen Spärck Jones knew in 1999, systems that assist human users cannot replace them — they can only narrow the gap. As Peter Norvig warned, the gap between "here are some suggestions" and "here is what you need" demands a higher standard of trust.

The arrival of LLMs makes these distinctions more important, not less. When a single model can retrieve, rank, recommend, and explain — all in natural language — the question is no longer "can we unify?"

The question is: "do we know which one we're doing right now?"

The answer must be yes. Because search competes with ignorance — it helps people find what they know they need. Recommendation competes with sleep — it surfaces what people didn't know they wanted, in a world of infinite alternatives. They are not the same fight. And the systems that win both are the ones that never forget which one they're fighting right now.



Open Questions

  1. The five principles in this post are derived from what worked at Netflix, Spotify, DoorDash, Airbnb, and Pinterest. What principles are missing because no company has solved certain problems yet — like fully autonomous switching between search and recommendation without explicit routing?

  2. LLMs as offline enrichment, purpose-built systems for online serving — this is the consensus in 2026. Will it hold? Or will end-to-end generative models become fast enough that the hybrid architecture becomes unnecessary overhead?

  3. "Do we know which one we're doing right now?" is the organizing question of this series. Five years from now, will that question still need a human-designed answer — or will the system infer it from context more reliably than any explicit signal?

  4. Search competes with ignorance. Recommendation competes with sleep. What does a system that does both compete with — and how do we measure whether it's winning?

References

  1. Nicholas J. Belkin and W. Bruce Croft. Information Filtering and Information Retrieval: Two Sides of the Same Coin?. Communications of the ACM, 35(12): 29–38, 1992.

  2. Francesco Ricci, Lior Rokach, and Bracha Shapira (editors). Recommender Systems Handbook, 3rd Edition. Springer, 2022.

  3. Yutao Zhu et al. Large Language Models for Information Retrieval: A Survey. ACM TOIS, 2024.

  4. Yongqi Li et al. A Survey of Generative Search and Recommendation in the Era of Large Language Models. arXiv:2404.16924, 2024.

  5. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. Machine Learning: The High Interest Credit Card of Technical Debt. SE4ML, NeurIPS 2014.

  6. Xavier Amatriain and Justin Basilico. Recommender Systems in Industry: A Netflix Case Study. RecSys 2014 Tutorial.

  7. Martin Kleppmann. Designing Data-Intensive Applications. O'Reilly, 2017. The canonical reference on system design for data-intensive applications.

  8. Gregor Hohpe and Bobby Woolf. Enterprise Integration Patterns. Addison-Wesley, 2003.

  9. Eugene Yan et al. Applying ML to Search and Recommendation. Blog, 2024.

  10. Chip Huyen. Designing Machine Learning Systems. O'Reilly, 2022.


S&R: If LLMs Can Do Both, Does the Distinction Still Matter?

LLMs transform search and recommendation in different ways — RAG, generative retrieval, conversational recommendation, agentic pipelines. But the distinction survives in objective alignment, evaluation, serendipity tolerance, and infrastructure.

searchrecommendationllmraggenerative-retrievalagenticsurveyseries

S&R stands for Search & Recommendation. LLMs don't know which one they're doing. They see tokens in, tokens out. The distinction that traditional retrieval systems encode in architecture — inverted index vs. embedding store, NDCG vs. retention, sub-100ms keystroke latency vs. nightly batch refresh — must now be encoded somewhere else. The question is where.

Ask an LLM "What is the capital of France?" That is search. The model retrieves a fact that matches the explicit query. Fidelity to the question is all that counts. If the model answers "Rome" because it has learned that users who ask about capitals often enjoy Italian geography, the model is broken. The query is a contract. The answer must honor it.

Ask an LLM "What should I watch tonight? I loved Dark and Severance." That is recommendation. The model infers preferences from the examples and generates suggestions the user might not have thought of. Serendipity is not a bug — it is the point. If the model says "watch Dark again," it has failed. The same model performed both tasks. The same prompt interface hid the difference. The difference did not disappear. It moved into the prompt, the objective function, the evaluation framework, and the human expectation of what "good" means. LLMs are infrastructure. They do not eliminate the distinction between search and recommendation. They make it more important to get right.

The arrival of LLMs is the most significant development in both information retrieval and recommendation since BERT [7]. But the way LLMs affect each field is different, and understanding the difference is essential for engineering.

Two recent surveys capture the scope. Zhu et al. (2024) organize LLM-IR integration into four roles: query rewriter, retriever, reranker, and reader [1]. Li et al. (2024) provide the cross-cutting view — framing both search and recommendation as instances of generative retrieval and identifying where they converge and where they diverge [2].

The key insight: LLMs don't eliminate the distinction. They reveal it at a higher level of abstraction. Both fields are moving toward generative paradigms, but the thing being generated is different. Search generates answers from documents. Recommendation generates item predictions from user histories.

LLMs operate at four levels in the search pipeline:

# Level 1: Query Understanding — LLMs disambiguate and expand queries
def llm_query_understanding(raw_query: str, llm) -> dict:
    """Transform natural language into structured search intent."""
    prompt = f"""
    Parse this search query into structured intent:
    Query: "{raw_query}"

    Return JSON with: intent_type, entities, constraints, expansions.
    """
    response = llm.generate(prompt)
    return json.loads(response)

# "Show me movies like Inception but funnier"
# → {intent_type: "similar_items_with_constraint",
#    entities: ["Inception"],
#    constraints: ["genre: comedy", "tone: lighter"],
#    expansions: ["mind-bending heist films", "comedic thrillers"]}

# Level 2: Document Understanding — LLMs generate richer representations
def llm_enrich_document(doc: str, llm) -> dict:
    """Generate structured metadata that transcends keyword matching."""
    prompt = f"""
    For this document, generate:
    1. A 2-sentence summary
    2. 5-10 key phrases
    3. Entity tags (people, places, concepts)
    4. A semantic embedding-friendly description

    Document: {doc[:4000]}
    """
    return llm.generate_structured(prompt)

# Level 3: RAG — Retrieve then generate a synthesized answer
def rag_search(query: str, retriever, llm, k: int = 5) -> str:
    """The dominant LLM-search paradigm in 2025."""
    # Retrieve relevant documents
    docs = retriever.retrieve(query, k=k)

    # Generate answer grounded in retrieved documents
    context = "\n\n".join(f"[{i+1}] {doc.text}" for i, doc in enumerate(docs))
    prompt = f"""
    Answer the query using ONLY the provided documents.
    Cite sources by number.

    Query: {query}

    Documents:
    {context}
    """
    return llm.generate(prompt)

# Level 4: Generative Retrieval — the model IS the index
# The most radical approach: the LLM directly generates document IDs
# without an explicit retrieval index. Still experimental.

The core search user need — "I have a question, find me the answer" — aligns naturally with LLM capabilities. The LLM augments retrieval at every stage without replacing it [6]. This paradigm extends the text-to-text framework [10] that first showed how retrieval tasks could be unified under a generative umbrella.

How LLMs Transform Recommendation

For recommendation, the transformation is more structural:

# Level 1: Feature Engineering Automation
# LLMs generate rich profiles from sparse structured data.
# DoorDash's content embeddings and Consumer Memory Blocks are paradigmatic:
# the LLM produces the fuel, retrieval and ranking remain purpose-built.

def llm_generate_item_profile(item: dict, llm) -> str:
    """Produce a rich narrative that a standard encoder can embed."""
    return llm.generate(f"Describe this item for a recommendation system: {item}")

# Level 2: Generative Recommendation
# Instead of scoring candidates, the LLM autoregressively generates item tokens.
# Meta's HSTU, Kuaishou's OneRec, Google's TIGER are examples.

class GenerativeRecommender:
    """Recommendation as sequence-to-sequence: predict next-item tokens."""
    def recommend(self, user_history: list[int], llm, k: int = 10) -> list[int]:
        prompt = f"User interaction history (item IDs): {user_history}\nPredict next items:"
        # LLM autoregressively generates item IDs
        return llm.generate_tokens(prompt, max_tokens=k)

# Level 3: Conversational Recommendation
# Multi-turn dialogue where the system elicits preferences and adapts.

def conversational_recommendation_loop(user_id: str, llm, catalog):
    """Search and recommendation blur into conversation."""
    context = {"history": get_user_history(user_id)}
    for turn in range(5):
        user_input = get_user_response()
        if is_search_like(user_input):    # "I want a sci-fi movie"
            results = catalog.search(user_input)
            context["mode"] = "search"
        else:                              # "something cerebral, not too long"
            results = catalog.recommend(context)
            context["mode"] = "recommendation"
        llm_response = llm.generate_response(results, context)
        show_to_user(llm_response)

# Level 4: Agentic Recommendation
# LLM-powered agents that plan, use tools, maintain memory, reason.
# ARAG (SIGIR 2025): 4 specialized agents — User Understanding, NLI,
# Context Summary, Item Ranker — achieve +42.1% NDCG@5 over vanilla RAG.

Why the Distinction Survives

Despite this convergence, the distinction remains essential for four reasons:

1. Objective alignment. A search system optimizing for engagement shows addictive content instead of answering the query. A recommendation system optimizing for precision produces an echo chamber.

# Wrong: one objective for both
def bad_unified_objective(model_output, labels):
    return cross_entropy(model_output, labels)  # what are we even optimizing?

# Right: task-aware objectives
def search_objective(predictions, relevance_labels):
    return ndcg_loss(predictions, relevance_labels)  # did we retrieve the right thing?

def recommendation_objective(predictions, engagement_labels):
    return binary_cross_entropy(predictions, engagement_labels)  # will user engage?
    # Plus: diversity bonus, freshness decay, exploration budget, long-term retention proxy

2. Evaluation. Search: NDCG, MRR, precision@K against relevance judgments. Recommendation: retention, discovery, session length, long-term satisfaction. These are correlated but not identical.

3. The serendipity gradient. In search, serendipity is a bug. In recommendation, it's a feature.

def demonstrate_serendipity_gradient():
    """The same behavior is a bug in one context and a feature in the other."""

    # SEARCH: User types "The Godfather"
    # System returns "Goodfellas" because "people who watch The Godfather
    # also love Goodfellas"
    # → BROKEN. The user wanted The Godfather. Return The Godfather.

    # RECOMMENDATION: User browses homepage on Friday night
    # System shows "The Godfather" — which the user has already seen 4 times
    # → BROKEN. The user wants something new. Show Goodfellas.

    # Both failures happen when teams optimize for the wrong thing.
    # The first optimized for engagement instead of relevance.
    # The second optimized for precision instead of discovery.

4. Infrastructure. Search: inverted indices, real-time query parsing, sub-100ms latency at high QPS. Recommendation: user profile stores, embedding indices, offline batch pipelines amortized across daily refresh cycles.

# Search infrastructure
class SearchServing:
    latency_sla: float = 0.100  # seconds — user is waiting
    qps: int = 10_000           # every keystroke is a query
    freshness: str = "real-time"  # new documents must be searchable immediately
    index_type: str = "inverted"  # keyword → posting list

# Recommendation infrastructure
class RecServing:
    latency_sla: float = 0.200  # seconds — page load, more forgiving
    qps: int = 1_000            # one request per page view
    freshness: str = "daily"      # embeddings recomputed nightly
    index_type: str = "ANN"      # vector → nearest neighbors

The Survey Papers Agree

The Li et al. (2024) survey on generative search and recommendation puts it well: the fields share a common mathematical framework — matching entities across representation spaces — but differ in the nature of the mismatch [2]. Search deals with query–document mismatch: different words for the same concept. Recommendation deals with user–item mismatch: users and items live in completely different semantic spaces. The first is a lexical/semantic gap. The second is a modality gap. Solving them requires different tools, even when those tools share a common transformer backbone.

The Gupta et al. (2025) survey on generative recommendation reinforces this: industrial systems like TIGER, LIGER, OneRec, and HSTU all treat recommendation as its own generative problem with its own tokenization, its own evaluation, and its own cold-start challenges [3]. None of them use the same architecture unmodified for search.

LLMs are infrastructure. They don't eliminate the distinction between search and recommendation — they make it more important to get right.



Open Questions

  1. LLMs are infrastructure, not a replacement for retrieval. But the history of software is infrastructure absorbing what was once application logic. Will retrieval become just another token prediction task?

  2. Generative recommendation (TIGER, HSTU, OneRec) treats item prediction as autoregressive generation. If the model is the index, the retriever, and the ranker — where does the search/recommendation boundary go? Does it vanish, or does it move into the prompt?

  3. The surveys agree: LLMs affect search and recommendation differently. But most LLM research papers treat them as interchangeable downstream tasks. Is the research community making the same mistake the engineering community is learning to avoid?

  4. Conversational recommendation collapses the boundary between search and discovery into dialogue. If users can't tell whether they're searching or being recommended to — does the distinction still matter? For whom?

References

  1. Yutao Zhu, Huaying Yuan, Shuting Wang, et al. Large Language Models for Information Retrieval: A Survey. ACM Transactions on Information Systems, 2024.

  2. Yongqi Li et al. A Survey of Generative Search and Recommendation in the Era of Large Language Models. arXiv:2404.16924, 2024.

  3. Shashank Gupta et al. Generative Recommendation: A Survey of Models, Systems, and Industrial Advances. TechRxiv, 2025.

  4. Chi Zhang et al. ARAG: Agentic Retrieval Augmented Generation for Personalized Recommendation. SIGIR 2025.

  5. Zhuang Liu et al. A Comprehensive Survey on LLM-Powered Recommender Systems. IEEE Access, 2024.

  6. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.

  7. Tom Brown et al. Language Models are Few-Shot Learners. NeurIPS 2020. The GPT-3 paper.

  8. Sébastien Bubeck et al. Sparks of Artificial General Intelligence: Early Experiments with GPT-4. arXiv, 2023.

  9. Hugo Touvron et al. LLaMA: Open and Efficient Foundation Language Models. arXiv, 2023.

  10. Colin Raffel et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR, 2020.


S&R: Where Should You Draw the Line Between Search and Recommendation?

How four leading engineering organizations handle the search–recommendation boundary in production: Spotify's LLM intent router, DoorDash's content-first embeddings and Consumer Memory Blocks, Airbnb's listing embeddings for real-time search personalization, and Pinterest's two-tower architectures.

searchrecommendationspotifydoordashairbnbpinteresttwo-towerembeddingsseries

S&R stands for Search & Recommendation. Every company that does both draws the line somewhere. Spotify routes on intent. DoorDash separates the pipelines. Airbnb shares embeddings but splits the ranking features. The line is a choice, and the choice has architectural consequences. This post examines four different answers to the same question: where does search end and recommendation begin?

On the search side of the line, a user articulates intent. The system retrieves. Fidelity to the query is the metric. Personalization is optional, applied sparingly, with guardrails. If a DoorDash user types "Sichuan noodle soup" and the system shows pizza because their order history says they love pizza, the system has crossed the line in the wrong direction. The query is a contract. The system's job is to honor it.

On the recommendation side of the line, the user articulates nothing. The system infers from behavior. Surprise is a feature, not a bug. If the system only shows what the user has already ordered, it has failed to recommend — it has only retrieved from memory. The error is subtler. A bad search result is visible: "I asked for X and got Y." A bad recommendation is invisible: the user didn't see what they would have loved, and they will never know. The line between the two is where engineering meets judgment.

Netflix isn't the only company navigating the search–recommendation boundary. Spotify, DoorDash, Airbnb, and Pinterest each handle it differently — and their engineering blogs document the trade-offs. This part examines four production architectures.

Spotify: Intent-Based Routing with LLMs

Spotify's 2025 paper "You Say Search, I Say Recs" describes an LLM-based router that classifies user intent and dispatches accordingly [1]:

from enum import Enum

class QueryIntent(Enum):
    NAVIGATIONAL = "navigational"    # "find song X" → search
    EXPLORATORY = "exploratory"      # "Italian 80s disco nostalgia" → recs
    MIXED = "mixed"                  # both paths, parallel → fused results

class SpotifyIntentRouter:
    """
    LLM-based router: classify intent, route to search or recommendation.

    Key design decisions:
    - Only the query goes to the router (NOT user features) → max cache hits
    - User features go to downstream tools → personalization where it belongs
    - Small distilled LLM: 60% latency reduction, 99% cost reduction vs teacher
    """

    def __init__(self, llm_model):
        self.llm = llm_model  # small fine-tuned model, p75 latency ~450ms

    def classify(self, query: str) -> QueryIntent:
        """Classify query intent. Cached aggressively — no user features."""
        if cached := self.cache.get(query):
            return cached
        intent = self.llm.classify(query)  # "navigational", "exploratory", "mixed"
        self.cache.set(query, intent)
        return intent

    def route(self, query: str, user_features: dict) -> SearchResult:
        intent = self.classify(query)

        if intent == QueryIntent.NAVIGATIONAL:
            return self.search_backend.search(query, user_features)
            # Elasticsearch + BM25 + Voyager ANN

        elif intent == QueryIntent.EXPLORATORY:
            return self.recs_backend.recommend(query, user_features)
            # Collaborative filtering + content-based + Voyager ANN

        else:  # MIXED — both paths in parallel, fuse results
            search_result = self.search_backend.search_async(query, user_features)
            recs_result = self.recs_backend.recommend_async(query, user_features)
            return self.fuse(search_result, recs_result)

# Results vs. regular search:
# +115% for finding similar artists
# +91% for new music discovery
# +25% for broad music searches

# Spotify's Unified Embedding Infrastructure (Voyager) [8]:
# HNSW-based, 10x faster than Annoy at same recall.
# Powers both search AND recommendation — shared infrastructure,
# task-specific usage patterns.

Spotify's key finding: Semantic IDs optimized for search don't generalize to recommendation, and vice versa. A multi-task bi-encoder achieves a Pareto-optimal trade-off, but you can't optimize one embedding space for both tasks without compromise [2].

DoorDash: Content-First Embeddings, Task-Aware Retrieval

DoorDash's environment is intent-driven and transactional. A click tells you less than you think:

# DoorDash's core insight: clicks are poor proxies for semantics.
# A click on a Sichuan noodle soup doesn't distinguish spicy preference
# from noodle preference — or from just being hungry.

# Their solution: LLMs generate rich content profiles, embeddings follow.

class DoorDashContentPipeline:
    """Content-first: LLM profiles → off-the-shelf encoder → ANN retrieval."""

    def generate_item_profile(self, item: dict) -> str:
        """LLM produces a standardized narrative for every item."""
        prompt = f"""
        Describe this menu item for a food recommendation system:
        Name: {item['name']}
        Category: {item['category']}
        Ingredients: {item['ingredients']}
        Preparation: {item['preparation']}
        Cuisine type: {item['cuisine']}

        Cover: ingredients, preparation method, cuisine attributes,
        dietary properties (spicy, vegetarian, etc.), eating context,
        flavor profile.
        """
        return self.llm.generate(prompt)

    def embed(self, profile: str) -> np.ndarray:
        """Encode the profile with an off-the-shelf encoder."""
        return self.encoder.encode(profile)  # gemini-embedding-001, 256-dim MRL

    def retrieve(self, query_embedding: np.ndarray, k: int = 100) -> list[int]:
        """ANN search over pre-computed item embeddings."""
        return self.milvus_index.search(query_embedding, k)

    # Key finding: upgrading profile quality → +31% improvement.
    # Upgrading the encoder on raw metadata → only +6%.
    # Data quality dominates model choice.

# Results:
# -3.65% null search rate, +0.66% CVR, +0.072% 7D active customers

For recommendations, DoorDash developed Consumer Memory Blocks — typed, composable representations of everything known about a user:

class ConsumerMemoryBlock:
    """
    Structured, namespaced user state serialized as JSON for LLM prompt input.

    Properties:
    - Composable: different use cases request different sub-blocks
    - Evidenced, not inferred: derived from observed behavior with provenance
    - Extensible: new sub-blocks without downstream disruption
    """

    def build(self, user_id: str, sub_blocks: list[str]) -> dict:
        blocks = {}
        if "long_term_preferences" in sub_blocks:
            blocks["long_term"] = {
                "cuisines": ["Thai", "Sichuan", "Italian"],
                "dietary": ["prefers spicy", "avoids dairy"],
                "price_range": "$$-$$$",
                "avg_order_value": 42.50,
            }
        if "behavioral_patterns" in sub_blocks:
            blocks["patterns"] = {
                "order_days": ["Fri", "Sat"],
                "peak_time": "19:00-21:00",
                "group_orders": True,
                "repeat_rate": 0.35,
            }
        return blocks

    def to_prompt(self, blocks: dict) -> str:
        """Serialize blocks as compact JSON for the LLM carousel generator."""
        return json.dumps(blocks)

# Generated carousels are embedded OFFLINE, retrieved via Milvus ONLINE.
# No LLM in the request path — cost amortized across the refresh interval.
# Results: +2.4% order rate.

Airbnb: Embeddings That Bridge Search and Discovery

Airbnb's KDD 2018 paper trained listing embeddings on 800 million search click sessions using skip-gram adapted from word2vec [3]:

class AirbnbListingEmbeddings:
    """Domain-specific embedding training for travel search."""

    def generate_training_pairs(self, sessions: list[list[int]]) -> list[tuple[int, int]]:
        """
        Key innovations over standard word2vec:

        1. In-market negative sampling: users search within a single city.
           Negatives drawn globally are trivially easy to reject.
           Draw negatives from the SAME MARKET for harder discrimination.

        2. Booked listing as global context: the booked listing is ALWAYS
           treated as the context being predicted, regardless of position in
           the click sequence. The booking is the signal. Everything else is noise.
        """
        pairs = []
        for session in sessions:
            booked_id = session[-1]  # last item is the booking
            for clicked_id in session[:-1]:
                pairs.append((clicked_id, booked_id))
        return pairs

    def compute_serving_features(self, user_id: int, candidates: list[int],
                                 embeddings: dict[int, np.ndarray]) -> dict[str, np.ndarray]:
        """
        Embedding-based features for real-time ranking.

        Five embedding features ranked among the TOP 20 of 104 total features.
        """
        user_recent = self.get_recent_interactions(user_id)
        return {
            # Similarity to listings the user recently clicked
            "EmbClickSim": self.mean_sim(user_recent['clicked'], candidates, embeddings),
            # Similarity to listings the user skipped
            "EmbSkipSim": self.mean_sim(user_recent['skipped'], candidates, embeddings),
            # Similarity to the last listing the user spent significant time on
            "EmbLastLongClickSim": self.mean_sim([user_recent['last_long_click']],
                                                  candidates, embeddings),
        }

# Results: +2.27% offline NDCG, statistically significant booking gain online.
# Similar listing recommendations: +20% CTR over prior algorithm.

Airbnb's 2019 follow-up on deep learning documented instructive failures: listing ID embeddings overfit (too few bookings per listing), and multi-task learning for bookings + long views increased views but not bookings — because expensive listings get looked at but not booked [4].

Pinterest's two-tower model, influenced by the Wide & Deep architecture [9], powers both homefeed and search:

class PinterestTwoTower(torch.nn.Module):
    """Two-tower: user tower and item tower, dot-product scoring."""

    def __init__(self, user_input_dim: int, item_input_dim: int, hidden: int = 256):
        super().__init__()
        # User tower: long-term history, context, real-time sequences
        self.user_tower = torch.nn.Sequential(
            torch.nn.Linear(user_input_dim, hidden * 4),
            torch.nn.ReLU(),
            MaskNet(hidden * 4, num_blocks=3),   # bitwise feature crossing
            torch.nn.Linear(hidden * 4, hidden),
        )
        # Item tower: category, description, image features
        self.item_tower = torch.nn.Sequential(
            torch.nn.Linear(item_input_dim, hidden * 4),
            torch.nn.ReLU(),
            torch.nn.Linear(hidden * 4, hidden),
        )

    def forward(self, user_features, item_features):
        user_emb = F.normalize(self.user_tower(user_features), dim=1)
        item_emb = F.normalize(self.item_tower(item_features), dim=1)
        return (user_emb * item_emb).sum(dim=1)  # dot product

# Item embeddings pre-computed offline, indexed in Manas (HNSW-based ANN).
# User tower runs once per request at serving time.
# This decoupling is the key to ~3ms median latency at 300K QPS.

For search, Pinterest extended this into OmniSearchSage — a multi-task, multi-entity framework where a single unified query embedding retrieves pins, products, and related queries simultaneously [5]. A teacher-student distillation approach followed: an 8B Llama cross-encoder (teacher, +12–20% improvement) → bi-encoder student trained on 100× more data from daily search logs (85% query cache hit rate).

The Pattern Across All Four Companies

Every system that successfully spans search and recommendation does so by encoding which task it's doing as a first-class signal. Spotify uses an explicit router. DoorDash uses separate retrieval pipelines with shared embeddings. Airbnb uses the same embeddings but different ranking features for search vs. discovery. Pinterest uses the same two-tower but different objectives for homefeed vs. search.

The common pattern: unification where it reduces cost, separation where it preserves correctness.



Open Questions

  1. Spotify found that Semantic IDs optimized for search don't generalize to recommendation, and vice versa. Is this a fundamental property of the two tasks, or an artifact of how we train embeddings? What would a truly unified embedding space look like?

  2. DoorDash's finding — data quality (+31%) dominates model choice (+6%) — echoes a broader truth. How much of the search-vs-recommendation gap is actually a data quality gap in disguise?

  3. Every successful system in this survey encodes task identity as a first-class signal. Is the search/recommendation distinction a permanent architectural feature, or a temporary crutch that better models will eventually absorb?

  4. Airbnb's failed multi-task experiment (bookings + long views) is a warning about proxy objectives. What other proxy objectives are we optimizing across the industry that don't actually measure what we think they measure?

References

  1. Spotify Research. You Say Search, I Say Recs: A Scalable Agentic Approach to Query Understanding and Exploratory Search. September 2025.

  2. Spotify Research. Semantic IDs for Generative Search and Recommendation. September 2025.

  3. Mihajlo Grbovic and Haibin Cheng. Real-time Personalization using Embeddings for Search Ranking at Airbnb. KDD 2018.

  4. Malay Haldar et al. Applying Deep Learning to Airbnb Search. KDD 2019.

  5. DoorDash Engineering. Using LLMs to Build Content Embeddings for Search and Recommendations. 2025.

  6. DoorDash Engineering. Offline LLMs, Online Personalization: Generating Carousels at DoorDash. 2025.

  7. Pinterest Engineering. Advancements in Embedding-Based Retrieval at Pinterest Homefeed. 2024.

  8. Spotify Engineering. Introducing Voyager: Spotify's New Nearest-Neighbor Search Library. October 2023.

  9. Heng-Tze Cheng et al. Wide & Deep Learning for Recommender Systems. RecSys 2016.

  10. Mihajlo Grbovic et al. E-commerce in Your Inbox: Product Recommendations at Scale. KDD 2015.

  11. Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. BPR: Bayesian Personalized Ranking from Implicit Feedback. UAI 2009.


S&R: What Happens When You Build Search and Recommendation at the Same Company?

Deep dive into Netflix's search and recommendation systems — Elasticsearch/Flink indexing pipelines, three-tier serving architecture, the Foundation Model for personalized recommendation, and UniCoRn's unified approach to search and recs.

searchrecommendationnetflixelasticsearchcollaborative-filteringfoundation-modelunicornseries

S&R stands for Search & Recommendation. Netflix runs both at global scale on the same catalog, the same users, the same infrastructure — and two completely different contracts with the person on the other side of the screen.

When a Netflix user types "time travel movies" into the search bar, they have intent. They want results ranked by relevance to those three words. The system's job is fidelity to the query. If UniCoRn shows The Notebook because the user's watch history says they love romance, the system is broken — personalization has overpowered relevance. Netflix learned this the hard way and added guardrails.

When a Netflix user opens the homepage, they have no query. They are waiting to be told what they want. The system's job is to infer it from everything it knows — watch history, time of day, device, what similar users binged last weekend. If it shows the same ten rows every day, the system is broken — it has failed to recommend, and only retrieved. Same movies. Same user. Different broken. The contracts are not the same. UniCoRn encodes this difference as a task_type feature because it cannot be ignored.

No company illustrates the search–recommendation distinction better than Netflix. They run both systems at global scale, on the same catalog, for the same users — and they've published extensively about each. This part examines both stacks, then looks at UniCoRn, their 2024 attempt to unify them.

Reed Hastings, Netflix's co-founder, articulated the recommendation goal simply: "If the Starbucks secret is a smile when you get your latte, ours is that the website adapts to the individual's taste." The emotional target — "Netflix gets me" — is fundamentally different from search's target of "Netflix found what I asked for." Recommendation is about identity. Search is about utility.

Netflix Studio Search indexes federated GraphQL data across the entire content production pipeline. The architecture is built on Elasticsearch (running Apache Lucene's BM25 under the hood) with a custom query DSL [1].

# Conceptual: Netflix's Studio Search indexing pipeline

# 1. Events stream into Kafka from applications and CDC
# 2. Apache Flink processes consume events, enrich via GraphQL, sink to Elasticsearch
# 3. A custom ANTLR-based query parser translates SQL-like DSL → Elasticsearch queries
# 4. Search returns entity keys only; results are hydrated via federated GraphQL
# 5. Authorization is late-binding, translated into Boolean filters AND-ed with the query

# Simplified: what a search request looks like
class NetflixSearchRequest:
    query: str            # e.g., "movies shooting in Mexico with Arnold Schwarzenegger"
    filters: dict         # e.g., {"content_type": "movie", "status": "in_production"}
    user_id: str          # for late-binding authorization
    facets: list[str]     # e.g., ["genre", "release_year", "language"]

class NetflixSearchResponse:
    entity_keys: list[str]       # just IDs — lean index
    facets: dict[str, list]      # aggregation results
    total_count: int

# Results are hydrated by the client via GraphQL:
# query { nodes(ids: $entity_keys) { title, synopsis, cast { name }, ... } }

Their Asset Management Platform (AMP) indexes over 7TB of digital media metadata. The hard lesson: their original design (one index per asset type, ~900 indices, 16,200 shards) caused CPU hotspots because shard sizes ranged from thousands to millions of documents. The fix — time-bucket-based indices with uniform sizes — dropped CPU from 70% to 10% [2].

# AMP's metadata model: handling 1000+ asset types with heterogeneous schemas
# Solution: nested metadata field with typed value columns

es_document = {
    "asset_id": "asset_12345",
    "created_at": "2026-01-15T10:30:00Z",
    "metadata": [
        {"key": "resolution", "string_value": "4K"},
        {"key": "runtime_minutes", "long_value": 142},
        {"key": "color_space", "string_value": "Rec.2020"},
        {"key": "file_size_gb", "double_value": 287.5},
        {"key": "has_subtitles", "boolean_value": True},
        {"key": "shoot_date", "date_value": "2025-11-03"},
    ]
}

# Query: "find all 4K assets shot after October 2025 larger than 200GB"
# This becomes nested Elasticsearch queries matching both key AND value fields

Netflix also uses Percolate Queries for reverse search — matching documents to queries instead of queries to documents [3]. When a production asset changes (e.g., "movie shooting in Mexico City without a key role assigned"), percolation identifies which saved searches match, enabling targeted notifications.

# Conceptual: percolation for production monitoring
class ReverseSearch:
    """Store queries as documents, match new data against them."""

    def index_saved_search(self, search_id: str, criteria: dict):
        """A creative executive saves: 'alert me when a thriller shoots in Thailand'."""
        es_query = translate_to_elasticsearch(criteria)
        es.index(index="saved_searches", id=search_id,
                 body={"query": es_query, "owner": "exec_42"})

    def percolate(self, asset_data: dict) -> list[str]:
        """When a new asset arrives, find all searches that match it."""
        result = es.percolate(index="saved_searches",
                             body={"doc": asset_data})
        return [match["_id"] for match in result["matches"]]

# This is the kind of problem that only exists when you think of search
# as a retrieval infrastructure problem, not just ranking.

Netflix Recommendation: Three Tiers, One Goal

Netflix's 2013 tech blog post describing the three-tier serving architecture remains the conceptual backbone [4]; Gomez-Uribe and Hunt later published the definitive description of the full recommendation system in 2015 [9]:

class NetflixRecommendationPipeline:
    """
    Three-tier architecture: offline, nearline, online.

    Offline: batch model training, feature pre-computation (hours/days)
    Nearline: event-triggered async processing (seconds/minutes)
    Online: real-time scoring with strict latency SLAs (milliseconds)
    """

    def offline_training(self):
        """Run nightly: train models, pre-compute features."""
        # Matrix factorization / foundation model training
        # Compute item-item similarity matrices
        # Pre-compute user embeddings for active users
        # Generate candidate sets per user cluster
        pass

    def nearline_update(self, user_id: str, event: str):
        """Triggers on user action: update recs after a viewing session."""
        if event == "finished_watching":
            # Update user embedding based on the completed title
            # Refresh the "Because You Watched" row
            # Adjust candidate generation weights
            pass

    def online_serve(self, user_id: str, context: dict, k: int = 40) -> list[str]:
        """Called on every page load. Must return in <200ms."""
        # Lookup pre-computed candidates
        # Score with real-time context (time of day, device, session)
        # Re-rank for diversity and freshness
        # Return the rows for the homepage
        pass

But the models themselves evolved dramatically. Their 2021 AI Magazine article contains a sobering finding: when only user–item interaction data is available, properly tuned non-deep-learning baselines remain competitive [5]. The power of deep learning emerged only when heterogeneous features (metadata, context, images, text) were incorporated.

This is a crucial point that connects back to the search–recommendation distinction: search has always been multi-modal — query text, document text, links, anchor text, click data, freshness. Recommendation was historically impoverished: just a ratings matrix. The deep learning era equalized this by giving recommendation systems the same heterogeneous-feature diet that search had enjoyed since the 2000s.

The Foundation Model (2025)

Building on a decade of personalization trends documented by Basilico and Raimond [11], in 2025 Netflix published their most ambitious recommendation paper: a Foundation Model for Personalized Recommendation that treats user interaction histories as sequences and uses autoregressive next-token prediction [6]:

class NetflixFoundationModel(torch.nn.Module):
    """
    Autoregressive model: user actions → next item prediction.

    Inspired by GPT but for user behavior instead of text.
    """

    def __init__(self, vocab_size: int, embed_dim: int = 1024,
                 num_layers: int = 24, num_heads: int = 16):
        super().__init__()
        self.token_embedding = torch.nn.Embedding(vocab_size, embed_dim)
        self.positional_encoding = PositionalEncoding(embed_dim)

        # Sparse attention for long sequences (hundreds of interactions)
        self.transformer_blocks = torch.nn.ModuleList([
            SparseTransformerBlock(embed_dim, num_heads)
            for _ in range(num_layers)
        ])

        # Multi-token prediction: predict next n items, not just one
        self.output_heads = torch.nn.ModuleList([
            torch.nn.Linear(embed_dim, vocab_size) for _ in range(5)
        ])

        # Auxiliary prediction heads as regularizers
        self.genre_head = torch.nn.Linear(embed_dim, num_genres)
        self.language_head = torch.nn.Linear(embed_dim, num_languages)

    def forward(self, token_ids: torch.Tensor, mask: torch.Tensor
                ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
        """Predict next items, genre, and language from interaction history."""
        x = self.token_embedding(token_ids)
        x = self.positional_encoding(x)

        for block in self.transformer_blocks:
            x = block(x, mask)

        # Next-item predictions (positions 1-5 ahead)
        item_preds = [head(x) for head in self.output_heads]
        # Auxiliary predictions
        genre_pred = self.genre_head(x.mean(dim=1))
        lang_pred = self.language_head(x.mean(dim=1))

        return item_preds, genre_pred, lang_pred

# Key design decisions:
# - User actions are "tokenized" via an analog of Byte Pair Encoding
# - Interaction tokens include watch duration, device, locale, time, item metadata
# - Cold-start titles: metadata+ID embedding with attention-based mixing weighted
#   by entity "age" — new titles lean on metadata, established ones on ID embeddings
# - Scaling laws confirmed: more data, more parameters, longer context → better

Hastings revealed why behavioral data trumps stated preferences: "What happens is, when we rate, we're meta-cognitive about quality — that's sort of our aspirational self. It works out much better, to please people, to look at the actual choices that they make." Users say they want documentaries; they watch reality TV. A recommender that trusts stated preferences fails. A search engine that second-guesses the query fails. This is the central tension between the two problems.

It's also why Hastings famously said Netflix "competes with sleep" — not just with other streaming services. The recommender's job is not to satisfy a stated need but to command attention in a world of infinite alternatives. Search competes with ignorance. Recommendation competes with every other possible way to spend time. Different competitions, different rules, different systems.

The Convergence: UniCoRn (2024)

Before UniCoRn, Netflix ran three separate models for search, homepage recommendations, and "More Like This." UniCoRn replaced all three [7].

class UniCoRn(torch.nn.Module):
    """
    Unified Contextual Recommender: one model, three tasks.

    The insight: treat task identity as a first-class feature.
    """

    def __init__(self, user_vocab_size: int, item_vocab_size: int,
                 embed_dim: int = 512):
        super().__init__()
        self.user_embedding = torch.nn.Embedding(user_vocab_size, embed_dim)
        self.item_embedding = torch.nn.Embedding(item_vocab_size, embed_dim)
        self.query_encoder = TextEncoder(embed_dim)  # encodes search query text
        self.task_embedding = torch.nn.Embedding(3, embed_dim)  # search/recs/similar

        # Shared transformer backbone
        self.shared_backbone = TransformerEncoder(
            d_model=embed_dim, num_layers=6, num_heads=8
        )

        # Task-specific output heads
        self.search_head = torch.nn.Linear(embed_dim, 1)        # P(click | query)
        self.recs_head = torch.nn.Linear(embed_dim, 1)          # P(watch | profile)
        self.similar_head = torch.nn.Linear(embed_dim, 1)       # P(watch | source item)

    def forward(self, user_id: int, item_ids: torch.Tensor,
                query_text: str | None, source_entity_id: int | None,
                task_type: int) -> torch.Tensor:
        """
        task_type: 0 = search, 1 = homepage recs, 2 = More Like This

        Missing context imputation:
        - Search: source_entity_id = null (no source entity)
        - Recs: query_text = null (no explicit query)
        - Similar: query_text = entity title text (imputed)
        """
        user_vec = self.user_embedding(user_id)
        item_vecs = self.item_embedding(item_ids)
        task_vec = self.task_embedding(task_type)

        # Impute missing contexts
        query_vec = (self.query_encoder(query_text)
                     if query_text
                     else torch.zeros_like(user_vec))
        source_vec = (self.item_embedding(source_entity_id)
                      if source_entity_id is not None
                      else torch.zeros_like(user_vec))

        # Shared representation
        combined = torch.cat([user_vec, item_vecs, query_vec, source_vec, task_vec],
                             dim=-1)
        hidden = self.shared_backbone(combined)

        # Task-specific scoring
        if task_type == 0:
            return self.search_head(hidden)
        elif task_type == 1:
            return self.recs_head(hidden)
        else:
            return self.similar_head(hidden)

# Results:
# +7% lift for search tasks
# +10% lift for recommendations tasks
# Fewer models to maintain, shared learnings across tasks

The critical detail: UniCoRn succeeded not by ignoring the distinction but by making the model explicitly aware of it. task_type tells the model which behavior to invoke. The imputation strategy respects the structural difference. The separate output heads optimize for different objectives because, at Netflix, a successful search is not the same thing as a successful recommendation.

Netflix also documented the central tension: personalization can overpower query relevance. If a user searches for "documentaries about World War II" and the recommender knows the user loves romantic comedies, should it show The Notebook? Obviously not. UniCoRn added personalization incrementally, with guardrails [8].



Open Questions

  1. UniCoRn achieved +7% for search and +10% for recommendations by making task identity explicit. What would the numbers be if they had collapsed the distinction entirely — one model, one objective, no task_type feature? Would it outperform the separate models at all?

  2. Netflix confirmed that scaling laws apply to recommendation foundation models. Is there a point where a large enough recommender, trained on enough user behavior, internalizes search as a special case — without being told?

  3. Percolate queries (reverse search) solve a problem that only exists when you think of search as infrastructure, not ranking. What other search primitives are we missing because we default to the ranked-list mental model?

  4. Hastings said Netflix competes with sleep. If recommendation is competing for attention and search is competing with ignorance, what happens when an LLM can do both? Does the system compete with everything?

References

  1. Netflix Technology Blog. Netflix Studio Search: Using Elasticsearch and Apache Flink to Index Federated GraphQL Data. March 2022.

  2. Netflix Technology Blog. Elasticsearch Indexing Strategy in Asset Management Platform (AMP). 2023.

  3. InfoQ. Netflix Uses Elasticsearch Percolate Queries to Implement Reverse Searches Efficiently. April 2024.

  4. Netflix Technology Blog. System Architectures for Personalization and Recommendation. March 2013.

  5. Harald Steck et al. Deep Learning for Recommender Systems: A Netflix Case Study. AI Magazine, 42(3): 7–18, 2021.

  6. Netflix Technology Blog. Foundation Model for Personalized Recommendation. March 2025.

  7. Moumita Bhattacharya et al. Joint Modeling of Search and Recommendations Via an Unified Contextual Recommender (UniCoRn). RecSys 2024.

  8. Sudarshan Lamkhede and Christoph Kofler. Recommendations and Results Organization in Netflix Search. RecSys 2021.

  9. Carlos A. Gomez-Uribe and Neil Hunt. The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM TMIS, 6(4), 2015.

  10. Xavier Amatriain and Justin Basilico. Netflix Recommendations: Beyond the 5 Stars. Netflix Technology Blog, 2012.

  11. Justin Basilico and Yves Raimond. Recent Trends in Personalization at Netflix. Netflix Technology Blog, 2020.


S&R: Why Is Guessing What People Want Harder Than Finding What They Asked For?

A technical history of recommender systems with working Python code for each major paradigm: user-based CF, item-based CF, matrix factorization (SVD++), two-tower neural models, and the Netflix Prize story.

searchrecommendationcollaborative-filteringmatrix-factorizationnetflix-prizedeep-learningseries

S&R stands for Search & Recommendation. This post is about recommendation — thirty years of trying to guess what people want before they know they want it. The title is not a rhetorical question. Guessing is genuinely harder.

Recommendation is the harder problem because the user never tells you what they want. You infer it from what they watched, clicked, rated, skipped, and abandoned. Every technique in this post — user-based CF, item-based CF, matrix factorization, two-tower neural models, autoregressive foundation models — is an attempt to extract intent from behavior the user may not even be conscious of. When the inference is wrong, it is always the system's fault. The user never said what they wanted. The system guessed. The guess was wrong.

Search is the easier problem because the user tells you. The query is explicit. The intent is stated. The system's job is fidelity. If the query is wrong — if the user typed "Byzantine Empire" when they meant "Ottoman Empire" — the results are wrong, but that is the user's problem, not the system's. The system's contract is to match the query, not to correct it. Search competes with ignorance. Recommendation competes with every other possible way to spend attention. The second competition is harder to win.

If search is about matching what the user says to what exists, recommendation is about guessing what the user wants before they say it — and, in the hardest cases, before they even know they want it. This is a fundamentally harder information problem. In search, the user tells you what they want and you try to find it. In recommendation, you infer what they want from behavior they may not even be conscious of.

Wu et al. (2023), in their comprehensive survey of neural recommendation models published in IEEE TKDE, organize the field into two broad families: models that use only interaction data (collaborative filtering) and models that incorporate side information (content, context, sequences) [1]. The progression from one to the other mirrors the search field's own evolution — from impoverished signals to rich, multi-modal representations. But the starting point is different. Search began with text and added behavior. Recommendation began with behavior and added text.

This part traces that evolution with working Python.

Tapestry — The First Collaborative Filtering System (1992)

In 1992, researchers at Xerox PARC built Tapestry, an email filtering system that let users annotate messages and write queries referencing others' annotations. They coined the term collaborative filtering. The insight was that relevance is social: if people with similar tastes found something useful, you probably will too.

# Tapestry's conceptual model: manual, query-based CF
# "Show me emails that Bob found interesting"

class TapestryFilter:
    def __init__(self):
        self.annotations: dict[str, dict[int, str]] = {}  # user -> {msg_id -> annotation}

    def annotate(self, user: str, msg_id: int, label: str):
        """Bob annotates a message as 'interesting' or 'boring'."""
        self.annotations.setdefault(user, {})[msg_id] = label

    def query(self, annotator: str, label: str) -> list[int]:
        """'Show me all messages Bob found interesting'."""
        return [msg_id for msg_id, ann in self.annotations.get(annotator, {}).items()
                if ann == label]

# Tapestry required users to write explicit queries. Powerful for power users,
# unusable for everyone else. The automation was coming.

GroupLens — Automating Collaborative Filtering at Scale (1994)

The GroupLens project at the University of Minnesota automated the process. Users rated Usenet articles (1–5 stars), and the system automatically predicted ratings for unread articles based on similar users' ratings.

import numpy as np
from collections import defaultdict

def user_based_cf(ratings: dict[int, dict[int, float]],
                  user_id: int, item_id: int, k: int = 50) -> float:
    """
    User-based collaborative filtering: predict a user's rating for an item
    based on how similar users rated it.

    ratings: {user_id: {item_id: rating}}
    """
    if user_id not in ratings:
        return 3.0  # global mean for cold start

    # Step 1: Find users who rated this item
    co_raters = [(other, ratings[other][item_id])
                 for other in ratings
                 if item_id in ratings[other] and other != user_id]

    if not co_raters:
        return np.mean(list(ratings[user_id].values()))  # user's mean rating

    # Step 2: Compute similarity between target user and each co-rater
    similarities = []
    for other, _ in co_raters:
        sim = pearson_similarity(ratings[user_id], ratings[other])
        similarities.append((other, sim))

    similarities.sort(key=lambda x: x[1], reverse=True)
    neighbors = similarities[:k]

    # Step 3: Weighted average of neighbors' ratings
    weighted_sum = sum(sim * ratings[neighbor][item_id] for neighbor, sim in neighbors
                       if sim > 0)
    norm = sum(abs(sim) for _, sim in neighbors if sim > 0)

    return weighted_sum / norm if norm > 0 else np.mean(list(ratings[user_id].values()))


def pearson_similarity(user_a: dict[int, float], user_b: dict[int, float]) -> float:
    """Pearson correlation between two users' rating vectors."""
    common_items = set(user_a) & set(user_b)
    if len(common_items) < 3:
        return 0.0  # not enough overlap

    mean_a = np.mean([user_a[i] for i in common_items])
    mean_b = np.mean([user_b[i] for i in common_items])

    num = sum((user_a[i] - mean_a) * (user_b[i] - mean_b) for i in common_items)
    den_a = np.sqrt(sum((user_a[i] - mean_a) ** 2 for i in common_items))
    den_b = np.sqrt(sum((user_b[i] - mean_b) ** 2 for i in common_items))

    return num / (den_a * den_b) if den_a and den_b else 0.0

GroupLens spawned Net Perceptions, a company that served Amazon, CDnow, and others. In 2010, the team won the ACM Software System Award. But user-based CF had a scaling problem: computing user-user similarity is O(N²) in the number of users.

Amazon Item-to-Item CF — Scaling to Millions (2003)

Amazon's Greg Linden, Brent Smith, and Jeremy York flipped the problem. Instead of finding similar users, find similar items:

def item_based_cf(ratings: dict[int, dict[int, float]],
                  user_id: int, k: int = 10) -> list[tuple[int, float]]:
    """
    Item-based collaborative filtering: recommend items similar to what
    the user already liked.

    The expensive part — computing the item-item similarity matrix —
    runs offline, not at serving time.
    """
    # Build item-item similarity matrix (OFFLINE — runs daily)
    item_sim = build_item_similarity_matrix(ratings)

    # Online: look up items similar to what the user liked (MILLISECONDS)
    user_ratings = ratings.get(user_id, {})
    liked = [(item, r) for item, r in user_ratings.items() if r > 3.5]

    candidates: dict[int, float] = defaultdict(float)
    total_weight: dict[int, float] = defaultdict(float)

    for item, rating in liked:
        for similar_item, sim in item_sim.get(item, {}).items():
            if similar_item not in user_ratings:  # don't recommend what they've rated
                candidates[similar_item] += sim * rating
                total_weight[similar_item] += abs(sim)

    # Normalize
    scored = [(item, candidates[item] / total_weight[item])
              for item in candidates if total_weight[item] > 0]
    scored.sort(key=lambda x: x[1], reverse=True)
    return scored[:k]


def build_item_similarity_matrix(ratings: dict[int, dict[int, float]]
                                 ) -> dict[int, dict[int, float]]:
    """O(N_items² × N_users) — run offline, results cached."""
    # Transpose: item -> {user: rating}
    item_users: dict[int, dict[int, float]] = defaultdict(dict)
    for user_id, user_ratings in ratings.items():
        for item_id, rating in user_ratings.items():
            item_users[item_id][user_id] = rating

    items = list(item_users.keys())
    sim_matrix: dict[int, dict[int, float]] = defaultdict(dict)

    for i, item_a in enumerate(items):
        for item_b in items[i + 1:]:
            common = set(item_users[item_a]) & set(item_users[item_b])
            if len(common) < 5:
                continue
            sim = cosine_similarity(
                {u: item_users[item_a][u] for u in common},
                {u: item_users[item_b][u] for u in common}
            )
            if sim > 0:
                sim_matrix[item_a][item_b] = sim
                sim_matrix[item_b][item_a] = sim

    return sim_matrix

Greg Linden later explained why this beat search-based approaches: "Rather than matching the user to similar customers, item-to-item collaborative filtering matches each of the user's purchased and rated items to similar items." Search-based methods — constructing queries from purchase history to find items with similar keywords — produced recommendations that were either too general (bestsellers) or too narrow (more books by the same author). Collaborative filtering discovered cross-category connections: people who bought Into Thin Air also bought The Perfect Storm. No keyword match connects a mountaineering disaster to a fishing boat tragedy. Only behavior does.

The Netflix Prize — Matrix Factorization Takes Over (2006–2009)

In 2006, Netflix offered $1M to any team that could improve their recommendation algorithm by 10%. The winning approach reshaped the field: matrix factorization.

class MatrixFactorization:
    """Learn user and item latent factors via stochastic gradient descent."""

    def __init__(self, num_users: int, num_items: int, k: int = 50,
                 lr: float = 0.01, lambda_reg: float = 0.02):
        # Initialize latent factor matrices randomly
        self.P = np.random.normal(0, 0.1, (num_users, k))  # user factors
        self.Q = np.random.normal(0, 0.1, (num_items, k))  # item factors
        self.bu = np.zeros(num_users)  # user biases
        self.bi = np.zeros(num_items)  # item biases
        self.mu = 0.0                  # global mean
        self.lr = lr
        self.lambda_reg = lambda_reg

    def fit(self, ratings: list[tuple[int, int, float]], epochs: int = 100):
        """Train via SGD."""
        self.mu = np.mean([r for _, _, r in ratings])

        for epoch in range(epochs):
            np.random.shuffle(ratings)
            total_loss = 0.0

            for u, i, r in ratings:
                # Prediction: global mean + user bias + item bias + latent interaction
                pred = self.mu + self.bu[u] + self.bi[i] + np.dot(self.P[u], self.Q[i])
                error = r - pred
                total_loss += error ** 2

                # SGD updates with L2 regularization
                self.bu[u] += self.lr * (error - self.lambda_reg * self.bu[u])
                self.bi[i] += self.lr * (error - self.lambda_reg * self.bi[i])

                # Update latent factors
                pu_old = self.P[u].copy()
                self.P[u] += self.lr * (error * self.Q[i] - self.lambda_reg * self.P[u])
                self.Q[i] += self.lr * (error * pu_old - self.lambda_reg * self.Q[i])

            if epoch % 20 == 0:
                rmse = np.sqrt(total_loss / len(ratings))
                print(f"Epoch {epoch}: RMSE = {rmse:.4f}")

    def predict(self, u: int, i: int) -> float:
        return self.mu + self.bu[u] + self.bi[i] + np.dot(self.P[u], self.Q[i])

    def recommend(self, u: int, rated_items: set[int], k: int = 10) -> list[int]:
        """Generate top-k recommendations for user u."""
        scores = [(i, self.predict(u, i))
                  for i in range(len(self.Q))
                  if i not in rated_items]
        scores.sort(key=lambda x: x[1], reverse=True)
        return [item for item, _ in scores[:k]]

# Usage with MovieLens-100K style data
# ratings = [(user, item, rating), ...]
# mf = MatrixFactorization(num_users=943, num_items=1682, k=50)
# mf.fit(ratings, epochs=100)
# recommendations = mf.recommend(user_id=42, rated_items={item for _, item, _ in ratings if _ == 42})

Yehuda Koren's SVD++ extended this by incorporating implicit feedback — what you browsed, not just what you rated — and achieved the winning RMSE of 0.8556. The Netflix Prize established matrix factorization as the dominant paradigm for nearly a decade.

Rendle's Factorization Machines generalized the matrix factorization idea to arbitrary feature vectors, modeling all pairwise interactions through a factorized parametrization — the same mechanism that powers MF but applicable to any set of sparse categorical features [8].

It also revealed the field's central tension: the metric that drove the competition (RMSE on withheld ratings) doesn't actually measure whether users are satisfied. Reed Hastings articulated this later: "When we rate, we're meta-cognitive about quality — that's sort of our aspirational self. It works out much better, to please people, to look at the actual choices that they make." Users say they want documentaries; they watch reality TV. A recommender that trusts stated preferences over revealed preferences fails.

The Deep Learning Era — YouTube and the Multi-Stage Pipeline (2016–2020)

YouTube's 2016 paper marked deep learning's entry into production recommendation. The architecture was two-stage:

He et al.'s Neural Collaborative Filtering (NCF) showed that replacing the inner product in matrix factorization with a learned multi-layer perceptron could capture non-linear user–item interactions, unifying collaborative filtering with deep learning [7].

class YouTubeCandidateGenerator(torch.nn.Module):
    """YouTube-style candidate generation: narrow millions → hundreds."""

    def __init__(self, vocab_sizes: dict[str, int], embedding_dim: int = 256):
        super().__init__()
        # Embedding layers for categorical features
        self.embeddings = torch.nn.ModuleDict({
            name: torch.nn.Embedding(size, embedding_dim)
            for name, size in vocab_sizes.items()
        })
        # Concatenated embeddings → hidden layers
        input_dim = len(vocab_sizes) * embedding_dim
        self.layers = torch.nn.Sequential(
            torch.nn.Linear(input_dim, 1024),
            torch.nn.ReLU(),
            torch.nn.Linear(1024, 512),
            torch.nn.ReLU(),
            torch.nn.Linear(512, 256),
        )
        # Output: softmax over all video IDs (treated as classes)
        self.output = torch.nn.Linear(256, vocab_sizes['video_id'])

    def forward(self, features: dict[str, torch.Tensor]) -> torch.Tensor:
        """Predict the next video a user will watch."""
        embedded = [self.embeddings[name](features[name])
                    for name in self.embeddings]
        concat = torch.cat(embedded, dim=1)
        hidden = self.layers(concat)
        return self.output(hidden)

# Key innovation: the model was trained on ALL YouTube watches, including those
# on embedded players. The training objective was next-video prediction —
# treating recommendation as an extreme multi-class classification problem.

# But the real insight was about WHAT to optimize. YouTube didn't optimize for
# clicks — they optimized for expected watch time. Clicks are easy to game
# (clickbait thumbnails). Watch time is harder to fake and better aligned
# with user satisfaction.

The three-stage pipeline that emerged became industry standard:

def production_recsys_pipeline(user_id: int, user_features: dict,
                               all_items: list[int], k: int = 10) -> list[int]:
    """Standard three-stage recommendation pipeline."""

    # Stage 1: Candidate Generation — millions → thousands
    # Multiple parallel generators: collaborative filtering, trending,
    # new releases, content-based. High recall, low cost per item.
    candidates_cf = cf_retriever.retrieve(user_id, k=500)
    candidates_trending = trending_retriever.retrieve(k=200)
    candidates_similar = similar_items_retriever.retrieve(
        user_features['last_watched'], k=200)
    candidates = list(set(candidates_cf + candidates_trending + candidates_similar))

    # Stage 2: Ranking — thousands → hundreds
    # Deep neural network scores each candidate using hundreds of features.
    # High precision, moderate cost per item.
    ranked = ranker.score(user_id, candidates, user_features)
    ranked.sort(key=lambda x: x.score, reverse=True)
    top = ranked[:200]

    # Stage 3: Re-Ranking — hundreds → tens
    # Apply diversity, freshness boost, business rules, exploration.
    # Low cost per item (post-processing), high impact on user experience.
    final = re_ranker.apply(top, user_features,
                            diversity_factor=0.3,
                            freshness_boost=1.2,
                            max_same_genre=3)
    return [item.id for item in final[:k]]

By 2020, the search field had always been multi-stage. Recommendation caught up — and that convergence is one reason the two fields are so often conflated.



Open Questions

  1. Matrix factorization dominated recommendation for a decade after the Netflix Prize, but the Prize's metric (RMSE) didn't measure user satisfaction. What would a recommendation competition look like today if the metric were retention, not rating prediction? Could we even run one?

  2. Hastings' insight — that users' stated preferences differ from their revealed preferences — has uncomfortable implications. Should recommenders ever ignore what users explicitly tell them? When is the aspirational self a feature rather than noise?

  3. The deep learning era equalized recommendation and search by giving both access to heterogeneous features. But search had decades of multi-modal infrastructure first. Did recommendation catch up, or did it just inherit search's architecture without adapting it?

  4. Item-to-item CF at Amazon discovered cross-category connections no keyword match could find. What connections are today's models missing because they optimize for engagement rather than surprise?

References

  1. Le Wu, Xiangnan He, Xiang Wang, Kun Zhang, and Meng Wang. A Survey on Accuracy-Oriented Neural Recommendation: From Collaborative Filtering to Information-Rich Recommendation. IEEE TKDE, 35(5): 4425–4445, 2023.

  2. David Goldberg, David Nichols, Brian M. Oki, and Douglas Terry. Using Collaborative Filtering to Weave an Information Tapestry. Communications of the ACM, 35(12): 61–70, 1992.

  3. Paul Resnick et al. GroupLens: An Open Architecture for Collaborative Filtering of Netnews. CSCW 1994.

  4. Greg Linden, Brent Smith, and Jeremy York. Amazon.com Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Computing, 7(1): 76–80, 2003.

  5. Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix Factorization Techniques for Recommender Systems. IEEE Computer, 42(8): 30–37, 2009.

  6. Paul Covington, Jay Adams, and Emre Sargin. Deep Neural Networks for YouTube Recommendations. RecSys 2016.

  7. Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural Collaborative Filtering. WWW 2017.

  8. Steffen Rendle. Factorization Machines. ICDM 2010.

  9. James Bennett and Stan Lanning. The Netflix Prize. KDD Cup Workshop, 2007.

  10. Gediminas Adomavicius and Alexander Tuzhilin. Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions. IEEE TKDE, 17(6): 734–749, 2005.


S&R: What Can Fifty Years of Search Technology Teach Us About Finding Things?

A technical history of information retrieval with working Python code for each major paradigm: Boolean retrieval, TF-IDF, BM25, PageRank, Learning to Rank, and neural IR with embeddings.

searchinformation-retrievaltf-idfbm25pagerankbertneural-irseries

S&R stands for Search & Recommendation. This post is about search — fifty years of it, from Boolean to BERT. But you cannot understand what search is without understanding what it is not.

Search begins with a query. The user types words. Those words are a contract: "I am looking for this. Find me the best matches." Every technique in this post — TF-IDF, BM25, PageRank, Learning to Rank, DPR — is an attempt to narrow the gap between the words the user typed and the documents they meant. The gap is irreducible because language is ambiguous and intent is underspecified, but fifty years of work has made it narrower.

Recommendation begins with silence. No query. No typed words. No explicit contract. The only signal is behavior — what the user clicked, watched, lingered on, skipped, abandoned. The system infers a query the user never wrote. The gap here is not lexical. It is psychological. The user may not know what they want. The system's job is to know it anyway.

Search answers one question: given a query and a collection of documents, which documents are most relevant, and in what order? Every major advance in information retrieval has come from realizing the previous generation's answer was incomplete — not wrong, just missing a dimension of what "relevance" means.

This part traces fifty years of that evolution with working Python. You'll see how each generation built on the last, what each solved, and what each left unsolved.

Boolean Retrieval — Exact Matching and Its Limits (1960s–1970s)

The earliest computerized search systems used Boolean logic: a query was a logical expression of terms, and documents either matched or did not. No ranking — just set intersection.

from collections import defaultdict
from typing import list[str]

class BooleanIndex:
    """The simplest retrieval engine: exact match, no ranking."""

    def __init__(self):
        self.index: dict[str, set[int]] = defaultdict(set)
        self.documents: dict[int, str] = {}

    def add(self, doc_id: int, text: str):
        self.documents[doc_id] = text
        for token in set(text.lower().split()):
            self.index[token].add(doc_id)

    def search_and(self, terms: list[str]) -> set[int]:
        """Boolean AND: all terms must appear."""
        result = None
        for term in terms:
            docs = self.index.get(term.lower(), set())
            result = docs if result is None else result & docs
        return result or set()

    def search_or(self, terms: list[str]) -> set[int]:
        """Boolean OR: any term can appear."""
        result = set()
        for term in terms:
            result |= self.index.get(term.lower(), set())
        return result

# Usage
idx = BooleanIndex()
idx.add(1, "the tragedy of star crossed lovers in Verona")
idx.add(2, "a tragic love story set in medieval Italy")

# The vocabulary mismatch problem in action
assert idx.search_and(["tragic", "love", "story"]) == {2}   # finds doc 2
assert idx.search_and(["star", "crossed", "lovers"]) == {1} # finds doc 1
# But: no document contains BOTH phrasings, so they never appear together.
# Searching for "tragic love story" misses Shakespeare entirely.

This worked for trained librarians. It failed for everyone else. The vocabulary mismatch problem — users describe their needs with different words than authors use — is the fundamental condition of language, not a bug to be fixed. Every subsequent generation of retrieval technology is an attempt to narrow the gap.

Karen Spärck Jones, who invented inverse document frequency (the weighting scheme that would become half of TF-IDF) [10], understood this better than anyone. In a 1999 reflection, she wrote: "Classical document retrieval thus falls in the class of AI tasks that assist the human user but cannot, by definition, replace them." The gap can be narrowed. It cannot be closed.

The Vector Space Model — When Documents Became Points in Space (1975)

Gerard Salton at Cornell proposed representing documents and queries as sparse vectors in a high-dimensional term space. A document was no longer a set of words — it was a point whose coordinates were term weights.

The weighting scheme was TF-IDF:

import math
from collections import Counter

def compute_tf_idf(documents: list[list[str]]) -> dict[int, dict[str, float]]:
    """Compute TF-IDF vectors for a document collection."""
    N = len(documents)
    df: dict[str, int] = Counter()

    # Count document frequency for each term
    for doc_tokens in documents:
        for term in set(doc_tokens):
            df[term] += 1

    tfidf_vectors: dict[int, dict[str, float]] = {}
    for i, doc_tokens in enumerate(documents):
        tf = Counter(doc_tokens)
        doc_len = len(doc_tokens)
        tfidf_vectors[i] = {}
        for term, count in tf.items():
            # TF: normalized term frequency
            tf_norm = count / doc_len
            # IDF: log(N / df) — rare terms get higher weight
            idf = math.log((N - df[term] + 0.5) / (df[term] + 0.5) + 1.0)
            tfidf_vectors[i][term] = tf_norm * idf

    return tfidf_vectors

def cosine_similarity(vec_a: dict[str, float], vec_b: dict[str, float]) -> float:
    """Cosine similarity between two sparse vectors."""
    common_terms = set(vec_a) & set(vec_b)
    if not common_terms:
        return 0.0

    dot = sum(vec_a[t] * vec_b[t] for t in common_terms)
    norm_a = math.sqrt(sum(v ** 2 for v in vec_a.values()))
    norm_b = math.sqrt(sum(v ** 2 for v in vec_b.values()))
    return dot / (norm_a * norm_b) if norm_a and norm_b else 0.0

def search_tfidf(query: list[str], tfidf_vectors: dict[int, dict[str, float]],
                 k: int = 10) -> list[tuple[int, float]]:
    """Search using TF-IDF cosine similarity."""
    # Build query vector (TF only — no IDF for ad-hoc queries in simplest form)
    query_tf = Counter(query)
    query_len = len(query)
    query_vec = {t: c / query_len for t, c in query_tf.items()}

    scores = [(doc_id, cosine_similarity(query_vec, doc_vec))
              for doc_id, doc_vec in tfidf_vectors.items()]
    scores.sort(key=lambda x: x[1], reverse=True)
    return scores[:k]

# Usage
docs = [
    "the tragedy of star crossed lovers in Verona".split(),
    "a tragic love story set in medieval Italy".split(),
    "machine learning algorithms for classification tasks".split(),
]
tfidf = compute_tf_idf(docs)
# Now "tragic love story" finds BOTH documents — TF-IDF bridges the vocabulary gap
# through shared high-IDF terms
results = search_tfidf("tragic love story".split(), tfidf)

TF-IDF doesn't understand semantics. It exploits a statistical regularity: rare terms are more discriminating, and frequent local terms are more important. It's a heuristic — but an extraordinarily robust one that remains a strong baseline fifty years later.

BM25 — The Probabilistic Framework That Still Powers Production (1970s–1990s)

Stephen Robertson, Karen Spärck Jones, and collaborators at City, University of London asked a different question: given a document and a query, what is the probability that the document is relevant? The answer was BM25.

def bm25_score(query_terms: list[str], doc_tokens: list[str],
               doc_lengths: list[int], avg_dl: float,
               df: dict[str, int], N: int,
               k1: float = 1.5, b: float = 0.75) -> float:
    """
    BM25 scoring function.

    k1: controls term frequency saturation (higher = more linear)
    b:  controls document length normalization (0 = none, 1 = full)
    """
    dl = len(doc_tokens)
    tf = Counter(doc_tokens)
    score = 0.0

    for term in set(query_terms):
        if term not in df:
            continue

        # IDF component (Robertson-Spärck Jones formulation)
        idf = math.log((N - df[term] + 0.5) / (df[term] + 0.5) + 1.0)

        # TF with saturation: as count grows, additional occurrences matter less
        f = tf.get(term, 0)
        tf_saturated = (f * (k1 + 1)) / (f + k1 * (1 - b + b * dl / avg_dl))

        score += idf * tf_saturated

    return score

# BM25's key innovations over TF-IDF:
# 1. Diminishing returns for term frequency (a term appearing 10x is not 10x as
#    important as appearing 1x — the saturation curve handles this)
# 2. Document length normalization that's tunable (longer docs aren't inherently
#    more relevant just because they have more words)

# BM25 remains the default scoring in Elasticsearch and Lucene — which means it
# powers production search at Netflix, Spotify, DoorDash, and most of the web.

BM25's IDF is derived from the Binary Independence Model — it has a probabilistic foundation that TF-IDF lacks. It is not obsolete in 2026. In modern hybrid systems, BM25 is the fast, cheap first stage that feeds candidates to neural re-rankers.

Content-based retrieval treats documents independently. But on the web, documents link to each other — and those links carry information. Brin and Page's insight at Stanford was that the link graph is itself a relevance signal.

import numpy as np

def pagerank(adjacency: dict[int, list[int]], N: int,
             d: float = 0.85, max_iter: int = 100, tol: float = 1e-6) -> dict[int, float]:
    """
    Compute PageRank scores from a link graph.

    adjacency: {page_id: [outgoing_link_page_ids]}
    d: damping factor — probability the random surfer continues clicking
    """
    pr = {p: 1.0 / N for p in adjacency}  # uniform initialization

    for iteration in range(max_iter):
        new_pr = {}
        for page in adjacency:
            # Random jump component: (1-d)/N — every page gets baseline probability
            # Link component: sum of PR from inbound links, divided by their out-degree
            inbound_sum = 0.0
            for other_page, out_links in adjacency.items():
                if page in out_links:
                    inbound_sum += pr[other_page] / len(out_links)
            new_pr[page] = (1 - d) / N + d * inbound_sum

        # Check convergence
        delta = sum(abs(new_pr[p] - pr[p]) for p in pr)
        if delta < tol:
            break
        pr = new_pr

    return pr

# Usage: a tiny web of 4 pages
web = {
    0: [1, 2],       # page 0 links to 1 and 2
    1: [2],          # page 1 links to 2
    2: [0, 3],       # page 2 links to 0 and 3
    3: [2],          # page 3 links to 2
}
scores = pagerank(web, N=4)
# Page 2 gets the highest score: it has the most inbound links from important pages

PageRank didn't replace content-based retrieval — it augmented it. A modern search engine computes hundreds of features (BM25, PageRank, proximity, freshness, click-through rate, spam score) and feeds them into a learned ranking function.

Learning to Rank — When Ranking Became a Supervised ML Problem (2000s–2010s)

The insight: ranking is just a machine learning problem. Given a query, a set of candidate documents with relevance labels, and hundreds of features, learn a function that orders them optimally.

from sklearn.ensemble import GradientBoostingRegressor
import numpy as np

def train_pointwise_ltr(training_data: list[tuple[np.ndarray, float]]) -> GradientBoostingRegressor:
    """
    Pointwise LTR: predict relevance score for each (query, document) pair.

    Features might include: BM25, PageRank, click rate, freshness, title match, etc.
    """
    X = np.array([features for features, _ in training_data])
    y = np.array([score for _, score in training_data])
    model = GradientBoostingRegressor(n_estimators=500, max_depth=5)
    model.fit(X, y)
    return model

def rank_candidates(model, candidates: list[tuple[int, np.ndarray]]) -> list[int]:
    """Score and sort candidates using the learned model."""
    scored = [(doc_id, model.predict(features.reshape(1, -1))[0])
              for doc_id, features in candidates]
    scored.sort(key=lambda x: x[1], reverse=True)
    return [doc_id for doc_id, _ in scored]

# In practice, LambdaMART — gradient-boosted trees with a listwise LambdaRank
# objective — became the industry standard. The key difference from pointwise:
# it optimizes the ordering directly (NDCG), not individual relevance scores.

# Real search engines at Microsoft, Yahoo, and Google used ~500-1000 features.
# The hand-crafted feature era produced the best search quality we'd ever seen —
# but it was also a maintenance nightmare. Every new signal needed a new feature.

The Neural Turn — From Word2Vec to BERT to Dense Retrieval (2013–2020)

Three papers changed search again. Let's see what each one did to the retrieval pipeline.

Word2Vec (2013): Dense word vectors that capture semantic relationships.

# Conceptual: Word2Vec trains on the task "predict surrounding words"
# The learned vectors capture analogies: king - man + woman ≈ queen
# In retrieval: query and document terms can be matched even when they
# don't share exact words, by comparing their vector representations.

def embed_query(query: str, word_vectors: dict[str, np.ndarray]) -> np.ndarray:
    """Simple averaging of word vectors — the simplest dense query representation."""
    tokens = query.lower().split()
    vectors = [word_vectors[t] for t in tokens if t in word_vectors]
    return np.mean(vectors, axis=0) if vectors else np.zeros(300)

# But: word vectors are context-independent. "bank" has the same vector
# in "river bank" and "investment bank." That's the problem BERT solved.

BERT (2018): Deeply contextualized embeddings. A word's representation depends on the words around it.

# With BERT, "bank" in "river bank" and "investment bank" have different vectors.
# This transformed query–document matching.

# In retrieval, BERT is typically used as a RERANKER, not a first-stage retriever:
# 1. BM25 retrieves top-1000 candidates (fast, cheap)
# 2. BERT cross-encoder scores each (query, candidate) pair (slow, expensive, accurate)

# The cross-encoder concatenates query and document, passes them through BERT
# jointly, and produces a single relevance score. This captures fine-grained
# interactions but is too slow to run over the entire collection.

Dense Passage Retrieval (DPR) (2020): Bi-encoder architecture that makes neural retrieval fast enough for first-stage retrieval.

import torch
import torch.nn.functional as F

class DPRBiEncoder(torch.nn.Module):
    """Dense Passage Retrieval: separate encoders for queries and passages."""
    def __init__(self, query_encoder, passage_encoder):
        super().__init__()
        self.query_encoder = query_encoder    # e.g., BERT-base
        self.passage_encoder = passage_encoder # e.g., BERT-base

    def encode_query(self, query_texts: list[str]) -> torch.Tensor:
        """Encode queries into dense vectors."""
        return self.query_encoder(query_texts)  # shape: (batch, 768)

    def encode_passages(self, passage_texts: list[str]) -> torch.Tensor:
        """Encode passages into dense vectors (can be pre-computed offline)."""
        return self.passage_encoder(passage_texts)  # shape: (batch, 768)

    def retrieve(self, query: torch.Tensor, passage_embeddings: torch.Tensor,
                 k: int = 10) -> tuple[torch.Tensor, torch.Tensor]:
        """ANN retrieval using dot product similarity."""
        scores = torch.matmul(query, passage_embeddings.T)  # (1, num_passages)
        top_scores, top_indices = torch.topk(scores, k=k)
        return top_indices, top_scores

# Key property: passage embeddings are pre-computed and indexed in FAISS.
# At query time, only the query encoder runs. This makes neural first-stage
# retrieval feasible at scale.

Guo et al.'s comprehensive survey A Deep Look into Neural Ranking Models for Information Retrieval (2020) catalogs this transition from hand-crafted features to learned representations, noting that the key shift was not just better accuracy — it was the elimination of feature engineering as the bottleneck in search quality improvement [1].

The Modern Stack: Hybrid All the Way Down

No production search system in 2026 uses a single model. The standard architecture is a multi-stage cascade:

def production_search_pipeline(query: str, k: int = 10) -> list[Document]:
    """Multi-stage search: each stage is more expensive but operates on fewer candidates."""

    # Stage 1: Lexical retrieval (BM25 via inverted index)
    # Cost: O(query_terms) — sub-millisecond
    # Coverage: full corpus (millions of documents)
    # Recall: high; Precision: low
    candidates_bm25 = bm25_retrieve(query, top_k=1000)

    # Stage 2: Dense retrieval (bi-encoder + ANN)
    # Cost: O(log N) with FAISS — ~10ms
    # Coverage: candidates from stage 1 re-ranked by embedding similarity
    # Recall: high; Precision: moderate
    query_embedding = query_encoder.encode(query)
    candidates_dense = faiss_index.search(query_embedding, k=200)

    # Stage 3: Cross-encoder re-ranking (BERT)
    # Cost: O(candidates) with full transformer — ~100ms for 200 candidates
    # Coverage: top-200 from dense retrieval
    # Recall: moderate; Precision: high
    merged = merge_and_deduplicate(candidates_bm25, candidates_dense)
    scored = []
    for doc in merged[:200]:
        score = cross_encoder.score(query, doc.text)
        scored.append((doc, score))
    scored.sort(key=lambda x: x[1], reverse=True)

    return [doc for doc, _ in scored[:k]]

# Each stage compensates for the limitations of the one before it.
# BM25 handles exact matches that confuse embeddings (rare names, IDs, codes).
# Dense retrieval handles semantic matches that BM25 misses (synonyms, paraphrases).
# Cross-encoder handles fine-grained relevance that dot products miss.

The comprehensive survey by Hambarde and Proença (2023) organizes this pipeline into two stages — term-based retrieval and semantic retrieval — and catalogs the models available at each level [2]. Their key insight: modern search is never one model. It's a pipeline where each stage compensates for the limitations of the one before it.

Manning, Raghavan, and Schütze's textbook remains the canonical reference for the information retrieval fundamentals — inverted indexes, scoring functions, and evaluation methodology — that underpin every stage of this pipeline [7].

This is the critical difference from recommendation. Search has always been multi-stage. Recommendation was historically single-stage — and the move to multi-stage pipelines in recommendation was one of the key convergences between the two fields.



Open Questions

  1. BM25 has survived fifty years and still anchors production search pipelines. What properties make a retrieval model durable across paradigm shifts? Will transformer-based retrieval have the same half-life?

  2. Multi-stage pipelines (BM25 → dense → cross-encoder) are engineering compromises, not elegant solutions. What would a single-stage retrieval architecture look like — and what would it cost?

  3. Learning to Rank replaced hundreds of hand-crafted features with learned combinations. But feature engineering is returning — this time as prompt engineering for LLM rerankers. Are we going in circles, or is this a spiral?

  4. The vocabulary mismatch problem that Spärck Jones identified is fundamental to language, not a bug to fix. If the gap can only be narrowed, never closed, what is the theoretical ceiling on retrieval quality — and how close are we?

References

  1. Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, W. Bruce Croft, et al. A Deep Look into Neural Ranking Models for Information Retrieval. Information Processing & Management, 57(6), 2020.

  2. Kailash Hambarde and Hugo Proença. Information Retrieval: Recent Advances and Beyond. arXiv:2301.08801, 2023.

  3. Gerard Salton, Anita Wong, and Chung-Shu Yang. A Vector Space Model for Automatic Indexing. Communications of the ACM, 18(11): 613–620, 1975.

  4. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-3. Proceedings of TREC-3, 1994.

  5. Sergey Brin and Lawrence Page. The Anatomy of a Large-Scale Hypertextual Web Search Engine. Computer Networks and ISDN Systems, 30(1–7): 107–117, 1998.

  6. Christopher J.C. Burges. From RankNet to LambdaRank to LambdaMART: An Overview. Microsoft Research Technical Report MSR-TR-2010-82, 2010.

  7. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.

  8. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.

  9. Amit Singhal. Modern Information Retrieval: A Brief Overview. IEEE Data Engineering Bulletin, 24(4): 35–43, 2001.

  10. Karen Spärck Jones. A Statistical Interpretation of Term Specificity and Its Application in Retrieval. Journal of Documentation, 28(1): 11–21, 1972. The original IDF paper.


S&R: Why Do We Keep Confusing Search with Recommendation?

Why search and recommendation are different problems — and why conflating them is a costly engineering mistake. Covers the three traps, the user-posture divide, Peter Norvig's 80% rule, and the library/restaurant mental models that clarify the boundary.

searchrecommendationinformation-retrievalcollaborative-filteringarchitectureseries

S&R stands for Search & Recommendation — the two fundamental modes of information access. This post defines the distinction that the rest of the series builds on, and explains why conflating them is not merely a terminology error but an engineering one with production consequences.

Search is what happens when someone types "history of the Byzantine Empire" into a box and hits enter. They know what they want. They told you — explicitly, in words, 500 milliseconds ago. Your job is fidelity to those words. If they typed "Byzantine" and you returned results about the Ottoman Empire because "people who search for Byzantium also search for Ottomans," you have broken the contract.

Recommendation is what happens when someone opens an app and waits. They didn't type anything. They may not know what they want. They may not even know there is something to want. Your job is to infer it from what they did yesterday, last week, and five minutes ago — and to be right often enough that they keep opening the app. If you show them what they already know about, you have broken a different contract.

Search and recommendation are often described as "two sides of the same coin." Both match users with items. Both rank results. Both drive discovery. The phrase appears in conference papers and engineering blog posts alike.

It is also wrong — or at least, incomplete enough to be dangerous.

Search and recommendation are different problems. They emerged from different research communities, solved different user needs, developed different mathematical frameworks, and measure success differently. Confusing them produces systems that are bad at both: search results that drift into irrelevance under the weight of personalization, and recommendation feeds that fail to respect explicit intent.

This series traces the history, the mathematics, the architectures, and the convergence — arguing that the best engineering organizations don't collapse search and recommendation into one. They build systems that respect the distinction even as they blur it.

Let's start with the definitions.

What Search Is — And What It Is Not

Search is query-driven retrieval. A user has an information need, formulates it as a query, and expects the system to return items relevant to that query. The contract is: I tell you what I want. You find it.

# Search: the user specifies intent explicitly
def search(query: str, index: InvertedIndex, k: int = 10) -> list[Document]:
    """Return the top-k documents matching the query."""
    query_tokens = tokenize(query)
    candidates = set()
    for token in query_tokens:
        candidates.update(index.postings[token])  # exact or fuzzy match

    scored = [(doc, score(query_tokens, doc)) for doc in candidates]
    scored.sort(key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in scored[:k]]

# The key: query comes from the user, right now.
# If the query is wrong, the results are wrong — but that's the user's problem,
# not the system's. The system's job is fidelity to the query.
results = search("history of the Byzantine Empire", index, k=10)

The critical property: the user actively formulates their need. They lean forward. They have a goal. Search is a tool the user operates.

What Recommendation Is — And What It Is Not

Recommendation is preference-driven filtering. The system infers what a user might want from their history, behavior, and context, then surfaces items unprompted. The contract is: I know something about you. Let me suggest what you might like.

# Recommendation: the system infers intent from behavior
def recommend(user_id: str, item_sim: dict, ratings: dict, k: int = 10) -> list[Item]:
    """Return top-k items the user might like, based on what they liked before."""
    liked_items = [item for item, rating in ratings[user_id].items() if rating > 3.5]

    candidates = {}
    for liked in liked_items:
        for similar_item, sim_score in item_sim[liked].items():
            if similar_item not in ratings[user_id]:  # don't recommend what they've seen
                candidates[similar_item] = candidates.get(similar_item, 0) + sim_score

    ranked = sorted(candidates.items(), key=lambda x: x[1], reverse=True)
    return [item for item, _ in ranked[:k]]

# The key: the user did NOT ask for anything. The system guessed.
# If the guess is wrong, that's the system's failure.
suggestions = recommend("user_42", item_similarity_matrix, ratings_matrix, k=10)

The critical property: the system initiates. The user leans back. They are open to suggestion. Recommendation is an experience the user receives.

The Difference in One Table

Dimension Search Recommendation
User intent Active: user formulates a query Passive: system surfaces items unprompted
Input Short, explicit query text Implicit user profile (history, behavior, context)
Core operation Query–document matching User–item matching
Information need Known: "I want to find X" Unknown: "Show me what I might like"
Evaluation Precision, recall, NDCG, MRR RMSE, AUC, CTR, retention, discovery
Serendipity Undesirable (should return what was asked for) Desirable (should surface unexpected gems)
Personalization Optional; applied sparingly at re-ranking Core to the entire pipeline
Theoretical roots Information Retrieval (library science, linguistics) Collaborative Filtering (HCI, ML)
User posture Lean forward — goal-directed Lean back — open to suggestion

Manning, Raghavan, and Schütze's canonical IR textbook formalizes this distinction in its opening chapter: information retrieval is query-driven; information filtering is profile-driven [5].

Why People Confuse Them — The Three Traps

If the distinction is so clear, why do smart engineers keep conflating them? Three traps account for most of the damage.

Trap 1: They share the same surface shape. Both produce a ranked list of items. Type a query into Google, get a list. Open Netflix, get a list. The visual output is identical, so the mental model defaults to "they're the same thing with different inputs." This is like assuming a taxi and a personal chauffeur are the same because both are cars. The interface is the same; the contract is not.

Trap 2: They use the same mathematical machinery. Both problems can be formulated as learning a matching function f(x, y) → relevance_score. Search learns f(query, document). Recommendation learns f(user, item). The architectures — two-tower encoders, dot-product scoring, ANN retrieval — are often identical:

# Both problems use the same architecture shape — but the semantics differ
class TwoTowerModel(nn.Module):
    """A shared architecture. But what the towers encode is completely different."""
    def __init__(self, input_dim: int, hidden_dim: int):
        super().__init__()
        self.tower_a = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim // 2)
        )
        self.tower_b = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim // 2)
        )

    def forward(self, x_a, x_b):
        return torch.cosine_similarity(self.tower_a(x_a), self.tower_b(x_b))

# FOR SEARCH: tower_a encodes QUERY text, tower_b encodes DOCUMENT text
# The query is an explicit string the user typed 500ms ago.

# FOR RECOMMENDATION: tower_a encodes USER profile, tower_b encodes ITEM features
# The user profile is a latent representation of years of behavior, updated daily.

# Same architecture. Completely different semantics.
# When the code looks the same, the problems feel the same. They aren't.

This is the second trap. The math is identical. The semantics are not.

Trap 3: LLMs make the boundary genuinely fuzzy. Ask an LLM "What should I watch tonight? I loved Dark and Severance" and it will produce recommendations. Ask the same LLM "Find me mind-bending sci-fi shows like Dark" and it will produce search results. The same model, the same prompt interface, the same output format. The temptation to conclude "search and recommendation are the same now" is powerful — and wrong. The LLM is a substrate, not a solution. It can perform either task, but optimizing for both simultaneously without awareness of which mode is active produces the worst of both worlds.

Peter Norvig's Insight: The Error Bar Is Different

Peter Norvig, Director of Research at Google, captured the consequence of the search–recommendation distinction with characteristic precision. When asked about the shift from retrieval to proactive assistance, he noted:

"With information retrieval, anything over 80% recall and precision is pretty good — not every suggestion has to be perfect, since the user can ignore the bad suggestions. With assistance, there is a much higher barrier."

The bar is different because the contract is different. In search, the user is the arbiter — they scan results, skip what's irrelevant, and reformulate if necessary. In recommendation, the system is the arbiter — it decides what the user sees. If it's wrong, the user doesn't skip a result; they lose trust in the system itself.

def measure_search_satisfaction(precision_at_10: float) -> str:
    """Norvig's rule: 80% is good enough for search."""
    if precision_at_10 >= 0.80:
        return "acceptable — user can scroll past the 20% that's wrong"
    return "needs improvement — too many irrelevant results visible"

def measure_recommendation_satisfaction(ndcg_at_10: float) -> str:
    """No equivalent 80% rule for recommendations."""
    if ndcg_at_10 >= 0.95:
        return "acceptable — nearly everything shown is relevant"
    # A recommendation that misses 20% is showing wrong items to users
    # who can't scroll past — they just see a bad experience.
    return "needs improvement — every wrong item erodes trust cumulatively"

Getting search wrong loses a query. Getting recommendation wrong loses a user.

Herlocker et al. established the standard evaluation framework for collaborative filtering, noting that recommendation metrics must capture user satisfaction in ways that precision and recall alone cannot [6].

The Library Analogy

Three scenarios make the distinction concrete:

  1. You walk into a library and ask the librarian, "Where are the books on the history of the Byzantine Empire?" The librarian walks you to a specific shelf. That's search.

  2. You walk into the same library and the librarian says, "I notice you've been reading a lot about medieval trade routes. We just got a new book on the Silk Road you might enjoy." That's recommendation.

  3. You walk into the library, glance around, and say, "I'm not sure what I want. Something historical but not too heavy, maybe with a good story?" The librarian pauses, then suggests three books and asks which one sounds right. That's the boundary — where search and recommendation blur into conversation.

These three scenarios feel different. They engage different cognitive postures, demand different system designs, and tolerate different kinds of errors:

  • A search error is visible and attributable: "This librarian doesn't know where the books are."
  • A recommendation error is invisible and cumulative: "This librarian doesn't understand my taste."
  • A boundary error is recoverable through conversation: "Not that one — what else do you have?"

The Restaurant Analogy

When you sit down at a restaurant and scan the menu, you're doing search. You have an intent — "I feel like pasta" — and you're scanning a structured catalog for matches. Your satisfaction depends on whether the menu accurately represents what the kitchen can deliver. A mistake here: "I ordered the carbonara and got the bolognese."

When the chef sends out a tasting menu — "trust me, you'll love this" — you're receiving a recommendation. The chef has built a model of what you might enjoy and is making predictions. Your satisfaction depends on whether the chef's model of you is accurate, and whether they can surprise you in a good way. A mistake here: "The chef brought me a dish centered on mushrooms, which I hate."

The first failure is a retrieval error. The second is a modeling error. Different failures, different fixes, different systems.

The Surveys Agree: Different Fields, Different Literatures

The best survey papers in each field make the distinction structurally. The Recommender Systems Handbook (Ricci, Rokach, and Shapira, 3rd edition, 2022) — the canonical 1,060-page reference — organizes recommendation into its own taxonomy: collaborative filtering, content-based, context-aware, session-based, and sequential methods [1]. The parallel information retrieval surveys — Guo et al.'s A Deep Look into Neural Ranking Models for Information Retrieval (2020) and Hambarde and Proença's Information Retrieval: Recent Advances and Beyond (2023) — organize around query-document matching, retrieval stages, and ranking objectives that have no equivalent in the recommendation literature [2][3].

These surveys do not reference each other much. That is not an accident. It is evidence that the fields have different problem statements, different evaluation cultures, and different assumptions about what "good" means.

What's Ahead

The remaining posts in this series cover: the fifty-year history of search technology, the thirty-year history of recommendation, the Netflix case study, how Spotify, DoorDash, Airbnb, and Pinterest draw the boundary, what LLMs change — and what they don't, and a practical architecture for building systems that handle both. Each stands alone; read in any order.


Open Questions

  1. If the distinction is so clear, why do most ML curricula still teach search and recommendation as variants of the same ranking problem? What would a curriculum look like that treated them as separate disciplines with shared infrastructure?

  2. Norvig's 80% rule suggests search tolerates error better than recommendation. But as search engines become answer engines (via RAG), the error bar rises. At what point does search become assistance — and inherit recommendation's trust problem?

  3. The three traps explain why engineers do conflate search and recommendation. But what incentives — organizational, metric, career — make them want to? Is unification sometimes a political choice rather than a technical one?

  4. Belkin and Croft asked in 1992 whether IR and information filtering were two sides of the same coin. Thirty-four years later, with LLMs in the picture, is the answer different than it was then? Or did they get it right the first time? Jannach et al.'s introductory textbook on recommender systems reinforces the distinction structurally: it treats the search/recommendation boundary as a design choice, not a convergence point — search is about satisfying a stated need, while recommendation is about anticipating an unstated one [8].

References

  1. Francesco Ricci, Lior Rokach, and Bracha Shapira (editors). Recommender Systems Handbook, 3rd Edition. Springer, 2022.

  2. Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, W. Bruce Croft, et al. A Deep Look into Neural Ranking Models for Information Retrieval. Information Processing & Management, 57(6), 2020.

  3. Kailash Hambarde and Hugo Proença. Information Retrieval: Recent Advances and Beyond. arXiv:2301.08801, 2023.

  4. Nicholas J. Belkin and W. Bruce Croft. Information Filtering and Information Retrieval: Two Sides of the Same Coin?. Communications of the ACM, 35(12): 29–38, 1992.

  5. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008. The canonical IR textbook covering Boolean retrieval, vector space models, probabilistic retrieval, and evaluation.

  6. Jonathan L. Herlocker, Joseph A. Konstan, Loren G. Terveen, and John T. Riedl. Evaluating Collaborative Filtering Recommender Systems. ACM Transactions on Information Systems, 22(1): 5–53, 2004. The definitive paper on recommendation evaluation metrics.

  7. Carlos A. Gomez-Uribe and Neil Hunt. The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Transactions on Management Information Systems, 6(4): 1–19, 2015. Netflix's own description of their recommendation architecture and business impact.

  8. Dietmar Jannach, Markus Zanker, Alexander Felfernig, and Gerhard Friedrich. Recommender Systems: An Introduction. Cambridge University Press, 2010. A comprehensive introductory textbook.

  9. W. Bruce Croft, Donald Metzler, and Trevor Strohman. Search Engines: Information Retrieval in Practice. Pearson, 2010. A practical textbook covering search engine architecture, indexing, and ranking.

  10. Tefko Saracevic. Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science, Part II. Journal of the American Society for Information Science and Technology, 58(13): 1915–1933, 2007. A comprehensive review of relevance — the central concept in information retrieval.


Loop Engineering is what the NATO conference asked for in 1968

Loop Engineering — designing systems that prompt agents instead of prompting them yourself — is 2026's dominant AI paradigm. The idea is not new. It is the practical realization of a principle that has been at the core of software engineering since the field was named in Garmisch, 1968: feedback is everything. Build the loop. Then get out of it.

loop-engineeringagentsnatosoftware-engineeringfeedback-loopshistorydijkstrabrooks

In June 2026, Peter Steinberger tweeted:

"Here's your monthly reminder that you shouldn't be prompting coding agents anymore. You should be designing loops that prompt your agents."

Two million views. Within days, Claude Code's Boris Cherny said at a public event:

"I don't prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops."

Cherny wasn't speaking hypothetically. He reported that 100% of contributions to the Claude Code repository in the prior thirty days — 259 merged PRs — had been written by Claude Code itself, driven by loops he had designed. He deleted his IDE in November 2025 and hasn't reopened one since. The loops wake up, read GitHub issues, scan Twitter for feedback, parse Slack, decide what to build, and build it. Cherny watches the output. He does not participate in the loop.

Google's Addy Osmani gave it a name: Loop Engineering.

"Loop engineering is replacing yourself as the person who prompts the agent. You design the system that does it instead."

By July, the term was everywhere — blog posts, conference talks, the obligatory LinkedIn thought leaders. The old thing was Prompt Engineering: you write better and better prompts to get better and better outputs. The new thing is Loop Engineering: you build a system that writes the prompts, runs the agent, checks the output, and feeds the result back into the next cycle. You don't talk to the agent. You talk to the loop. The loop talks to the agent. You design the loop.

The industry treated this as a breakthrough. It is not a breakthrough. It is a homecoming. Software engineering was invented to solve exactly this problem. The name was coined at a conference in 1968. The conference report said, on page 11, four sentences in: "The need for feedback was stressed many times." Everything since has been an attempt to build better loops.

Garmisch, 1968

In October 1968, about fifty people from eleven countries gathered in Garmisch, Germany, for a NATO-sponsored conference. The chair was Friedrich Bauer. The topic was something nobody had a name for yet. Bauer gave it one: software engineering. He chose the word deliberately — provocative, aspirational, implying that software manufacture needed the same theoretical foundations and practical disciplines as the established branches of engineering. The term described a discipline that did not yet exist.

The conference report, edited by Peter Naur and Brian Randell, runs over two hundred pages. It introduced the phrase "software crisis" — the widening gap between what people wanted software to do and what they could actually build. Doug McIlroy, one of the participants, later described the editorial method — stitching together direct quotations from papers and transcribed discussions — as:

"A triumph of misapplied quotation."

It was an affectionate jab. The report is remarkable precisely because it preserves the voices of the participants in something close to raw form. You can hear them thinking. You can hear them discovering that they all have the same problem and nobody has the solution.

The line that matters most, the one that still has not been fully absorbed, appears early:

"The need for feedback was stressed many times."

Not discovered. Not proposed. Stressed. The participants already knew feedback was essential. They were stressing it because they were watching an industry ignore it. The dominant method of building software in 1968 was what J.W. Graham described during a panel on feedback through monitoring and simulation:

"Today we tend to go on for years, with tremendous investments to find that the system, which was not well understood to start with, does not work as anticipated. We build systems like the Wright brothers built airplanes — build the whole thing, push it off the cliff, let it crash, and start over again."

Years of investment. No intermediate validation. Catastrophic late-stage discovery of fundamental problems. The feedback loop was the entire project lifecycle. You learned whether the system worked only when it was done. Usually it didn't. You started over. This was not a straw man. This was the state of the art in 1968. The people in the room had lived it. Some were still living it.

The conference attendees understood that this was the problem. The question was what to do about it. Kinslow pointed out that the stages everyone assumed were separate — design, production — were not separate at all:

"The design process is iterative, as certain specifications and issues may not be realised until production of the program begins."

Some argued more aggressively: since designers make decisions during production about how to fit a design using available tools, the boundary between design and production should be minimal. A tight, continuous feedback loop between specification and implementation. This was 1968. They were describing continuous delivery. They were describing what we now call an inner loop. They did not have the tooling to make it real, but they had the concept. The concept was waiting fifty years for the tooling to catch up.

Alan Perlis went further. He argued that testing should not come after design — it should be woven into the act of designing. And then he said something that took three decades to fully land:

"The critical point is that the simulation becomes the system."

The simulation becomes the system. The test harness becomes the specification. The specification is executable. The system grows through iterations of a loop where each cycle produces a working artifact that is also the specification for the next cycle. This is exactly what Test-Driven Development would become in the late 1990s. It is exactly what Loop Engineering proposes for AI agents now. The thing being looped is not a test suite. It is an agent with tools, a verifier, and a feedback mechanism. The structure is identical. The components have changed. The loop has not.

Perlis understood the significance of the report. He gave copies to his graduate students at Carnegie Mellon with the words: "Here, read this. It will change your life."

The history of software engineering is the history of tightening loops

If you read the history of software engineering as a history of loops, a clear pattern emerges. Every major methodology shift was an attempt to tighten a feedback loop — to reduce the time between making a decision and discovering whether it was right.

Structured programming (late 1960s). Dijkstra's "Go To Statement Considered Harmful" was published in the March 1968 issue of Communications of the ACM — the same year as Garmisch. The argument was not about aesthetics. It was about whether you can reason about a program without running it. Dijkstra's exact reasoning is worth reading in full:

"My second remark is that our intellectual powers are rather geared to master static relations and that our powers to visualize processes evolving in time are relatively poorly developed. For that reason we should do (as wise programmers aware of our limitations) our utmost to shorten the conceptual gap between the static program and the dynamic process, to make the correspondence between the program (spread out in text space) and the process (spread out in time) as trivial as possible."

The goto statement broke this correspondence. It made it impossible to locate yourself in the program's execution using the structure of the source text alone. You needed to also track the values of variables — but the meaning of those variables could only be understood relative to where you were in the program, which you couldn't determine without the variables. The loop between reading code and understanding it was broken. Structured programming restored it. The loop became local. Before: read the code, trace every possible goto, get confused, give up, run it and hope. After: read the code, understand it. The distance between the static text and the dynamic process collapsed.

This was not a stylistic preference. It was a feedback argument disguised as a language design argument. Dijkstra wanted the loop between the programmer's eye and the programmer's understanding to be as short as possible. He wanted the compiler to provide feedback — through structure, through enforceable constraints — rather than requiring the programmer to simulate execution in their head. The structured programming movement, at its core, was about tightening the comprehension loop.

Unix philosophy (1970s). McIlroy, the same McIlroy who called the NATO report a triumph of misapplied quotation, wrote the memo that proposed Unix pipes. His argument:

"We should have some ways of coupling programs like garden hose — screw in another segment when it becomes necessary to massage data in another way."

Small programs that do one thing well. Text streams as universal interface. The loop: write a program, test it in isolation, compose it with others, observe the output. Each program is a closed loop. The composition is a larger loop. The feedback is immediate because the programs are small and the interface is uniform. McIlroy's pipes made the loop between idea and result as short as the time it takes to type |. This is the same McIlroy who appreciated the NATO report enough to gently mock it. He understood what the report was asking for. He built part of the answer.

Test-driven development (late 1990s). Beck's insight: write the test first, watch it fail, write the code, watch it pass, refactor. The test is the specification. The specification is executable. The loop — red, green, refactor — runs in seconds. Perlis said "the simulation becomes the system" in 1968. Beck made it mechanical thirty years later. The loop was always the idea. The infrastructure caught up.

Continuous integration and delivery (2000s–2010s). Fowler and Beck on CI: every commit triggers a build and a test suite. Humble and Farley on CD: every green build is potentially shippable. The loop that TDD ran in seconds on a developer's machine now runs across an entire team, continuously. The time between writing a line of code and discovering it broke something dropped from weeks to minutes. Brooks, in No Silver Bullet (1986), had already described the principle:

"Using rapid prototyping as part of a planned iteration in establishing software requirements."

And:

"Growing software organically, adding more and more function to systems as they are run, used, and tested."

He was describing the loop. He was also describing why it was hard in 1986: the tooling for rapid prototyping was primitive, the cost of iteration was high, and the organizational structures that made CI/CD possible did not yet exist. The idea was there. The infrastructure was not. The pattern repeats: first the concept, then a long wait, then the tooling.

Agentic software engineering (2020s). AI coding agents write the code. Other agents review it. Still others test it. The human writes the specification. The loop runs without the human in it. This is what Steinberger and Cherny and Osmani are describing. It is what StrongDM has been running in production since mid-2025. It is what Shapiro's five-level dark factory framework predicts. The loop has been progressively tightening for fifty-seven years. At the end of the tightening, the human is no longer inside the loop. The human designs the loop. The loop runs itself.

Lehman's laws (1980). Meir Lehman observed that large software systems must be continually adapted or they become progressively less useful. The environment changes around them. The system must change in response. This is a loop: the system observes its environment, detects drift, and adapts. Lehman formalized what the Garmisch attendees felt intuitively: software is not a product. It is a process. The process is a loop. The loop never ends. Lehman stated it as a law because it is not optional. You can ignore it only if your system is dead.

What changed in 2026

If the loop was always the idea, what actually changed? Why did Loop Engineering become a named discipline in 2026 and not 2016 or 2006?

Three things changed at roughly the same time.

First, the agent became capable enough to close the loop autonomously. A loop needs a generator and a verifier. The generator produces candidates. The verifier checks them. If the generator is not good enough to produce candidates worth verifying, or the verifier is not good enough to distinguish correct from plausible, the loop does not close. You need a human inside it — providing the right prompt, inspecting the output, adjusting, trying again. Prompt Engineering was the era where the human was the verifier. Loop Engineering is what becomes possible when the verifier can be automated. The human moves from inside the loop to outside it, designing the verification criteria, the stopping conditions, the fallback paths. The loop runs. The human inspects only the final result, or only the failures.

This is the same transition that happened when TDD automated the test-authoring loop, and when CI automated the integration loop, and when CD automated the deployment loop. Each time, something that previously required human judgment became mechanical. Each time, the human's role shifted from performing the verification to designing the verification. The 2026 transition is different in scale, not in kind. The scope of what can be verified automatically has expanded from "does this function return the right value" to "does this pull request correctly implement the specification." That is a large expansion. But the structure — human designs the verification, machine runs it — is the same structure Perlis described in 1968.

Second, the infrastructure matured. Claude Code shipped /loop, /goal, and dynamic workflows. OpenClaw shipped worktree-isolated agent runners. The major platforms converged on the same primitives: time-based triggers, isolated workspaces, project memories, MCP connectors, sub-agents. Steinberger followed his viral tweet with a concrete example:

"Tell codex to maintain your repos, wake up every 5 minutes and direct work to threads."

This is not a metaphor. It is a command. The infrastructure exists. The loop is no longer a research project. It is an engineering discipline with known primitives, known failure modes, known design patterns. Osmani identified five core components: automations, worktrees, skills, plugins/connectors, and sub-agents. Cherny's workflow — loops reading GitHub issues, scanning feedback, parsing Slack, dispatching work — is a composition of these components. The loop is an architecture. The architecture has primitives. The primitives are documented.

Third, the economics inverted. For most of software engineering history, the constraint was implementation. You could specify faster than you could build. The bottleneck was the rate at which intent could be translated into working code. With capable coding agents, the constraint moved from implementation to specification. You can build faster than you can specify. The expensive thing is no longer writing code. The expensive thing is writing precise enough specifications that the agent produces what you actually want.

The loop is the mechanism that discovers the gap between your specification and your intent. Each turn of the loop reveals something you forgot to say, an edge case you didn't consider, an assumption you didn't state. The loop does not just produce code. It produces knowledge about what you actually want. This is exactly what Kinslow was describing in 1968: "certain specifications and issues may not be realised until production of the program begins." The discovery that your specification was wrong is not a failure of the process. It is the process. The loop is the machine that converts specification errors into knowledge.

The loop is a capital investment

There is an economic principle here that is easy to miss. A prompt is a labor expense. You pay it every time. A loop is a capital investment. You pay to build it once. It pays you back over every turn.

The distinction is the same one that separates craftsmanship from manufacturing. A craftsperson makes each item by hand. Quality depends on skill, attention, and energy. A manufacturer builds a production line. The line has a higher upfront cost. It requires design, tooling, calibration. Once built, it produces items at marginal cost approaching zero. The craftsman's advantage is flexibility. The manufacturer's advantage is scale.

Prompt Engineering is craftsmanship. Each interaction is hand-tuned. The quality of the output is a function of the quality of the prompt, which is a function of the skill and attention of the prompter. This works well at low volume. It breaks at scale. When your agent generates a hundred candidate solutions and you need to evaluate all of them, you cannot hand-craft a hundred evaluation prompts. You need a loop.

Loop Engineering is manufacturing. You invest upfront in designing the generator-verifier-refiner pipeline, the stop conditions, the fallback paths, the state management. Once built, the loop runs at marginal cost approaching the API bill. You pay the design cost once. You pay the compute cost per turn. You pay zero attention cost. Your attention is the scarcest resource. The loop conserves it.

This is why Steinberger called it a "monthly reminder." The economics keep shifting toward loops and away from prompts. Every month, the agents get better, the infrastructure gets more capable, and the case for remaining inside the loop gets weaker. The reminder is monthly because the threshold keeps moving. What required a human in the loop last month might not this month. The capital investment that made sense today would have been premature a year ago and will feel obvious a year from now.

The loop is the system

0xCodez synthesized the emerging practice into a fourteen-step roadmap. The central thesis:

"Self-improvement is a property of the system, not the model — build the system."

The model's weights are fixed. The model does not learn during use. The model does not remember what worked last time unless you build memory around it. The system accumulates. STATE files record what was tried and what succeeded. Skills capture reusable patterns. Eval loops measure whether the output is getting better or worse. Each run writes lessons to memory. The next run inherits sharper context. The model is unchanged. The framework around it gets sharper. That is the self-improvement. Not the model learning. The framework accumulating.

This was always true, even before AI. A team with a good CI pipeline and a bad codebase will, over time, improve the codebase. The pipeline provides feedback. The feedback drives improvement. A team with a good codebase and no pipeline will, over time, degrade the codebase. The absence of feedback allows drift. The codebase is not self-healing. The pipeline does the healing. The pipeline is the loop. The loop is the system. The system improves because it is a loop.

The same principle applies to agentic systems. A single agent with a powerful model but no loop produces one output and stops. The output is as good as the model can make it in one shot. A weaker model wrapped in a well-designed loop — generate, verify, refine, repeat — can outperform a stronger model used once. The loop provides the improvement the model cannot provide on its own. The model is a component. The loop is the system. The system outperforms the component.

Osmani captured the relationship between the loop and its components:

"Loop Engineering sits one floor above the harness. The harness runs on a timer, it spawns little helpers, and it feeds itself."

The harness is the scaffolding. The loop is the design that gives the scaffolding purpose. The harness wakes up, spawns agents, collects their output. The loop decides what to do with the output: was it good enough? Should we try again with a different approach? Should we escalate to the human? The harness is mechanical. The loop is architectural. You can build a harness without understanding loops. You will just have a very expensive timer.

This is why Loop Engineering matters beyond the buzzword. It is not about whether you prompt by hand or design loops. It is about where you invest your design effort. Prompt Engineering invests in the input to the model. Loop Engineering invests in the structure around the model. The second investment compounds. The first does not. A better prompt produces a better output once. A better loop produces better outputs forever.

Dijkstra understood this in 1968. The goto statement was "just too primitive; it is too much an invitation to make a mess of one's program." He was not arguing against power. He was arguing for structure over ad-hoc power. A goto can do anything. That is the problem. A loop with well-designed stop conditions, verification gates, and state management can also do anything — but it does it reliably, repeatably, improvable. The difference between a goto and a for-loop is the difference between a prompt and a loop-engineered system. Both produce output. One produces output you can reason about. One produces output you can improve.

The trap

There is a trap. Osmani warned about it. The loop can become a machine for accelerating decline.

"Loops can also accelerate decline if used to avoid understanding."

If you don't understand what your loop is doing, the loop will drift. Each turn of the loop compounds small errors. The output gets worse. The verification gates, if they are weak, pass the degraded output through. The next turn starts from degraded input. The loop becomes a decay spiral. You discover the problem only when the output is visibly broken — by which point the loop has been producing subtly wrong results for weeks.

This is not a new problem. It is the same problem Graham described in 1968: "build the whole thing, push it off the cliff, let it crash, and start over again." The difference is that the loop accelerates the cycle. A waterfall project takes years to crash. A loop can degrade in hours. The loop tightens feedback in both directions — it tightens the feedback that improves the system, and it tightens the feedback that degrades it. A well-designed loop amplifies good decisions. A poorly-designed loop amplifies bad ones. The loop is amoral. It does not care what it amplifies.

The answer is not to avoid loops. The answer is to verify the verifier — to have meta-loops that check whether the primary loop is still producing useful output, to have escalation paths that flag anomalies for human review, to have circuit breakers that halt the loop when confidence drops below a threshold. The loop needs guardrails. The guardrails are themselves loops. The architecture is recursive. The recursion bottoms out in human judgment. The human does not need to be in every loop. The human needs to be at the escape hatch.

This is the hardest lesson of Loop Engineering. Removing yourself from the loop does not mean removing yourself from responsibility. It means relocating your attention to the places where it matters most: designing the verification criteria, monitoring the escape hatches, inspecting the anomalies. The loop handles the routine. You handle the exceptions. The exceptions are where the value is.

From Garmisch to the loop

The NATO conference ended with a set of recommendations. Better tools. Better education. Better management practices. Better theoretical foundations. They did not recommend Loop Engineering because the term did not exist and the infrastructure did not exist and the agents did not exist. But the idea was there. The need for feedback was stressed many times. The Wright brothers method was recognized as unsustainable. Perlis described the loop where testing and designing interlace and the simulation becomes the system. Kinslow described the loop where production reveals what specification missed. The entire conference was an argument for the loop. The argument was correct in 1968. It took fifty-seven years for the infrastructure to catch up.

What happened in those fifty-seven years was the progressive construction of the machinery that makes the loop practical. Structured programming made loops local. Unix made loops composable. TDD made loops executable. CI/CD made loops continuous. Agentic engineering made loops autonomous. Each step removed a human from some part of the loop. Each step tightened the time between action and feedback. Each step moved the industry closer to what the Garmisch attendees described as necessary but impossible with the tools of their time.

The tools are no longer the constraint. The constraint is the willingness to design the system instead of just operating inside it. Steinberger's tweet was not a technical insight. It was a restatement of page eleven of the NATO report, fifty-seven years later, in the language of the platform that finally made it possible. Feedback is everything. Build the loop. Then get out of it.

The loop has always been the idea. The loop is finally buildable. The question now is whether we will build loops that make us better, or loops that make us faster at being wrong. That choice — better or faster, understanding or avoidance, structure or chaos — is the same choice the Garmisch attendees faced. The technology has changed. The choice has not.

On Rule Engines — From RETE to MCP

Tracing a 40-year arc from Charles Forgy's RETE algorithm to the Model Context Protocol, this essay explores how rule engines and large reasoning models answer the same fundamental question through different mechanisms — and why the most important trend is not replacement but convergence.

llmrulesenterpriseautomationcomposite-aisymbolic-airetemcpfsm

Enterprise decision automation sits at an uncomfortable intersection. Large Language Models promise natural interaction and flexibility. Business decisions demand auditability, determinism, and compliance with regulations that do not negotiate. The question is not which technology wins. The question is how to combine them — and the answer has roots going back nearly half a century.

The question is not which technology wins. The question is how to combine them.

Pierre Feillet addressed this in his 2023 article Approaches in Using Generative AI for Business Automation and his 2026 follow-up Rule Engines Never Died — They're Running Alongside Your Large Reasoning Models. The second article traces a 40-year arc from Charles Forgy's RETE algorithm to the Model Context Protocol and argues that the most important trend in enterprise AI is not replacement but convergence.

This essay — the first in a five-part series — begins where the story begins: with the RETE algorithm, the architecture of rule engines, the rise of large reasoning models, and the fundamental distinction between finite state machines and rule engines that every system architect should understand.

The RETE algorithm (1979)

In 1979, Charles L. Forgy, a PhD student at Carnegie Mellon University, solved a problem that would shape the next four decades of enterprise computing. Given a large set of IF-THEN rules and a working memory full of facts, how do you efficiently determine which rules should fire?

The naive approach — test every rule against every fact on every cycle — is catastrophically slow. Ten thousand rules against a hundred thousand facts is a billion comparisons per cycle. Production systems in the 1970s ground to a halt under realistic workloads.

Forgy's insight was that working memory changes slowly between cycles. Typically only a few facts are added or removed at each step. Re-evaluating every rule from scratch means repeating vast amounts of work that has not changed. What if you could remember partial matches and only recompute what actually changed?

The RETE algorithm trades memory for speed — and achieves performance theoretically independent of the number of rules.

The result was the RETE algorithm — Latin for "net" — published in Forgy's 1979 PhD thesis and in a landmark 1982 paper in Artificial Intelligence: Rete: A Fast Algorithm for the Many Pattern/Many Object Pattern Match Problem.

How it works

RETE has two phases. Compilation: rules are compiled into a discrimination network — a directed acyclic graph where each node tests a condition. Conditions appearing in multiple rules are compiled once and shared. If twenty rules check customer.tier == "premium", there is one node, not twenty.

Consider these three loan underwriting rules:

# Rule 1: High-value loan requires collateral
if loan.amount > 500_000 and loan.type == "mortgage":
    loan.requires_collateral = True

# Rule 2: Premium customers get rate discount
if customer.tier == "premium" and loan.type == "mortgage":
    loan.rate_discount = 0.25

# Rule 3: High-value premium mortgage gets priority review
if loan.amount > 500_000 and customer.tier == "premium" and loan.type == "mortgage":
    loan.review_priority = "high"

A RETE network compiles these into shared condition nodes. The test loan.type == "mortgage" appears in all three rules but exists once in the network. The test loan.amount > 500_000 appears in two rules but exists once. When a fact changes — say the loan amount updates — only the branches downstream of the loan.amount node re-evaluate. Rules 1 and 3 might be affected; Rule 2 is untouched.

Runtime: facts enter at the root and propagate through the network. Each node caches the facts — or partial matches, at join nodes — that satisfy its condition. When a fact changes, only affected branches re-evaluate. When a fact is removed, its matches are retracted from the caches. The algorithm trades memory for speed — storing partial match state at every node — and achieves performance that is, in Forgy's words, theoretically independent of the number of rules.

The lineage

RETE became the core of OPS5, which powered R1/XCON — an expert system that configured VAX computer orders for Digital Equipment Corporation, one of the first commercially successful AI systems, reportedly saving DEC millions of dollars per year by catching configuration errors that human order processors missed.

It went on to become the backbone of production rule systems across the industry: CLIPS (NASA's C Language Integrated Production System), Jess (the Java Expert System Shell), Drools (open-source, now Apache, which evolved RETE into ReteOO and later PHREAK), IBM Operational Decision Manager (enterprise-grade decision automation with governance, versioning, and deployment pipelines), Soar (the cognitive architecture), Blaze Advisor, and TIBCO BusinessEvents.

The RETE network was invisible infrastructure, humming inside systems that made consequential decisions about people's money, health, and legal status.

Banks used it for loan origination. Insurers for claims adjudication. Governments for eligibility determination. For four decades, RETE was the silent backbone of automated decision-making.

And then, around 2022, the world became captivated by a different kind of AI — one that produced fluent text rather than deterministic decisions. The question became: are rule engines obsolete?

Feillet's 2026 answer is unequivocal: no. They never died. They are running alongside your LLM right now.

Beyond RETE

Modern rule engines have evolved beyond the discrimination network. RuleGo — an open-source rule engine in Go (Apache 2.0) — uses a Directed Acyclic Graph. Business logic is composed of component nodes wired into rule chains; messages flow along predetermined DAG paths rather than being matched against all rules. A RuleGo rule chain looks like this:

// A RuleGo rule chain for loan application intake
ruleChain := rulego.NewRuleChain("loan-intake",
    // Node 1: Validate required fields
    rulego.NewTransformNode("validate-fields").
        WithScript(`msg.Metadata.valid = msg.Data.amount > 0 && msg.Data.applicantId != ""`),
    // Node 2: Branch on validation result
    rulego.NewSwitchNode("route").
        WithCase("valid", "extract-intent").
        WithCase("invalid", "return-error"),
    // Node 3: LLM intent extraction
    rulego.NewRestNode("extract-intent").
        WithEndpoint("https://api.llm.example/v1/chat").
        WithBodyTemplate(`Extract intent from: {{.Data.description}}`),
    // Node 4: Route to decision engine
    rulego.NewRestNode("decision-engine").
        WithEndpoint("http://odm.example/decisions/loan-eligibility"),
)

This trades RETE's expressive forward-chaining for deterministic execution with extremely low resource consumption — ~19 MB of memory under 500 concurrent requests on a Raspberry Pi 2. Both approaches have their place.

Two ways to answer the same question

Feillet's 2026 article opens with a deceptively simple observation: both rule engines and Large Reasoning Models address the identical problem — given what we know, what should we conclude?

The methods differ. The objective is the same. This means they can — and should — be compared, contrasted, and combined.

How rule engines reason

Rule engines separate knowledge from execution. Business logic is declarative. Facts enter working memory. The inference engine evaluates which rules are satisfied. When multiple rules fire, a conflict resolution mechanism — the agenda — determines firing order.

# Declarative: you say WHAT, not HOW
rules = [
    Rule(
        when=[Fact("loan.amount", ">", 500_000), Fact("loan.type", "==", "mortgage")],
        then=[Action("set", "loan.requires_collateral", True)]
    ),
    Rule(
        when=[Fact("customer.tier", "==", "premium"), Fact("loan.type", "==", "mortgage")],
        then=[Action("set", "loan.rate_discount", 0.25)]
    ),
]

# The engine handles: which rules to evaluate, in what order,
# what happens when multiple rules fire, how to resolve conflicts.
engine = RuleEngine(rules)
engine.insert(loan_facts)
engine.run()  # produces: {requires_collateral: True, rate_discount: 0.25}

Every inference step is explicit. An auditor can trace precisely which rule fired, what triggered it, and why a decision was reached. Rule engines do not generate post-hoc rationalizations. They produce proofs.

How Large Reasoning Models reason

LRMs store knowledge not as explicit rules but as distributed numerical representations learned across billions of examples. Their reasoning is what Feillet calls "an approximation of modus ponens" executed through statistical pattern completion.

Prompt: "Should this $600K mortgage for a premium-tier customer require collateral?"

LLM: "Based on the loan amount of $600K exceeding the typical threshold of
$500K for mortgages, and considering this is a premium-tier customer..."

The power is flexibility — generalization across novel inputs, ambiguity handling, cross-domain reasoning without explicit programming. The fragility is hallucination — producing plausible-sounding but logically invalid conclusions.

Plausibility and logical validity are different things. One yields proofs. The other yields rationales. Both are useful. Confusing them is dangerous.

The core distinction

Feillet gives a concrete example from the 2023 article: a pizza ordering bot that "depending on the runs, provides the expected outcome or a surprising one." Wrong pizza toppings are annoying. Wrong loan decisions are lawsuits.

# Rule engine: same input → same output, every time
result1 = engine.decide(loan_application)  # deny, DTI 47%
result2 = engine.decide(loan_application)  # deny, DTI 47%
assert result1 == result2  # always passes

# LLM: same input → statistically similar output, maybe not the same
response1 = llm.generate("Review this loan: " + application)  # "I recommend denial..."
response2 = llm.generate("Review this loan: " + application)  # "This application should be..."
# response1 and response2 may differ in wording, tone, or even conclusion

The rule engine guarantees correctness relative to its encoded rules. The LRM generates text that sounds like reasoning. In enterprise contexts, you need the first and want the second — which is exactly why you combine them.

The hybrid inference loop

Modern reasoning models have evolved beyond raw token prediction. When a calculation is needed, the model generates and executes code — an exact result fed back into context. This is a pattern: stochastic reasoning delegating to deterministic execution at moments where precision matters.

The Model Context Protocol formalizes this delegation. An MCP tool definition for a decision service looks like:

{
  "name": "check_loan_eligibility",
  "description": "Evaluate a mortgage application against underwriting rules",
  "inputSchema": {
    "type": "object",
    "properties": {
      "loan_amount": {"type": "number"},
      "applicant_income": {"type": "number"},
      "credit_score": {"type": "integer"},
      "property_value": {"type": "number"},
      "loan_type": {"type": "string", "enum": ["mortgage", "heloc", "refinance"]}
    },
    "required": ["loan_amount", "applicant_income", "credit_score", "property_value"]
  }
}

The loop is:

1. Reason with available context
2. Identify what is uncertain or requires precision
3. Delegate to a tool (MCP call to the rule engine)
4. Receive deterministic facts
5. Continue reasoning with grounded information

Feillet draws a structural parallel to production rule systems invoking external actions — except orchestration is now performed by a neural model rather than a symbolic agenda. IBM ODM and Decision Intelligence expose their decision services via MCP, enabling a reasoning model to invoke a full rule engine at the precise point where governed, deterministic decisions are needed.

The LRM handles interpretation and context. The rule engine handles logic requiring correctness and traceability. Neither is asked to do the other's job.

Attention and RETE: same purpose, different mechanism

Feillet is careful not to equate the two, but he notes a shared architectural role. Both answer the question what is relevant right now?

RETE pre-compiles condition tests into a network and caches partial matches. Attention learns to compute relevance scores from data. The mechanisms differ. The function is identical.

RuleGo: the convergence in a Go library

The convergence is not confined to enterprise platforms. RuleGo embodies the hybrid architecture in a Go library. An LLM call is just another node in the rule chain:

// RuleGo rule chain: deterministic nodes + LLM nodes, same interface
chain := rulego.NewRuleChain("claims-fraud-check",
    // Deterministic: validate claim data
    rulego.NewTransformNode("validate").
        WithScript(`msg.Metadata.skip = msg.Data.claimAmount < 10000`),
    // Deterministic: branch
    rulego.NewSwitchNode("threshold-check").
        WithCase("skip", "approve").
        WithCase("review", "llm-analysis"),
    // LLM call: anomaly detection in claim notes
    rulego.NewRestNode("llm-analysis").
        WithEndpoint("https://api.llm.example/v1/chat").
        WithBodyTemplate(`Review for anomalies:
            Claim: {{.Data.description}}
            History: {{.Data.priorClaims}}
            Amount: {{.Data.claimAmount}}`),
    // Deterministic: route based on LLM result
    rulego.NewSwitchNode("fraud-route").
        WithCase("clean", "approve").
        WithCase("suspicious", "investigate"),
)

The rulego-components-ai extension provides LLM integration and MCP server/client support. A RuleGo instance exposes its rule chains as MCP tools, discoverable by any reasoning model. The hybrid inference loop Feillet describes — reason, identify uncertainty, delegate, receive, continue — is directly implementable.

The 40-year arc from RETE to MCP passes through go get github.com/rulego/rulego.

Finite state machines vs. rule engines

Before moving to composite AI patterns — the subject of the next essay — there is an architectural distinction worth making explicit.

A finite state machine answers what happens next? It encodes explicit states and transitions. Its logic is procedural.

A rule engine answers what should we conclude? It encodes condition-action pairs against a working memory. Its logic is declarative.

// FSM: procedural — you define the sequence
type LoanFSM struct { state string }
func (f *LoanFSM) Transition(event string) error {
    switch f.state {
    case "application":
        if event == "submit" { f.state = "review" }
    case "review":
        if event == "approve" { f.state = "underwriting" }
        if event == "reject" { f.state = "closed" }
    }
}

// Rule Engine: declarative — you define the conditions
rules := []Rule{
    {When: "loan.amount > 500K AND loan.type = mortgage", Then: "require_collateral"},
    {When: "customer.tier = premium AND loan.amount > 250K", Then: "priority_review"},
}

If the problem is procedural — steps, stages, sequences — reach for an FSM. If the problem is decisional — eligibility, pricing, risk, compliance — reach for a rule engine. Most real business processes are both. That is the subject of the fourth and fifth essays.

The full comparison and anti-patterns are explored in Part 4 of this series. The composed architecture — FSM as process skeleton, rule engine as decision muscle — is developed in Part 5.

The 40-year arc

The story from RETE to MCP is not obsolescence and replacement. It is infrastructure that works being augmented by capabilities that are new. The RETE network matches facts against rules with Boolean precision. The attention mechanism computes relevance over learned representations. Both answer the same question — what is relevant right now? — at different layers of the stack.

A decision system that cannot explain itself is not enterprise-grade. A decision system that cannot handle ambiguity is not useful. The composite approach accepts both constraints and designs for them.

The next essay examines the five composite AI patterns that Feillet and his co-authors proposed in 2023: concrete architectures for combining neuronal and symbolic AI in production systems.


References

  1. Charles L. Forgy. Rete: A Fast Algorithm for the Many Pattern/Many Object Pattern Match Problem. Artificial Intelligence, 19(1): 17–37, 1982.

  2. Pierre Feillet, Allen Chan, Luigi Pichett, Yazan Obeidi. Approaches in Using Generative AI for Business Automation. Medium, August 4, 2023.

  3. Pierre Feillet. Rule Engines Never Died — They're Running Alongside Your Large Reasoning Models. Medium, June 4, 2026.

  4. RuleGo — Lightweight, component-based rule engine for Go. Apache 2.0. Includes rulego-components-ai for LLM integration and MCP support.

Part 2: On Rule Engines — Five Patterns for Composite AI · Part 4: On Rule Engines — State Machines vs. Rule Engines

On Rule Engines — Five Patterns for Composite AI

Five architectural patterns for blending LLMs with rule-based decision engines — NLU to rules, rules to NLG, rules orchestrating LLMs, LLM-driven rule extraction, and chatbot delegation. Each pattern answers a different question about where the LLM sits in the decision pipeline and what role it plays.

llmrulesenterpriseautomationcomposite-aidecision-makingsymbolic-aifsm

The first essay in this series traced the 40-year arc from Forgy's RETE algorithm to the Model Context Protocol and established the core distinction: rule engines produce proofs, large reasoning models produce rationales. Both are useful. Neither alone suffices for production AI.

This essay — the second in a five-part series — examines the five architectural patterns that Pierre Feillet, Allen Chan, Luigi Pichett, and Yazan Obeidi proposed in their 2023 article Approaches in Using Generative AI for Business Automation. They let an architect reason about where in the pipeline the LLM should sit and what role it should play.

These are not theoretical constructs. They are battle-tested integration topologies from enterprise practice.

What an enterprise decision requires

Feillet's article enumerates eight criteria. They explain why the LLM-only approach keeps hitting a wall in production.

  1. Accuracy. A loan decision wrong 2% of the time is not 98% accurate — it is a regulatory finding and a lawsuit. Correctness means every time.
  2. Scalability. Millions of claims per day cannot degrade when the rule base grows to tens of thousands of rules. RETE's performance-independence from rule count matters here.
  3. Adaptability. Regulations and policies change. The system must accommodate new rules without a multi-month DevOps cycle.
  4. Latency. Budgets vary — milliseconds for fraud, seconds for pre-approval, minutes for underwriting — but must be predictable. LLM latency is variable; rule engine latency is not.
  5. Auditability. "The model's attention weights converged on that outcome" is not an acceptable answer to a regulator.
  6. Privacy. Enterprise decisions involve PII, financial data, and health records. Sending sensitive data to a third-party LLM API is often not an option.
  7. Monitoring. Decision volumes, rule firing frequencies, exception rates — operational requirements, not afterthoughts.
  8. Cost. LLM inference at scale is expensive. Rule engine execution is cheap. The composite system's cost profile varies dramatically by pattern.

Every one of these criteria pushes in the same direction: the LLM should not be the decision-maker.

Why LLMs alone fail

LLMs "show impressive results and some reasoning capabilities" yet "fail as easily when repeating the experience." This is the architecture, not a bug. LLMs are probability distributions over token sequences. For creative tasks, variation is a feature. For mortgage decisions, it is a liability.

# The fundamental problem in one line:
# An LLM called 3 times on the same loan application may give 3 different answers.
for i in range(3):
    decision = llm.generate(f"Decide on this loan: {application}")
    print(decision)  # "Approved" → "Denied" → "Approved with conditions"

The rule engine guarantees correctness relative to its encoded rules. The LRM generates text that sounds like reasoning. In enterprise contexts, you need the first and want the second — which is exactly why you combine them.

Five patterns for composite AI

Pattern 1: NLU → Rules

An LLM comprehends unstructured text and extracts structured data; a rule engine reasons deterministically on that data.

// Pattern 1: NLU extracts, rules decide
func intakeClaim(description string) (Decision, error) {
    // Step 1: LLM extracts structured fields from free text
    structured, err := llm.Extract(description, ClaimSchema{
        Fields: []string{"incident_type", "fault_party", "damage_types", "date"},
    })
    if err != nil {
        return Decision{}, fmt.Errorf("extraction failed: %w", err)
    }

    // Step 2: Rule engine decides on structured data
    decision, err := ruleEngine.Decide("claims-coverage", map[string]interface{}{
        "incident_type": structured.IncidentType,
        "fault_party":   structured.FaultParty,
        "damage_types":  structured.DamageTypes,
        "policy_terms":  loadPolicy(structured.PolicyID),
    })
    if err != nil {
        return Decision{}, fmt.Errorf("decision failed: %w", err)
    }

    return decision, nil
}

What works. Sequential API calls. The LLM never makes a business decision. The rule engine never parses messy language.

What doesn't. When expected data is missing, the system needs guardrails. Does it reject? Clarify? Escalate?

The abstraction leaks at the schema boundary. Schema design is the hidden engineering work in Pattern 1.

Pattern 2: Rules → NLG

The flow reverses: a rule engine decides on structured data, then an LLM generates natural language.

// Pattern 2: Rules decide, LLM communicates
func notifyCustomer(application LoanApplication) (string, error) {
    // Step 1: Rule engine produces a structured decision
    decision, err := ruleEngine.Decide("loan-underwriting", map[string]interface{}{
        "amount":        application.Amount,
        "credit_score":  application.CreditScore,
        "dti_ratio":     application.DTIRatio,
        "collateral":    application.CollateralValue,
    })
    if err != nil {
        return "", err
    }

    // Step 2: LLM generates the customer letter from the decision
    letter, err := llm.Generate(LetterTemplate, map[string]interface{}{
        "decision":     decision.Outcome,    // "approved"
        "amount":       decision.Amount,     // 350000
        "rate":         decision.Rate,       // 6.25
        "conditions":   decision.Conditions, // ["income_verification", "appraisal"]
        "tone":         application.Channel, // "email_formal"
        "language":     application.Locale,  // "es-MX"
    })

    return letter, err
}

This is the safest pattern from a compliance perspective. The LLM never influences the decision — it only communicates it.

What doesn't. Testing NLG output is genuinely hard. The LLM may phrase the same decision in dozens of valid ways. You need semantic testing:

func TestNotificationLetter(t *testing.T) {
    decision := Decision{Outcome: "denied", Reason: "DTI exceeds threshold"}
    letter := generateLetter(decision)

    // Can't assert on exact string — assert on invariants
    if !strings.Contains(letter, "denied") {
        t.Error("letter must communicate denial")
    }
    if !strings.Contains(letter, "debt-to-income") {
        t.Error("letter must mention the reason")
    }
    if strings.Contains(letter, "approved") {
        t.Error("letter must not imply approval")
    }
}

Constrained generation — not free-form text — is the requirement. A Spanish-language denial letter that subtly softens rejection language creates compliance exposure.

Pattern 3: Rules orchestrate LLM

The rule engine is the master orchestrator, invoking LLMs on demand for delegated NLP tasks.

// Pattern 3: Rules drive the process, LLM is a tool called on demand
func adjudicateClaim(claim Claim) (Decision, error) {
    engine := rulego.NewRuleChain("claims-adjudication",
        // Step 1: Validate coverage
        rulego.NewTransformNode("validate-coverage").
            WithScript(`msg.Metadata.covered = msg.Data.policyActive && msg.Data.claimType == "covered"`),
        // Step 2: If covered, check amount threshold
        rulego.NewSwitchNode("threshold-check").
            WithCase("below_10k", "auto-approve").
            WithCase("above_10k", "fraud-review"),
        // Step 3: Invoke LLM for fraud review — only when threshold exceeded
        rulego.NewRestNode("fraud-review").
            WithEndpoint("https://api.llm.example/v1/chat").
            WithBodyTemplate(`Analyze for anomalies:
                Claim: {{.Data.description}}
                History: {{.Data.priorClaims}}
                Amount: {{.Data.claimAmount}}
                Respond with JSON: {"risk": "low|medium|high", "findings": [...]}`),
        // Step 4: Route based on LLM result
        rulego.NewSwitchNode("fraud-route").
            WithCase("low_risk", "auto-approve").
            WithCase("medium_risk", "manual-review").
            WithCase("high_risk", "investigate"),
    )
    return engine.Run(claim)
}

What works. Costs are proportional to actual need — not every transaction invokes the LLM. The rule engine remains in control.

The rule engine needs orchestration rules: when to call the LLM, what prompt to send, how to interpret the response, what fallback to use if the LLM returns nonsense.

What doesn't. The coupling is tight. The rule engine must mediate the structured-unstructured frontier — taking probabilistic LLM output and converting it into deterministic actions. These are not business rules. They are meta-rules governing the LLM interaction itself.

Pattern 4: LLM extracts rules

The most ambitious pattern: LLMs at design time extract automation assets from plain-text policy documents.

# Pattern 4: LLM reads policy, generates executable rules
policy_text = """
4.2.3 Loan Eligibility: Applicants with a credit score below 620
shall be denied. Applicants with credit score 620-699 and debt-to-income
ratio below 43% may be approved with standard rates. Applicants with
credit score 700+ and DTI below 36% qualify for preferred rates.
"""

rules = llm.extract(policy_text, format="decision_table")
# Output:
# [
#   { "when": "credit_score < 620", "then": "deny" },
#   { "when": "620 <= credit_score <= 699 AND dti < 43%", "then": "approve_standard" },
#   { "when": "credit_score >= 700 AND dti < 36%", "then": "approve_preferred" },
# ]

# Rules are validated, reviewed by humans, then deployed to the engine.
# The LLM is not in the runtime path. The rules execute deterministically.
for rule in rules:
    ruleEngine.addRule(rule, source=policy_text, paragraph="4.2.3")

Pattern 4 moves the LLM from the runtime path — where its latency, cost, and non-determinism are liabilities — to the development path, where its ability to process large volumes of unstructured text is an asset.

What doesn't. Prompt chains or fine-tuned models, companion tools for validation and synchronization, and a human review gate. The LLM turns weeks of manual rule writing into hours of review — but cannot replace expert judgment. The lifecycle problem — keeping extracted rules in sync with evolving source documents — persists long after initial extraction. This is where the KU Leuven research program enters (Part 3).

Pattern 5: Chatbot delegates to rules

An LLM drives the conversation; when a business decision is needed, the chatbot delegates to a rule engine.

// Pattern 5: Chatbot handles conversation, delegates decisions
func handleMortgageChat(session ChatSession, userMessage string) (string, error) {
    // Step 1: LLM understands the user's intent and extracts parameters
    intent, params, err := llm.UnderstandIntent(userMessage, MortgageIntents{
        Intents: []string{"rate_inquiry", "apply", "check_status", "general_question"},
        Slots:   []string{"loan_amount", "income", "credit_score", "property_value"},
    })

    // Step 2: If this is a decision trigger, delegate to rule engine
    if intent == "apply" && allRequiredFieldsPresent(params) {
        decision, err := ruleEngine.Decide("mortgage-eligibility", params)
        if err != nil {
            return llm.Generate("Something went wrong. Let me connect you with a specialist.", nil)
        }

        // Step 3: LLM restitutes the deterministic decision in natural language
        return llm.Generate("mortgage-decision-response", map[string]interface{}{
            "decision":     decision,
            "missing_info": missingFields(params),
            "next_steps":   nextStepsForDecision(decision),
            "tone":         session.UserPreferences.Tone,
        })
    }

    // Not a decision trigger — LLM handles conversation freely
    return llm.Chat(session.Context, userMessage)
}

The delegation boundary needs a formal contract: the rule engine exposes a decision service with a defined input schema, and the chatbot populates that schema conversationally. A missing required field is not an ambiguous conversational state — it is a slot in the schema that has not been filled.

Two hard problems. Decision trigger detection — when has the user crossed from browsing to deciding? Incomplete context — "my income is around 80K" when the engine needs an exact figure. The chatbot must ask, not fabricate.

FSMs with rule engines: process skeleton, decision muscle

The five Feillet patterns describe how to combine LLMs with rule engines. But there is an orthogonal architectural dimension that predates LLMs entirely: combining finite state machines with rule engines.

// The composed architecture in one structure
type MortgagePipeline struct {
    fsm    *StateMachine           // owns process state and transitions
    rules  map[string]*RuleEngine  // one rule engine per decision gate
}

func (p *MortgagePipeline) Process(app LoanApplication) error {
    for p.fsm.State != "closed" {
        state := p.fsm.State
        // Delegate to the rule engine for this state
        decision, err := p.rules[state].Decide(app.AccumulatedFacts())
        if err != nil {
            return err
        }
        // The decision determines the next transition
        p.fsm.Transition(decision.NextState)
    }
    return nil
}

The FSM manages where you are in the process. The rule engine manages what you know and what you should conclude. The LLM manages how you communicate at the boundaries. Three concerns. Three tools. One system.

The FSM vs. rule engine distinction is explored in depth in Part 4. The composed architecture — with full code, testing strategies, and audit patterns — is developed in Part 5.

Choosing a pattern

Boundary Pattern Signal
Unstructured → structured 1: NLU → Rules LLM extracts, rules decide
Structured → unstructured 2: Rules → NLG Rules decide, LLM communicates
Complex NLP orchestration 3: Rules drive LLM Rules call LLM on demand
Policy → code 4: LLM extracts rules LLM at design time only
Conversational decisions 5: Chatbot + Rules LLM talks, rules decide

The patterns compose. A real system might use Pattern 4 to extract rules during development, Pattern 1 for intake, Pattern 2 for communications, and Pattern 5 for the conversational interface — with Pattern 3 orchestrating complex processes where LLM calls are needed selectively.


References

  1. Pierre Feillet, Allen Chan, Luigi Pichett, Yazan Obeidi. Approaches in Using Generative AI for Business Automation. Medium, August 4, 2023.

  2. Pierre Feillet. Rule Engines Never Died — They're Running Alongside Your Large Reasoning Models. Medium, June 4, 2026.

  3. RuleGo — Lightweight, component-based rule engine for Go. Apache 2.0. Includes rulego-components-ai for LLM integration and MCP support.

Part 1: On Rule Engines — From RETE to MCP · Part 3: On Rule Engines — Automating Decision Models · Part 5: State Machines Powered by Rule Engines

On Rule Engines — Automating Decision Models

The KU Leuven research program on extracting DMN decision models from text using deep learning and LLMs, generating chatbots from decision models, and building explainable assistants — the academic foundation for Pattern 4 (LLM extracts rules) and Pattern 5 (chatbot delegation).

llmrulesenterpriseautomationcomposite-aidecision-makingsymbolic-aidmnresearch

The first essay traced the 40-year arc from Forgy's RETE algorithm to MCP. The second examined the five architectural patterns for blending LLMs with rule engines. This third essay turns to the academic research that validates, extends, and operationalizes those patterns.

While Pierre Feillet was developing composite AI patterns from enterprise practice at IBM, a sustained research program at KU Leuven's LIRIS (Leuven Institute for Research on Information Systems), led by Professor Jan Vanthienen and driven primarily by Alexandre Goossens and Vedavyas Etikala, was systematically attacking the same problem from the academic side.

Feillet's five patterns describe what to build. The KU Leuven papers describe how to build it.

Mapping the landscape (KSEM 2021)

The research program began with a survey. Etikala and Vanthienen's An Overview of Methods for Acquiring and Generating Decision Models (KSEM 2021) provided a taxonomy of techniques for acquiring decision models from various knowledge sources. Business decisions are of significant value — but manually modeling them is costly, tedious, and time-consuming.

The survey classified approaches along three dimensions: source type (text, legacy code, models, event logs), extraction target (dependencies, logic, full models), and technique family (rule-based NLP, traditional ML, deep learning). Deep learning approaches were largely unexplored for DMN extraction. The Gauntlet had been thrown.

First extraction results (BPM 2021)

Goossens, Claessens, Parthoens, and Vanthienen took the first step in Extracting Decision Dependencies and Decision Logic from Text Using Deep Learning Techniques (BPM 2021 Workshops). This was the first systematic attempt to apply deep learning specifically to DMN extraction.

The approach: collect a labeled dataset of sentences from real use cases, train two architectures — BERT and Bi-LSTM-CRF — for two tasks:

# Task 1: Sentence classification — does this sentence describe decision logic?
sentences = [
    ("Applicants with credit score below 620 shall be denied.", "decision_logic"),
    ("The loan officer reviews the application package.", "process_description"),
    ("DTI ratio is calculated as total monthly debt / gross monthly income.", "definition"),
]
classifier = FineTunedBERT(sentences, labels=["decision_logic", "process_description", "definition"])

# Task 2: Dependency extraction — which decisions depend on which?
# Input: "The eligibility decision depends on the credit assessment and the income verification."
# Output: eligibility -> [credit_assessment, income_verification]
extractor = BiLSTMCRF(dependency_sentences)

The results demonstrated sufficiently high performance to support (semi)-automatic extraction. A preliminary version appeared at RuleML+RR 2021 as Deep Learning for the Identification of Decision Modelling Components from Text.

The "semi" in semi-automatic was established from the start: extraction works, but human review remains essential.

Full DMN extraction (Expert Systems with Applications, 2023)

The definitive study came with Goossens, De Smedt, and Vanthienen's Extracting Decision Model and Notation Models from Text Using Deep Learning Techniques (Expert Systems with Applications, Vol. 211, 2023). Five contributions:

  1. First investigation of deep learning specifically for extracting DMN models from text
  2. Sentence classification for logic/dependency detection with high accuracy
  3. Dependency extraction from sentences — the structural backbone of a DMN model
  4. First labeled dataset made publicly available for decision model extraction research
  5. First extraction tool made available as open source

The extracted model looks like:

# DMN model extracted from policy text by BERT-based pipeline
decisions:
  - id: eligibility
    label: "Determine Loan Eligibility"
    dependencies: [credit_assessment, income_verification, collateral_check]
    logic:
      - when: "credit_score < 620"
        then: "deny"
      - when: "620 <= credit_score <= 699 AND dti_ratio < 43%"
        then: "approve_standard"
      - when: "credit_score >= 700 AND dti_ratio < 36%"
        then: "approve_preferred"

  - id: pricing
    label: "Determine Interest Rate"
    dependencies: [eligibility, market_conditions]
    logic:
      - when: "eligibility = approve_preferred AND ltv_ratio < 80%"
        then: "rate = base_rate - 0.5%"
      - when: "eligibility = approve_standard"
        then: "rate = base_rate + 0.25%"

The leap from "can we extract?" to "here is the tool and the dataset" is what makes this paper the landmark in the field.

GPT-3 enters the picture (RuleML+RR 2023)

Goossens, Vandevelde, Vanthienen, and Vennekens explored the next logical step in GPT-3 for Decision Logic Modeling (RuleML+RR 2023 Companion). Replace fine-tuned BERT with prompt-engineered GPT-3:

# Fine-tuned approach (BPM 2021, ESWA 2023):
# Requires labeled dataset, domain-specific training, high accuracy on known formats
model = FineTunedBERT.train(labeled_sentences, labels)
rules = model.extract(policy_text)

# Prompt-engineered approach (RuleML+RR 2023):
# No training data needed, general model, potentially lower structured accuracy
rules = llm.extract(policy_text, prompt="""
    Extract decision rules from the following policy text.
    Output as a decision table in JSON format.
    Each rule must include: when (conditions), then (conclusion).
    Link each rule to its source paragraph.
""")

The shift trades training cost for prompt engineering cost. No fine-tuning dataset needed — but structured extraction accuracy may be lower.

A companion presentation by Vanthienen and Goossens at DecisionCamp 2023, GPT-3 for Decision Requirements Modeling and Advice, extended this to decision requirements modeling.

Explainable assistants (BPM 2022)

Extracting a model is half the problem. Goossens, Maes, Timmermans, and Vanthienen's Automated Intelligent Assistance with Explainable Decision Models in Knowledge-Intensive Processes (BPM 2022 Workshops) asks: once you have a DMN model, how do you make it accessible?

They propose a generic intelligent assistant that can reason with any DMN model to provide explanations:

class DecisionAssistant:
    """Generic assistant: works with any DMN model."""
    def __init__(self, dmn_model: DMNModel):
        self.model = dmn_model

    def explain(self, decision_id: str, inputs: dict) -> Explanation:
        """Explain why a decision reached its conclusion."""
        trace = self.model.execute(decision_id, inputs)
        return Explanation(
            decision=trace.outcome,
            fired_rules=[step.rule for step in trace.steps],
            input_facts=trace.facts_used,
            reasoning_chain=[
                f"Rule {step.rule.id} fired because {step.rule.condition} matched {step.matched_facts}"
                for step in trace.steps
            ],
        )

An extracted DMN model paired with an explanation-capable assistant satisfies the regulatory requirement to show why a decision was made — not in post-hoc rationalization but in a traceable chain from facts through rules to conclusions.

Chatbots from decision models (RuleML+RR 2021)

Etikala, Goossens, Van Veldhoven, and Vanthienen close the loop in Automatic Generation of Intelligent Chatbots from DMN Decision Models (RuleML+RR 2021). Their framework generates a chatbot directly from a DMN model's structure:

# A DMN model becomes a conversational interface automatically
dmn = DMNModel.load("mortgage-eligibility.dmn")

chatbot = ChatbotGenerator(dmn).generate()
# Generated chatbot behavior:
#   Slot 1: "What is the loan amount?"        → dmn.inputs.loan_amount
#   Slot 2: "What is your annual income?"     → dmn.inputs.applicant_income
#   Slot 3: "What is your credit score?"       → dmn.inputs.credit_score
#   Slot 4: "What is the property value?"      → dmn.inputs.property_value
#   --- all required inputs gathered ---
#   Invoke: dmn.decide("eligibility", inputs)
#   Response: "Based on your credit score of 720 and DTI of 32%,
#              you qualify for preferred rates at 6.0%."

# The DMN schema IS the conversation contract

A missing required field is not an ambiguous conversational state. It is a slot in the DMN input schema that has not been filled, and the chatbot knows it needs to ask for it.

This solves the two hard problems Feillet identified for Pattern 5: the DMN input schema provides the formal delegation contract, and missing required fields are unambiguously identifiable slots to ask about.

The research arc

Stage Paper Contribution
Survey Etikala & Vanthienen (KSEM 2021) Taxonomy of acquisition methods
Feasibility Goossens et al. (BPM 2021) First deep learning DMN extraction
Scale Goossens, De Smedt, Vanthienen (ESWA 2023) Full extraction, open dataset and tools
Modernize Goossens et al. (RuleML+RR 2023) GPT-3 for decision logic modeling
Explain Goossens et al. (BPM 2022) Explainable assistant from any DMN model
Converse Etikala et al. (RuleML+RR 2021) Chatbots from DMN models

RuleGo: an open-source implementation path

The patterns and research are not confined to academic papers and enterprise platforms. RuleGo provides a concrete implementation path:

// Pattern 4 implemented with RuleGo: LLM generates RuleGo chain JSON from policy
policyText := readPolicy("underwriting-policy-2026.txt")
chainJSON := llm.Extract(policyText, PromptConfig{
    Format: "rulego_chain",
    Schema: rulego.ChainSchema,
})

// The generated chain runs deterministically, LLM-free at runtime
chain := rulego.LoadChain(chainJSON)
decision := chain.Run(loanApplication)

// Generated chain structure:
// {
//   "ruleChain": {
//     "nodes": [
//       {"id": "credit-check", "type": "switch",
//        "cases": [
//          {"when": "credit_score < 620", "then": "deny"},
//          {"when": "credit_score >= 620 && credit_score <= 699", "then": "dti-check"},
//          {"when": "credit_score >= 700", "then": "preferred-check"}
//        ]},
//       {"id": "dti-check", "type": "switch", ...},
//       {"id": "preferred-check", "type": "switch", ...}
//     ]
//   }
// }

Open questions

The extraction quality bar. At what accuracy threshold does the economics flip? At 90%, a human reviews every rule. At 95%? At 99%? The savings come from turning a writing task into a reviewing task — but the threshold where you stop reviewing every rule is where the operational gains live.

Rule lifecycle management. When source documents change, extracted rules must change. Governed policy evolution, versioned extraction, conflict detection between old and new rules — this synchronization problem is where the next wave of research needs to go.

Testing composite systems. How do you test a system where one component is deterministic and the other probabilistic? Property-based testing: invariants that must hold regardless of surface variation.

Vendor neutrality. The patterns are general but the implementations assume IBM products. RuleGo demonstrates one open-source path, but the interfaces between components are not yet standardized.

The MCP interface standard. How does an LRM discover available decision services? What information passes between them? How are partial results and confidence signals communicated?

The composite AI thesis

The central thesis running through this series: the future of enterprise AI is composite. LLMs handle the perception layer — unstructured text, intents, entities, fluency. Rule engines, built on Forgy's insight from 1979, handle the reasoning layer — deterministic logic, auditable decisions, regulatory compliance. The FSM handles the process layer — sequencing decisions through states and transitions.

Let the neuronal system handle the messiness of natural language. Let the symbolic system handle the precision of business logic. Let the state machine handle the process that connects them.

A decision system that cannot explain itself is not enterprise-grade. A decision system that cannot handle ambiguity is not useful. The composite approach accepts both constraints and designs for them. That has been the engineering move since Forgy built the first discrimination network in 1979.


References

  1. Pierre Feillet, Allen Chan, Luigi Pichett, Yazan Obeidi. Approaches in Using Generative AI for Business Automation. Medium, August 4, 2023.

  2. Pierre Feillet. Rule Engines Never Died — They're Running Alongside Your Large Reasoning Models. Medium, June 4, 2026.

  3. Alexandre Goossens, Johannes De Smedt, Jan Vanthienen. Extracting Decision Model and Notation Models from Text Using Deep Learning Techniques. Expert Systems with Applications, 211: 118667, 2023.

  4. Alexandre Goossens, Simon Vandevelde, Jan Vanthienen, Joost Vennekens. GPT-3 for Decision Logic Modeling. RuleML+RR Companion, CEUR Vol. 3485, 2023.

  5. Alexandre Goossens, Ulysse Maes, Yves Timmermans, Jan Vanthienen. Automated Intelligent Assistance with Explainable Decision Models in Knowledge-Intensive Processes. BPM Workshops 2022, LNBIP 460, pp. 25–36.

  6. Alexandre Goossens, Michelle Claessens, Charlotte Parthoens, Jan Vanthienen. Extracting Decision Dependencies and Decision Logic from Text Using Deep Learning Techniques. BPM Workshops 2021, LNBIP 436, pp. 349–361.

  7. Vedavyas Etikala, Jan Vanthienen. An Overview of Methods for Acquiring and Generating Decision Models. KSEM 2021, LNCS 12817, pp. 200–208.

  8. Vedavyas Etikala, Alexandre Goossens, Ziboud Van Veldhoven, Jan Vanthienen. Automatic Generation of Intelligent Chatbots from DMN Decision Models. RuleML+RR 2021, LNCS 12851, pp. 142–157.

  9. Jan Vanthienen, Alexandre Goossens. GPT-3 for Decision Requirements Modeling and Advice. DecisionCamp 2023.

  10. RuleGo — Lightweight, component-based rule engine for Go. Apache 2.0. Includes rulego-components-ai for LLM integration and MCP support.

Part 1: On Rule Engines — From RETE to MCP · Part 2: On Rule Engines — Five Patterns for Composite AI · Part 4: On Rule Engines — State Machines vs. Rule Engines

On Rule Engines — State Machines vs. Rule Engines

A deep dive into the architectural distinction between finite state machines and rule engines — when to use each, anti-patterns that arise from using the wrong one, and a decision framework for choosing. With code.

llmrulesfsmstate-machinesarchitectureenterprisedesign

The first essay in this series introduced the distinction between finite state machines and rule engines. This essay — the fourth in a five-part series — develops that distinction in depth. Expect code, anti-patterns, and a framework for choosing.

Every system architect should be able to answer one question cold: is this a job for an FSM or a rule engine?

Two questions, two tools

A finite state machine answers what happens next? It is procedural. You define states. You define transitions between them. The machine is always in exactly one state. When an event arrives, the machine consults its transition table and moves — or stays put.

A rule engine answers what should we conclude? It is declarative. You define condition-action pairs. The engine evaluates all rules against a working memory of facts. Any rule whose conditions are satisfied is eligible to fire. The engine — not the author — determines evaluation order through its conflict resolution strategy.

// FSM: you define what happens next
type ClaimFSM struct {
    state string
    transitions map[string]map[string]string
}

func NewClaimFSM() *ClaimFSM {
    return &ClaimFSM{
        state: "intake",
        transitions: map[string]map[string]string{
            "intake":       {"submit": "review", "withdraw": "closed"},
            "review":       {"approve": "payment", "reject": "closed", "need_info": "intake"},
            "payment":      {"complete": "closed"},
        },
    }
}

// Rule engine: you define what should be concluded
rules := []Rule{
    {When: "claim.amount > 10000 AND claim.type = 'injury'", Then: Action("flag", "senior_review")},
    {When: "claimant.prior_claims > 3 AND claim.type = 'property'", Then: Action("flag", "fraud_check")},
    {When: "policy.active = false", Then: Action("deny", "coverage_lapsed")},
}

The FSM asks: given where I am and what just happened, where do I go? The rule engine asks: given everything I know, what should I conclude?

The distinction, in detail

FSM Rule Engine
Core question What happens next? What should we conclude?
Paradigm Procedural Declarative
State Explicit states + transition table Working memory of facts
Control flow Defined by transition graph Defined by rule firing (inference + agenda)
Author thinks about States, events, transitions Conditions, actions, conflict resolution
Best for Workflows, protocols, pipelines Policies, decisions, classifications
Complexity driver States × transitions Rules × fact combinations
Determinism Deterministic given state + event Deterministic given rule set + facts
Auditability Trace: state sequence + transitions taken Trace: fired rules + matched facts

The distinction is not academic. Using the wrong tool produces systems that work but are impossible to maintain.

Anti-pattern 1: Encoding policy as an FSM

Consider a loan eligibility policy: credit score bands, debt-to-income thresholds, collateral requirements, regulatory jurisdictions. Encode this as an FSM:

// ANTI-PATTERN: Policy encoded as states
// Each combination of conditions becomes a state. Explosion ensues.
type LoanPolicyFSM struct {
    creditScore int
    dti         float64
    state       string
}

func (f *LoanPolicyFSM) evaluate() string {
    // What should be 3 rules becomes a combinatorial state explosion
    switch {
    case f.creditScore < 620:
        return "deny"
    case f.creditScore >= 620 && f.creditScore <= 699 && f.dti < 0.43:
        return "approve_standard"
    case f.creditScore >= 700 && f.dti < 0.36:
        return "approve_preferred"
    // Add collateral rules? Multiply states by collateral types.
    // Add jurisdiction rules? Multiply by 50 states × federal regs.
    // The state space is the Cartesian product of all condition dimensions.
    }
    return "manual_review"
}

Two conditions produce 3 outcomes. Three conditions produce 9. Five conditions with regulatory variation produce hundreds. The FSM approach scales exponentially with condition dimensions.

The fix: policy belongs in a rule engine.

// CORRECT: Policy as rules — each dimension adds rules, not states
rules := []Rule{
    {When: "credit_score < 620", Then: Action("deny")},
    {When: "credit_score >= 620 AND credit_score <= 699 AND dti < 0.43", Then: Action("approve", "standard")},
    {When: "credit_score >= 700 AND dti < 0.36", Then: Action("approve", "preferred")},
    {When: "collateral.value < loan.amount * 0.8 AND loan.type = 'unsecured'",
     Then: Action("require", "additional_collateral")},
    // Adding a new condition dimension adds one rule, not a Cartesian product
    {When: "jurisdiction = 'CA' AND loan.amount > 500000",
     Then: Action("require", "california_disclosure")},
}
engine := NewRuleEngine(rules)
engine.Insert(loanFacts)
decision := engine.Run()

Policy dimensions add rules linearly. States multiply transitions exponentially. That is the entire argument for using the right tool.

Anti-pattern 2: Encoding workflow as flat rules

Now the inverse. Consider a claims processing pipeline: intake → review → investigation → payment → close. Encode this as flat rules:

# ANTI-PATTERN: Workflow smuggled through working memory facts
rules = [
    Rule("step = 'intake' AND form.complete = true", actions=[
        Action("validate", "form"),
        Action("set", "step", "review"),       # process state as a fact!
    ]),
    Rule("step = 'review' AND claim.amount < 1000", actions=[
        Action("set", "step", "payment"),       # implicit transition
    ]),
    Rule("step = 'review' AND claim.amount >= 1000", actions=[
        Action("set", "step", "investigation"), # implicit transition
    ]),
    Rule("step = 'investigation' AND fraud_check.complete = true", actions=[
        Action("set", "step", "payment"),       # implicit transition
        Action("set", "fraud_flag", fraud_check.result),
    ]),
    Rule("step = 'payment' AND payment.processed = true", actions=[
        Action("set", "step", "closed"),        # implicit transition
    ]),
]

What is wrong with this? Everything.

  1. The process state is smuggled. step is a working memory fact, not a first-class state. Nothing guarantees only one step fact exists. Nothing prevents contradictory transitions.
  2. Transitions are implicit. The author intended intake → review → payment → closed. But the rules don't express this as a graph. You cannot look at the rule set and see the process.
  3. Adding a state requires discipline. To add a "fraud_check" state between investigation and payment, you must update every rule that references step = 'investigation' and step = 'payment'. Miss one and you have a bug that only manifests when a specific combination of facts triggers the stale transition.
  4. Testing is combinatorial. Each rule depends on step plus domain facts. To test the payment transition, you must set up facts that satisfy both the step condition and all other conditions in the rule.

The process is there, but it is encoded indirectly through fact manipulation. You can read the rule set and not see the state machine. That is the definition of implicit.

The fix: workflow belongs in an FSM.

# CORRECT: Process as explicit states and transitions
class ClaimProcess:
    states = ["intake", "review", "investigation", "payment", "closed"]
    transitions = {
        "intake":       {"validated": "review", "incomplete": "intake"},
        "review":       {"low_value": "payment", "needs_review": "investigation"},
        "investigation": {"cleared": "payment", "flagged": "closed"},
        "payment":      {"processed": "closed"},
    }

    def __init__(self):
        self.state = "intake"

    def handle(self, event):
        if event in self.transitions[self.state]:
            self.state = self.transitions[self.state][event]
            return self.state
        raise InvalidTransition(self.state, event)

If you find yourself writing step = 'something' in every rule condition, you are encoding a state machine in a rule engine. Stop. Use an FSM.

The decision framework

When choosing between an FSM and a rule engine, ask three questions:

1. Is the problem primarily about sequence or about conditions?

Sequence-driven → FSM. Condition-driven → rule engine.

A claims process is about sequence: intake, then review, then investigation, then payment. A loan eligibility policy is about conditions: credit score, DTI, collateral value.

// Sequence-driven: the order of states IS the business logic
// Process: intake → review → underwriting → approval → closing
// You cannot skip review. You cannot go back from closing.
fsm := NewFSM() // correct choice

// Condition-driven: the combination of facts IS the business logic
// Policy: if credit > 700 AND DTI < 36% AND LTV < 80% → preferred rate
// The order of evaluating credit, DTI, and LTV does not matter.
engine := NewRuleEngine() // correct choice

2. Does the complexity grow with states or with conditions?

If adding a new business rule means adding a state → you are in the wrong tool. If adding a new process step means updating multiple rules → you are in the wrong tool.

// Complexity smell: adding a condition dimension explodes states
// Adding "jurisdiction" to a loan FSM: 50 states × existing states = explosion
// Adding "jurisdiction" to a rule engine: one rule, maybe a decision table

// Complexity smell: adding a process step touches many rules
// Adding "fraud_check" to flat rules: update every rule with step guards
// Adding "fraud_check" to an FSM: add one state, two transitions

3. Will an auditor need to trace the process or explain the decision?

Process trace → FSM. Decision explanation → rule engine. Both → compose them (Part 5).

// FSM audit: state sequence
// "Application reached underwriting via: intake → review → underwriting"
fsm.AuditTrail() // [intake, review, underwriting]

// Rule engine audit: rule trace
// "Decision: deny. Rules fired: R-17 (credit < 620), R-23 (DTI > 43%)"
engine.Explanation() // [{rule: R-17, reason: "credit_score=590 < 620"}, ...]

When the problem is both

Most real business problems are both procedural and decisional. A mortgage origination system has a process (application → review → underwriting → approval → closing) and decisions at each gate (eligibility rules, pricing rules, compliance rules).

The mistake is picking one tool and forcing the other concern into it. The solution — developed in Part 5 — is composition: FSM as process skeleton, rule engine as decision muscle.

[Application] → [Review] → [Underwriting] → [Approval] → [Closing]
                    |            |               |
                    v            v               v
               Rule Engine   Rule Engine    Rule Engine
               (completeness (credit risk,   (final conditions,
                check)        collateral)     compliance)

The FSM manages where you are. The rule engine manages what you conclude. Neither is asked to do the other's job. The composite handles both.


References

  1. Charles L. Forgy. Rete: A Fast Algorithm for the Many Pattern/Many Object Pattern Match Problem. Artificial Intelligence, 19(1): 17–37, 1982.

  2. RuleGo — Lightweight, component-based rule engine for Go. Apache 2.0.

Part 1: On Rule Engines — From RETE to MCP · Part 5: On Rule Engines — State Machines Powered by Rule Engines

On Rule Engines — State Machines Powered by Rule Engines

The composed architecture — FSM as process skeleton, rule engine as decision muscle. A complete mortgage origination pipeline with Go code, testing strategies, layered audit trails, and LLM composition at the boundaries.

llmrulesfsmstate-machinesarchitectureenterprisegolangtestingaudit

Part 4 established the distinction: FSMs handle sequence, rule engines handle decisions. This essay — the final in a five-part series — develops the composed architecture in full. An FSM manages process state and transitions. At each state where a decision is required, the FSM delegates to a rule engine. The LLM handles natural language at the boundaries. Three concerns. Three tools. One system.

The FSM knows where you are. The rule engine knows what to conclude. The LLM knows how to say it. Each does one thing. Together they do everything.

The mortgage origination pipeline

A mortgage application moves through five states. At each state gate, a rule engine makes a decision. The decision determines the transition.

[Intake] ──→ [Review] ──→ [Underwriting] ──→ [Approval] ──→ [Closing]
   │              │              │                  │              │
   v              v              v                  v              v
Rule Engine   Rule Engine   Rule Engine        Rule Engine    Rule Engine
(completeness (document     (credit risk,       (final         (funding
 check)        verification) collateral, DTI)    conditions)    verification)

The domain types

type LoanApplication struct {
    ID            string
    ApplicantID   string
    Amount        float64
    PropertyValue float64
    CreditScore   int
    AnnualIncome  float64
    MonthlyDebt   float64
    LoanType      string // "conventional", "fha", "va", "jumbo"
    Documents     []Document
    State         string // mirrors FSM state for persistence
}

type Decision struct {
    Outcome   string            // "approve", "deny", "need_info", "escalate"
    NextState string            // the FSM transition target
    Data      map[string]any    // structured decision payload
    Rules     []RuleTrace       // which rules fired and why
    Timestamp time.Time
}

type RuleTrace struct {
    RuleID      string
    Condition   string
    MatchedFacts map[string]any
}

The FSM

type MortgageFSM struct {
    state       string
    transitions map[string]map[string]string
    onEnter     map[string]func(*LoanApplication) error
    onExit      map[string]func(*LoanApplication) error
}

func NewMortgageFSM() *MortgageFSM {
    return &MortgageFSM{
        state: "intake",
        transitions: map[string]map[string]string{
            "intake": {
                "complete":   "review",
                "incomplete": "intake",
                "withdraw":   "closed",
            },
            "review": {
                "verified":     "underwriting",
                "missing_docs": "review",
                "deny":         "closed",
            },
            "underwriting": {
                "approve":      "approval",
                "deny":         "closed",
                "need_info":    "review",
            },
            "approval": {
                "conditions_met": "closing",
                "conditions_failed": "underwriting",
                "withdraw": "closed",
            },
        },
        onEnter: map[string]func(*LoanApplication) error{
            "intake":       validateApplication,
            "underwriting": lockRate,
            "closing":      generateClosingDocs,
        },
    }
}

func (f *MortgageFSM) Transition(app *LoanApplication, event string) error {
    target, ok := f.transitions[f.state][event]
    if !ok {
        return fmt.Errorf("invalid transition: %s --%s--> ?", f.state, event)
    }

    if fn, ok := f.onExit[f.state]; ok {
        if err := fn(app); err != nil {
            return fmt.Errorf("onExit %s: %w", f.state, err)
        }
    }

    f.state = target
    app.State = target

    if fn, ok := f.onEnter[target]; ok {
        if err := fn(app); err != nil {
            return fmt.Errorf("onEnter %s: %w", target, err)
        }
    }

    return nil
}

The FSM is a library concern — small, testable, and independent of both rule logic and LLM integration.

The rule engines

Each decision gate has its own rule engine, scoped to the decision at that state:

type MortgagePipeline struct {
    fsm   *MortgageFSM
    gates map[string]*RuleEngine // one engine per decision state
}

func NewMortgagePipeline() *MortgagePipeline {
    return &MortgagePipeline{
        fsm: NewMortgageFSM(),
        gates: map[string]*RuleEngine{
            "intake":       NewIntakeRules(),
            "review":       NewReviewRules(),
            "underwriting": NewUnderwritingRules(),
            "approval":     NewApprovalRules(),
            "closing":      NewClosingRules(),
        },
    }
}

func NewUnderwritingRules() *RuleEngine {
    return NewRuleEngine([]Rule{
        {
            ID:      "UW-01",
            When:    "credit_score < 620",
            Then:    Action{Outcome: "deny", NextState: "deny", Reason: "credit_score_minimum"},
        },
        {
            ID:      "UW-02",
            When:    "dti_ratio > 0.43 AND loan_type != 'va'",
            Then:    Action{Outcome: "deny", NextState: "deny", Reason: "dti_exceeds_threshold"},
        },
        {
            ID:      "UW-03",
            When:    "ltv_ratio > 0.95 AND loan_type = 'conventional'",
            Then:    Action{Outcome: "deny", NextState: "deny", Reason: "insufficient_equity"},
        },
        {
            ID:      "UW-04",
            When:    "credit_score >= 700 AND dti_ratio < 0.36 AND ltv_ratio < 0.80",
            Then:    Action{Outcome: "approve", NextState: "approve", Rate: "preferred", Adjustment: -0.25},
        },
        {
            ID:      "UW-05",
            When:    "credit_score >= 620 AND dti_ratio < 0.43 AND ltv_ratio < 0.95",
            Then:    Action{Outcome: "approve", NextState: "approve", Rate: "standard"},
        },
        {
            ID:      "UW-06",
            When:    "documents MISSING 'tax_returns' OR documents MISSING 'pay_stubs'",
            Then:    Action{Outcome: "need_info", NextState: "need_info", MissingDocs: []string{"tax_returns", "pay_stubs"}},
        },
    })
}

Each gate's rule engine knows only its own domain. The intake engine checks completeness. The underwriting engine evaluates credit risk. The approval engine applies final conditions. Changes to one gate's rules never affect another gate.

The pipeline

func (p *MortgagePipeline) Process(app *LoanApplication) (*Decision, []AuditEntry, error) {
    var audit []AuditEntry

    for p.fsm.State() != "closed" && p.fsm.State() != "deny" {
        state := p.fsm.State()
        engine, ok := p.gates[state]
        if !ok {
            return nil, audit, fmt.Errorf("no rule engine for state: %s", state)
        }

        // Gather facts accumulated so far
        facts := app.AccumulatedFacts()

        // Delegate to the rule engine for this state gate
        decision, err := engine.Decide(facts)
        if err != nil {
            return nil, audit, fmt.Errorf("decision at %s: %w", state, err)
        }

        // Record audit entry
        audit = append(audit, AuditEntry{
            State:      state,
            Decision:   decision.Outcome,
            RulesFired: decision.Rules,
            Transition: decision.NextState,
            Timestamp:  decision.Timestamp,
        })

        // The rule engine's decision drives the FSM transition
        if err := p.fsm.Transition(app, decision.NextState); err != nil {
            return nil, audit, fmt.Errorf("transition from %s: %w", state, err)
        }

        // If this was a terminal decision, return it
        if decision.Outcome == "deny" || decision.Outcome == "approve" {
            return &decision, audit, nil
        }
    }

    return nil, audit, fmt.Errorf("pipeline exited without terminal decision")
}

The FSM calls the rule engine. The rule engine returns a decision with a NextState. The FSM transitions. The loop continues until a terminal state. This is the entire architecture.

Testing the composite

Each component tests independently. The composition tests with mocks.

Testing the FSM in isolation

func TestMortgageFSM_Transitions(t *testing.T) {
    fsm := NewMortgageFSM()

    // Happy path: intake → review → underwriting → approval → closing
    app := &LoanApplication{ID: "LOAN-001"}

    if err := fsm.Transition(app, "complete"); err != nil {
        t.Fatal(err)
    }
    if fsm.State() != "review" {
        t.Errorf("expected review, got %s", fsm.State())
    }

    // Invalid transition: can't jump from review to closing
    if err := fsm.Transition(app, "conditions_met"); err == nil {
        t.Error("expected error for invalid transition review → closing")
    }
}

func TestMortgageFSM_InvalidTransitions(t *testing.T) {
    fsm := NewMortgageFSM()
    app := &LoanApplication{ID: "LOAN-002"}

    // Cannot approve from intake
    if err := fsm.Transition(app, "approve"); err == nil {
        t.Error("intake --approve--> ? should be invalid")
    }
}

Testing a rule engine in isolation

func TestUnderwritingRules_CreditDenial(t *testing.T) {
    engine := NewUnderwritingRules()

    decision, err := engine.Decide(map[string]any{
        "credit_score": 590,
        "dti_ratio":    0.30,
        "ltv_ratio":    0.75,
        "loan_type":    "conventional",
    })

    if err != nil {
        t.Fatal(err)
    }
    if decision.Outcome != "deny" {
        t.Errorf("expected deny for credit_score=590, got %s", decision.Outcome)
    }
    if decision.NextState != "deny" {
        t.Errorf("expected next_state=deny, got %s", decision.NextState)
    }
}

func TestUnderwritingRules_PreferredRate(t *testing.T) {
    engine := NewUnderwritingRules()

    decision, err := engine.Decide(map[string]any{
        "credit_score": 720,
        "dti_ratio":    0.32,
        "ltv_ratio":    0.75,
        "loan_type":    "conventional",
    })

    if err != nil {
        t.Fatal(err)
    }
    if decision.Outcome != "approve" {
        t.Errorf("expected approve, got %s", decision.Outcome)
    }
    if decision.Rate != "preferred" {
        t.Errorf("expected preferred rate for 720 credit, got %s", decision.Rate)
    }
}

Testing the pipeline end-to-end

func TestPipeline_HappyPath(t *testing.T) {
    pipeline := NewMortgagePipeline()
    app := &LoanApplication{
        ID:            "LOAN-003",
        Amount:        350000,
        PropertyValue: 450000,
        CreditScore:   720,
        AnnualIncome:  120000,
        MonthlyDebt:   3200,
        LoanType:      "conventional",
        Documents:     []Document{
            {Type: "tax_returns", Status: "verified"},
            {Type: "pay_stubs", Status: "verified"},
        },
    }

    decision, audit, err := pipeline.Process(app)
    if err != nil {
        t.Fatal(err)
    }
    if decision.Outcome != "approve" {
        t.Errorf("expected approve, got %s", decision.Outcome)
    }

    // Verify the process trace
    expectedStates := []string{"intake", "review", "underwriting", "approval", "closing"}
    for i, entry := range audit {
        if entry.State != expectedStates[i] {
            t.Errorf("step %d: expected state %s, got %s", i, expectedStates[i], entry.State)
        }
    }
}

func TestPipeline_LowCredit(t *testing.T) {
    pipeline := NewMortgagePipeline()
    app := &LoanApplication{
        CreditScore: 580,
        // ... other fields ...
    }

    decision, audit, err := pipeline.Process(app)
    if err != nil {
        t.Fatal(err)
    }
    if decision.Outcome != "deny" {
        t.Errorf("expected deny for credit 580, got %s", decision.Outcome)
    }

    // Verify the audit trail shows which rule fired
    lastEntry := audit[len(audit)-1]
    if lastEntry.State != "underwriting" {
        t.Errorf("expected denial at underwriting gate, got %s", lastEntry.State)
    }
}

Three layers of testing. FSM in isolation: does it transition correctly? Rule engine in isolation: does it decide correctly? Pipeline end-to-end: does the composition work? Each layer can be tested independently. The combinatorial explosion of testing all state × rule combinations is avoided by testing the FSM with mocked decisions and the rule engine with canned facts.

The layered audit trail

A regulator asks: "Why was loan LOAN-004 denied?" The composite system produces a layered answer.

type FullAuditReport struct {
    ApplicationID string
    ProcessTrace  []ProcessStep    // FSM layer: where were we and when?
    DecisionTrace []DecisionRecord // Rule engine layer: what did we conclude and why?
    LLMTrace      []LLMRecord      // LLM layer: what did the model generate?
}

func (p *MortgagePipeline) FullAudit(app *LoanApplication) (*FullAuditReport, error) {
    decision, audit, err := p.Process(app)
    if err != nil {
        return nil, err
    }

    report := &FullAuditReport{ApplicationID: app.ID}

    for _, entry := range audit {
        // Process layer: FSM trace
        report.ProcessTrace = append(report.ProcessTrace, ProcessStep{
            State:     entry.State,
            EnteredAt: entry.Timestamp,
            Event:     entry.Transition,
        })

        // Decision layer: rule trace
        for _, rule := range entry.RulesFired {
            report.DecisionTrace = append(report.DecisionTrace, DecisionRecord{
                State:        entry.State,
                RuleID:       rule.RuleID,
                Condition:    rule.Condition,
                MatchedFacts: rule.MatchedFacts,
            })
        }
    }

    return report, nil
}

The output is a traceable chain: process trace (which states, in which order), decision trace (which rules fired, against which facts), and — when LLMs are composed at the boundaries — generation trace (what was communicated, in which words).

LOAN-004 Audit Report
=====================
Process Trace:
  intake(09:15) --complete--> review
  review(09:22) --verified--> underwriting
  underwriting(09:23) --deny--> closed

Decision Trace:
  underwriting: UW-01 fired — credit_score=590 < threshold=620
  underwriting: UW-03 fired — ltv_ratio=0.97 > maximum=0.95

Conclusion: deny. Reasons: [credit_score_minimum, insufficient_equity]

The process says where. The rules say why. The audit combines both without conflating them.

Composing with LLMs at the boundaries

The FSM + rule engine architecture composes naturally with Feillet's five patterns from Part 2. The LLM handles natural language at intake and notification. The FSM and rule engines handle the process and decisions between them.

type MortgageSystem struct {
    pipeline *MortgagePipeline
    llm      *LLMClient
}

func (s *MortgageSystem) HandleInquiry(session ChatSession, message string) (string, error) {
    // Pattern 1: NLU at intake — LLM extracts structured fields
    intent, params, err := s.llm.UnderstandIntent(message, MortgageSchema)
    if err != nil {
        return s.llm.Generate("clarify", nil)
    }

    if intent == "apply" {
        app := params.ToLoanApplication()

        // FSM + Rule Engine: deterministic core
        decision, audit, err := s.pipeline.Process(app)
        if err != nil {
            return s.llm.Generate("error", map[string]any{"error": err})
        }

        // Pattern 2: NLG at notification — LLM communicates the decision
        return s.llm.Generate("mortgage_decision", map[string]any{
            "decision": decision,
            "audit":    audit,
            "tone":     session.UserPreferences.Tone,
            "language": session.UserPreferences.Locale,
        })
    }

    // Non-decision conversation: LLM handles freely
    return s.llm.Chat(session.Context, message)
}

The LLM handles the messiness of human language. The FSM handles the discipline of process state. The rule engine handles the precision of business logic. Each does one thing. Each is testable independently. The composition handles everything.

The design principles

1. Separate process from policy. The FSM encodes the sequence of states. The rule engine encodes the conditions for decisions. Never smuggle process state through working memory facts. Never encode branching policy logic in transition guards.

2. One rule engine per decision gate. Each gate's rules are scoped to that decision. Changing underwriting rules never affects intake rules. This is the single-responsibility principle applied to decision automation.

3. Test in layers. FSM with mocked decisions. Rule engine with canned facts. Pipeline end-to-end for integration. The combinatorial explosion of testing all state × rule combinations is avoided by layer isolation.

4. Audit in layers. Process trace (FSM) + decision trace (rule engine) + communication trace (LLM) = complete auditability. Each layer is independently queryable. Combined, they tell the full story.

5. The LLM is a boundary component. It handles natural language at intake and notification. It never makes a business decision. It never manages process state. The deterministic core — FSM + rule engines — is LLM-free at runtime.

The composite architecture is not a compromise. It is the recognition that no single tool handles all three concerns well. Use the FSM for process. Use the rule engine for policy. Use the LLM for language. The system that does all three is the system that survives.


References

  1. Charles L. Forgy. Rete: A Fast Algorithm for the Many Pattern/Many Object Pattern Match Problem. Artificial Intelligence, 19(1): 17–37, 1982.

  2. Pierre Feillet, Allen Chan, Luigi Pichett, Yazan Obeidi. Approaches in Using Generative AI for Business Automation. Medium, August 4, 2023.

  3. RuleGo — Lightweight, component-based rule engine for Go. Apache 2.0.

Part 1: On Rule Engines — From RETE to MCP · Part 4: On Rule Engines — State Machines vs. Rule Engines

I, Lighter

The disposable plastic lighter costs 15 cents, contains over 30 precision-engineered parts, holds pressurized gas, passes international safety compliance, and is shipped 10,000 miles across oceans — all for pocket change. One city in China produces 70% of the world's supply, earning less than a cent of profit per unit. This is not cheap junk. This is ruthless engineering grit, applied to the absolute physical and economic limit.

i-lightershaodongmanufacturingengineering-economicsmicro-marginsindustrial-clustersi-pencil

We have all held one. In a restaurant, lighting a candle. At a chai stall, the vendor flicking it absent-mindedly between customers. In your own pocket as you read this sentence. You lose them. You replace them. You barely give them a second thought. The disposable plastic lighter. Fifteen cents. The most undervalued engineering artifact on Earth.

In 1958, Leonard Read wrote "I, Pencil" — an essay narrated by a pencil, cataloguing the millions of specialized hands that contribute to its creation, none of whom know how to make a pencil. The pencil was the humble object that revealed the miracle of distributed knowledge. The lighter deserves the same treatment. But where the pencil's miracle is coordination without a coordinator, the lighter's miracle is optimization without a floor. The pencil teaches us that markets coordinate knowledge. The lighter teaches us that markets can push optimization to the thousandth of a cent, and keep pushing for twenty years without the price moving. The pencil is the miracle of spontaneous order. The lighter is the miracle of relentless compression.

The numbers

One city. Shaodong, a county-level city in Hunan Province, China. Population: roughly one million. It produces approximately 100 billion disposable lighters per year — about 70% of the world's supply. Laid end to end, the annual output would circle the Earth more than thirty times. Exported to over 120 countries. The industry employs over 80,000 people, about 7.6% of the local population. There are 114 lighter-related companies: 27 finished-product manufacturers, the rest producing components and materials. All within a 20-kilometer radius.

The retail price of the basic disposable lighter has held at one yuan — about 15 US cents — for over twenty years. Twenty years. Think about what else costs the same as it did in 2004. Nothing. Raw materials have risen. Labor has risen. Shipping has risen. The lighter costs the same. This is not normal. This is not what markets usually produce. Markets usually let prices float with costs. Shaodong didn't let the price float. Shaodong compressed every other variable to hold the price constant. The price is the constraint. Everything else gave way.

What's inside

A disposable lighter contains over 30 individual components: the gas valve, the flame height adjuster, the spark wheel, the flint, the spring, the nozzle, the plastic casing, the gas chamber, the fork, the seal ring, and dozens more. Some estimates count over 200 sub-parts requiring a dozen-plus manufacturing processes: injection molding, stamping, electroplating, ultrasonic welding, gas filling, flame adjustment, leak testing. This is not a stamped piece of metal with fuel inside. This is a precision pressure vessel. It must safely contain pressurized butane gas. It must withstand thousands of friction strikes without failure. It must not leak. It must not explode in your pocket. It must pass a 12-step international safety inspection including drop tests from 1.5 meters in three orientations and a four-hour heat resistance test at 65 degrees Celsius. It must do all of this and cost fifteen cents at retail after being shipped 10,000 miles across an ocean.

The engineer who reads that last sentence will feel a twitch. The twitch is recognition. Something that complex, that safe, that durable, at that price point, delivered over that distance — the math should not work. The math works. That is the miracle. Not the lighter. The math.

The thousandth-of-a-cent game

In early 2025, a video of an automated Shaodong lighter factory circulated on Chinese social media. Anand Mahindra, the Indian industrialist and chairman of Mahindra Group, shared it on X. He wrote that he "couldn't stop watching it." What grabbed him was not the automation — advanced manufacturing is everywhere. What grabbed him was the philosophy behind it:

"Nobody there is winning on cheap labour anymore. They're winning by shaving a thousandth of a cent off the thickness of a plastic wall, or redesigning a base so a few thousand more units fit into the same shipping container."

Let that sink in. They re-engineered the shape of the base — not to improve the product, not to make it prettier, not to add a feature — but to squeeze more units into a container. 100,000 more units per container. At fifteen cents retail, 100,000 units is $15,000 in additional revenue per container. But the profit per unit is less than one cent. So the actual gain per container is perhaps a few hundred dollars. They redesigned the base of a product to capture a few hundred dollars per shipping container. The optimization is not by the cent. It is by the thousandth of a cent. When your profit margin is measured in fractions of a penny, you do not cut costs once. You cut them every day, at every step, across every component, for twenty years. The cutting never stops. The moment it stops, the math breaks, and the price rises, and someone else in the 20-kilometer cluster takes your volume.

This is the mental model that Silicon Valley does not have. Venture-backed startups optimize for growth. They burn capital to acquire users. They measure margins in percentage points, and when margins are thin, they exit the business. Shaodong companies cannot exit the business. This is their business. They have no other. They are not burning someone else's money. They are earning less than a cent per unit and making it work at scale. The discipline is total. The margin for error is zero. The optimization is continuous. There is no off-ramp. There is no pivot. There is only the thousandth of a cent, every day, forever.

The cluster

Michael Porter, Mahindra's former professor at Harvard, theorized that competitive advantage does not come from cheap inputs. It comes from industrial clusters — dense concentrations of specialized firms and suppliers that push each other to improve continuously. When rivals and suppliers crowd into the same small geography for long enough, the competition transcends price. It becomes competition over process, over technique, over the thousandth of a cent. The cluster learns faster than any single firm. The cluster's collective brain — to borrow Matt Ridley's phrase from the pencil argument — is smarter than any individual company's management.

Shaodong is Porter's theory in steel and plastic. Nearly every component of a lighter, except plastic particles and chemical gas, is sourced from 87 local companies within a 20-kilometer radius. Need a spark wheel? Someone down the road has been making nothing but spark wheels for twenty years. Need a flint? Someone else, same road. The lead time for any component is measured in hours, not weeks. If one supplier raises prices, the buyer walks to the next supplier. If one supplier improves a process, the improvement diffuses across the cluster within months. The cluster compresses costs because the geography compresses everything: transport, information, trust, competition, imitation. The cluster is the moat. No single factory in another country can replicate what 114 companies doing one thing for thirty years have learned. The knowledge is distributed across a city. You cannot copy a city. You can copy a factory. The factory without the cluster is a building with expensive machines and no supply chain. The cluster without any individual factory is still the cluster. The barrier to entry is not the technology. It is the geography. It is the density. It is the thirty years of accumulated micro-optimizations that no competitor can reconstruct from scratch because the optimizations were never written down. They live in the tooling, in the workflows, in the hands of 80,000 workers who have been doing this their entire careers. You cannot reverse-engineer a culture. You cannot import a cluster in a shipping container.

The automation that ate the labor

Shaodong's first lighter factory opened in 1992. Fu Zaihua and Yao Hanyun, the founders of Shunfa Manufacturing, bought 50 disposable lighters, took them apart, and reverse-engineered every component. A factory visit had yielded nothing — the manufacturers guarded their processes. So they learned by destruction. They broke 50 lighters and built their knowledge from the fragments. Their first overseas sale went to an Indonesian buyer. By the mid-2000s, Shaodong was the lighter capital of the world. The early advantage was cheap labor. Workers assembled lighters by hand. A thousand workers producing a million lighters a day.

That model should have died when wages rose. It didn't. It evolved.

Hunan Dongyi Electric, the largest manufacturer in Shaodong, began investing in automation in 2013. They spent 60 to 70 million yuan on R&D. The results are staggering: a production line that once required 4,000 workers to produce one million lighters per day now produces over 10 million per day with roughly 2,000 workers. A 20-fold increase in capacity per worker. Labor cost per lighter dropped from 0.1 yuan to 0.015 yuan — an 85% reduction. The automated lines now handle all 12 production procedures: injection molding, stamping, electroplating, component assembly, gas filling, flame adjustment, ultrasonic welding, leak testing, packaging. One worker supervises what dozens once did by hand.

The industry-wide investment in automation and R&D is roughly 200 million yuan annually. A government-established research institute has developed over 30 new types of equipment and products, securing 276 intellectual property patents and 47 invention patents. Over 1,000 researchers work in the local lighter industry. Thirty-eight percent of products are refreshed each year. The Shaodong Lighter Industry Association, founded in 2002, maintains a new-product database specifically to prevent knockoffs and price wars — channeling competition toward genuine innovation rather than duplication. Price competition already has razor margins. Design duplication would cut them to zero. The association ensures the competition is over processes and features, not price. The cluster polices itself. The invisible hand, made visible and given a database.

The economics of the unthinkable

Let's do the math that makes no sense. A lighter wholesales for 0.3 yuan (about 4 cents). Profit per unit: 0.01 to 0.02 yuan — roughly a quarter of a US cent. At 0.01 yuan profit per lighter, and 10 billion units per year across the cluster, the entire Shaodong lighter industry earns approximately 100 million yuan in annual profit — about $14 million. The global disposable lighter industry, supplying nearly every lighter in every corner store and gas station on Earth, earns the annual profit of a mid-sized SaaS company. Let that land. The global supply of fire in your pocket generates the profits of a single B2B software firm with 200 employees.

The SaaS company spends 40% of revenue on sales and marketing. The lighter factory spends 0% on marketing. The SaaS company has 80% gross margins. The lighter factory has razor-thin margins that improve only through automation and process refinement. The SaaS company can raise prices with a new feature tier. The lighter factory cannot raise the price of a basic lighter — the price has been 1 yuan for twenty years, and any deviation loses the volume that makes the margin work. The SaaS company can pivot. The lighter factory makes lighters. There is no pivot. There is only the lighter. There is only the process. There is only the thousandth of a cent, every day, forever.

But here is the thing the SaaS comparison misses. The SaaS company may not exist in ten years. The lighter factories of Shaodong have already existed for thirty. They survived wage inflation that should have killed them. They survived the 2008 financial crisis. They survived COVID-19 — demand held steady because, as one manager noted dryly, "people stuck at home tended to smoke more." They survived the automation transition that required capital investments of tens of millions of yuan. They survived European and American safety regulations that add compliance cost to every exported unit. They survived competition from lower-wage countries by making wage irrelevant. When the labor cost per lighter is 0.015 yuan, moving production to a country with half the wages saves 0.0075 yuan per unit. The shipping cost difference eats the saving. The cluster premium — the 20-kilometer supply chain, the skilled workforce, the accumulated process knowledge — is worth more than 0.0075 yuan. The cluster won. The low-wage countries lost. The math reversed. Cheap labor was the original advantage. Now it's irrelevant. Automation ate the labor. Knowledge ate the labor arbitrage. The cluster became the moat.

The barrier

This is the part that should terrify the competitor. The barrier to entry is not the technology. It is not the capital. It is not the patents. The barrier is that the incumbent earns a quarter of a cent per unit and is profitable. A new entrant has to match the price from day one — the market will not pay 20 cents for a lighter when it can pay 15. The entrant has to match the quality — pass the 12-step safety inspection, survive the drop test, survive the heat test, not leak, not explode. The entrant has to match the supply chain — source 200 components from specialized suppliers who are all already in Shaodong, supplying the incumbents, operating at scale, earning their own fractions of a cent. The entrant has to match the automation — build or buy the custom equipment that Dongyi spent a decade refining. And the entrant has to do all of this while losing money for years, because the incumbent's marginal cost is below the entrant's average cost. The entrant bleeds. The incumbent earns a quarter of a cent per unit. The entrant cannot survive long enough to catch up. The incumbent has been optimizing for thirty years. The entrant starts at zero. The gap is not a technology gap. It is a learning gap disguised as a price gap. The price is the shadow. The learning is the thing.

This is the economics of the impossible, made real. When an entire industry dedicates itself to shaving thousandths of a cent for thirty years, the accumulated optimizations become a barrier higher than any patent wall. You cannot compete with the product. You have to compete with thirty years of learning. The product is visible. The learning is invisible. The visible is cheap to copy. The invisible is impossible.

Why this matters for software

Software engineers should study the Shaodong lighter. Not because software is like lighters. Because software margins are collapsing in the same way, and the response will be the same, and most software organizations are not ready.

AI-assisted development is the automation moment for software. What Dongyi's robots did to lighter assembly, coding agents are doing to software production. The labor cost per feature is dropping. The automation capital costs are rising — the frontier models are expensive to train and run. The incumbents with scale will invest in custom tooling, custom models, custom pipelines, custom verification infrastructure. The entrants will use off-the-shelf tools. The gap will widen. Not because the incumbents are smarter. Because they've been optimizing longer. The learning compounds.

The Shaodong cluster's lesson is not "manufacturing is impressive." The lesson is that micro-margins, pursued relentlessly for decades, create barriers higher than any intellectual property. The lesson is that the optimization never stops. The moment you think the process is good enough, someone down the road — or in a different cloud region, or a different open-source community — is shaving a thousandth of a cent off their equivalent of the plastic wall. The lesson is that the cluster matters more than the firm. The ecosystem matters more than the product. The density of exchange — of ideas, of components, of talent — is the moat. The product is the output of the cluster. You cannot compete with the output. You have to build a competing cluster. Nobody has built a competing cluster in thirty years. Nobody has even tried. The trying would cost billions. The return would be a quarter of a cent per unit. The business case does not close. That is the point. The cluster made the business case unclosable. That is the strategy.

For software platforms: your equivalent of the 20-kilometer supply chain is your API ecosystem, your plugin marketplace, your open-source community. The density of third-party development around your platform is your cluster. The platform with 10,000 extensions is harder to displace than the platform with better technology and 50 extensions. The technology gap can be closed. The ecosystem gap takes years. The years are the moat. The thousandth-of-a-cent optimizations — faster cold starts, lower latency, cheaper inference, simpler APIs — compound across the ecosystem. Every extension benefits. Every developer saves a fraction of a cent in compute, in latency, in cognitive overhead. The fractions add up. The platform that optimized for the thousandth of a cent wins not because it's better in any single dimension but because replacing it means rebuilding 10,000 extensions. Nobody rebuilds 10,000 extensions. The cluster is the moat. The product is the shadow. The ecosystem is the thing.

What the lighter teaches

The Shaodong lighter teaches something uncomfortable about mastery. Mastery is not brilliance. Mastery is doing the same thing for thirty years and never stopping the optimization. It is looking at a product that earns a quarter of a cent and asking: how can I reduce the thickness of this wall by a thousandth of a cent? Not because it's glamorous. Because it's there. Because if you don't, someone else will. Because the math demands it. Because the price is fixed and everything else must bend to the price. The bending is the discipline. The discipline is the mastery.

Next time you pick up a disposable lighter — to light a candle, a cigarette, a stove — look at it. The translucent plastic shell. The spark wheel. The gas button. Thirty parts. Twelve inspection stages. Ten thousand miles of ocean freight. Twenty years of stable pricing. Thirty years of accumulated process knowledge. Eighty thousand workers. One hundred fourteen companies. Twenty kilometers of supply chain. A quarter of a cent of profit. Seventy percent of the world's supply. It is not cheap junk. It is the physical manifestation of relentless optimization. It is the limit of what engineering can achieve when the price is fixed and everything else must give way. The lighter is not the product. The process is the product. The cluster is the product. The learning is the product. The lighter is the evidence that the process works.

Read wrote "I, Pencil" to show that no single person knows how to make a pencil. The knowledge is distributed across millions, coordinated by prices. The lighter extends the argument. Not only does no single person know how to make a lighter — no single person can know. The knowledge required to produce a 15-cent lighter that passes international safety standards after shipping 10,000 miles is not contained in any individual brain, any single factory, any single company. It is distributed across a city — across 80,000 workers, 114 companies, 1,000 researchers, 276 patents, and thirty years of accumulated process refinements. The knowledge is not just distributed. It is embedded. In the tools. In the molds. In the supply relationships. In the database of new-product designs. In the hands of the workers who supervise the automated lines. The knowledge is the cluster. The cluster is the knowledge. You cannot copy it. You can only build it. Nobody has built a second one. Nobody will. The barrier is the learning. The learning is the moat. The moat is permanent.


References:

  • Leonard E. Read, "I, Pencil: My Family Tree as Told to Leonard E. Read," The Freeman, December 1958. Full text at Econlib.
  • "Anand Mahindra couldn't stop watching this 15-cent gas lighter video; What it revealed about China left him thinking about India's future," The Economic Times, 2025. Article.
  • "Lighter hub shines as world leader in production," China Daily HK, December 2024. Article.
  • "Lighter industry mirrors China's strength in manufacturing," People's Daily Online, April 2023. Article.
  • "Shaodong keeping world's lighter prices affordable," China Daily, 2024. Article.
  • Michael E. Porter, "The Competitive Advantage of Nations," Harvard Business Review, March-April 1990.
  • Matt Ridley, "When Ideas Have Sex," TEDGlobal 2010. Video.
  • Matt Ridley, The Rational Optimist: How Prosperity Evolves, Harper, 2010.
  • Related posts: I, Pencil, Dispersed Knowledge, No solutions, only trade-offs, Engineering is art and philosophy, grounded in economic law, On Scarcity.

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

The lighter costs 15 cents. It contains 30 precision parts, pressurized gas, and a supply chain spanning 10,000 miles. The profit is a quarter of a cent per unit. The price hasn't moved in twenty years. Everything else gave way. This is not a product. It is a process at its limit — the physical manifestation of a question asked every day for thirty years: can we shave another thousandth of a cent? The answer is always yes. The optimization never stops. The moat is the learning. The learning is permanent.

On the Many Vs of Data

In 2001, Doug Laney wrote a Gartner note describing three dimensions of the data challenge: Volume, Velocity, Variety. It was not meant to launch a framework. It launched a framework. The Vs became the vocabulary for describing why data is hard.

data-engineeringbig-datavolumevelocityvarietylaney

In 2001, Doug Laney, an analyst at Gartner, wrote a research note titled "3D Data Management: Controlling Data Volume, Velocity, and Variety." The note was internal. It described three dimensions of the growing data management challenge. It was not meant to launch a framework. It launched a framework.

The "three Vs of big data" became the standard vocabulary for describing why data was getting harder to manage. The framework was simple enough to remember, flexible enough to extend. Over the following two decades, practitioners and researchers added more Vs — each naming a specific dimension of the challenge, each adding a word to the vocabulary.

The genius of the Vs is not their precision. It is their utility. They give engineers a shared language for diagnosing why a data project is struggling. Is the volume overwhelming the storage tier? Is the velocity exceeding the pipeline's throughput? Is the variety creating an unmanageable number of pipelines? Is the veracity undermining trust in the numbers? The Vs name the dimensions of difficulty. Naming them is the first step to managing them.

This series explores each V in depth. Each post asks: what does this V mean, why does it matter, what is the engineering constraint it imposes, and how do you respond?

  1. Data Volume — how much you have, and how much you can afford to keep
  2. Data Velocity — how fast data arrives, and how fast you must decide
  3. Data Variety — the diversity of formats, and the cost of integration
  4. Data Veracity — trustworthiness, and the gap between data and truth
  5. Data Value — the only V that justifies the platform's existence
  6. Data Variability — fluctuation over time, and why pipelines break
  7. Data Visualization — the bridge between data and decision
  8. Data Viscosity — resistance to movement, and technical debt in the supply chain
  9. Data Virality — how data usage spreads through people
  10. Data Volatility — the half-life of data value

The Vs are not a taxonomy. They are a diagnostic tool. When a data project is struggling, ask: which V is the problem? The answer points to the solution.


Reference: Doug Laney, "3D Data Management: Controlling Data Volume, Velocity, and Variety," Gartner, 2001.

On Data Volatility: the half-life of data value

Volatility is the rate at which data loses relevance. Stock prices matter for milliseconds. Medical records matter for decades. The half-life of data value determines how much to invest in its quality and how long to keep it.

data-engineeringvolatilityretentiontime-to-livedata-lifecycle

Volatility is not about how fast data changes. It is about how fast data dies. Every dataset has a half-life. After the half-life, the storage cost exceeds the remaining value. Keeping data past its half-life is not preservation. It is waste.

Volatility is the rate at which data loses relevance over time. It is the dimension that determines retention policy. High volatility: the data is valuable for milliseconds, then worthless. Low volatility: the data is valuable for decades. The half-life of data value — the time after which the data is half as useful as it was when created — determines how much to invest in its quality and how long to store it. A dataset with a half-life of one day should be stored on cheap storage, minimally curated, and deleted after a month. A dataset with a half-life of ten years should be stored on reliable storage, carefully curated, and retained indefinitely.

A stock trading system illustrates high volatility. Real-time price data is valuable for milliseconds — the window for executing an arbitrage. After the window closes, the data's value drops to near zero for trading. But it remains valuable for backtesting and compliance — the dual value curve: immediate and high, then residual and low. The curve determines the storage architecture: in-memory for the trading window, archival for compliance. The architecture is tiered because the volatility is dual. A single tier would either be too expensive (keeping millisecond-old data on fast storage forever) or too slow (keeping compliance queries waiting for archival retrieval).

Most organizations have no retention policy. The absence of a policy is a policy — keep everything forever. Forever is expensive. The storage cost is linear. The value of the oldest data is near zero. The gap between cost and value widens every year. The gap is waste. The waste is invisible because the storage bill is aggregated.

The engineering response is automated data lifecycle management. Define retention rules: transactional data kept for 7 years (compliance), web analytics kept for 2 years, ML training features kept for 6 months or until model retraining. Enforce the rules automatically — pipelines that delete or archive data when it exceeds its retention period. The enforcement must be auditable — if data is deleted, there must be a record of what was deleted, when, and under what policy. The audit trail is the defense against the accusation of destroying evidence. The accusation is rare. The defense is necessary.

Gordon Moore's 1965 paper predicted that transistor density would double every two years. The prediction held. Storage became cheap enough that keeping everything seemed rational. The rationality was an illusion. Storage is cheap per gigabyte. The total cost — storage plus maintenance plus cognitive load — is not cheap. The total cost accumulates. The accumulation is the reason retention policies exist. The policy is the recognition that data has a finite useful life. The useful life is shorter than the organizational memory. The data outlives its usefulness. The outliving is the problem. The deletion is the solution.

See: Apache Iceberg, "Table Spec: Snapshots and Time Travel" (Iceberg Documentation) — on the table format features that make retention management programmable. Netflix Tech Blog, "Evolution of Data Lifecycle Management at Netflix" (2021) — a real-world architecture for automated data retention. Gordon Moore, "Cramming More Components onto Integrated Circuits" (Electronics, 1965).

This post is part of a series on The Many Vs of Data, originating from Doug Laney's 2001 Gartner note. Each V names a dimension of why data is hard.

On Data Virality: how data spreads through people

Virality is the rate at which data usage spreads through an organization. A dataset that one team finds useful spreads to other teams. The spread is the measure of value. A dataset that nobody knows about is worthless, regardless of its quality.

data-engineeringviralityadoptiondata-catalogdiscoverability

Virality is not about how fast data moves through pipelines. It is about how fast data moves through people. A dataset that nobody knows about has no value. Discoverability is the prerequisite for value. Without it, the best dataset in the world is invisible.

Virality is the rate at which data usage spreads through an organization. It is the dimension that separates successful data platforms from unsuccessful ones. A dataset that one team finds useful spreads to other teams. The recruiting team discovers the employee movement dataset and uses it for time-to-fill metrics. The finance team discovers it and uses it for headcount forecasting. The facilities team discovers it and uses it for office space planning. The dataset spreads because it answers questions that multiple teams have. The spread is the evidence of value.

The opposite of virality is invisibility. The data exists. It is accurate, timely, well-modeled. Nobody knows about it. The team that built it uses it. Nobody else does. The dataset is technically successful and organizationally irrelevant. The irrelevance is not the fault of the data. It is the fault of the platform. The platform made no provision for discoverability — no catalog, no documentation, no mechanism for users to find datasets they didn't already know existed. The platform is a library with no catalog. The books are there. You cannot find them.

The engineering response to the virality problem is the data catalog. A catalog — Alation, Atlan, DataHub, Amundsen — indexes all data assets across the organization. It shows lineage — where data came from and where it goes. It shows ownership — who is responsible. It shows quality metrics — when the dataset was last validated. It shows usage — who queries it, how often. The catalog makes data discoverable. Discoverability is the precondition for virality. You cannot use data you cannot find.

But discoverability is not enough. The data must also be understandable. A dataset with cryptic column names and no documentation is discoverable but unusable. The user finds it, opens it, sees col_37 VARCHAR, and closes it. The data dictionary — column descriptions, business definitions, example values — is the mechanism for understandability. The data quality dashboard — freshness, completeness, accuracy metrics — is the mechanism for trust. Discoverability, understandability, trust. All three are required for virality. Most platforms have one. Some have two. Few have all three. The ones that have all three have viral data.

See: Prukalpa Sankar, "The Data Maturity Curve" (Atlan, 2021) — on the stages of data culture from fragmented to viral. Michelle Casbon et al., "Data Governance: The Definitive Guide" (O'Reilly, 2021) — on how governance enables rather than restricts data adoption. Alation, "The Data Catalog: The Foundation of Data Culture" (Whitepaper, 2020).

This post is part of a series on The Many Vs of Data, originating from Doug Laney's 2001 Gartner note. Each V names a dimension of why data is hard.

On Data Viscosity: resistance to flow

Viscosity is the resistance of data to movement. Data locked in legacy systems, undocumented schemas, departed engineers. The harder it is to extract data, the higher the viscosity. Viscosity is technical debt in the data supply chain.

data-engineeringviscositylegacyextractionapi-design

Viscosity is not about how far the data must travel. It is about how hard it is to get it moving. Data that is easy to extract is an asset. Data that resists extraction is a hostage. The ransom is engineering time.

Viscosity is the resistance of data to movement through the system. High viscosity: the data is locked in a legacy system with no API, the schema is undocumented, the original engineers have left, and every extraction requires a custom script maintained by the one person who understands the format. Low viscosity: the data is available through a standard API, the schema is published in a catalog, anyone with credentials can query it. The difference between high and low viscosity is the difference between a data platform that can add sources in days and one that adds sources in months.

A bank migrating customer data from a 30-year-old mainframe system illustrates the extreme. The mainframe stores data in VSAM files with COBOL copybooks defining the schema. The copybooks are the only documentation. The COBOL programmers who wrote them have retired. The data must be extracted, parsed according to copybook definitions, transformed to a modern schema, and loaded into the data platform. The extraction alone takes 18 months. The 18 months is the viscosity. The viscosity is the cost.

Viscosity is technical debt in the data supply chain. Every system that produces data but was not designed to expose it adds viscosity. The debt accumulates with every legacy system, every proprietary format, every undocumented schema. The interest on the debt is the engineering time spent extracting data that should have been available through an API. Unlike financial debt, data supply chain debt is invisible — it appears on no balance sheet, is tracked by no metric, and is discovered only when someone tries to use the data. The discovery is the moment the debt becomes visible. The visibility is painful.

The engineering response is to treat data extraction as a first-class requirement for any system that produces data. Every application should expose its data through a standard interface — a read replica, a change data capture feed, an API with documented schemas. The interface should be designed before the application ships. Retrofitting extraction onto an existing system is ten times more expensive than designing it in. The design-in is the investment. The retrofit is the debt repayment. The debt repayment is always more expensive than the investment would have been.

See: Zhamak Dehghani, "Data Mesh: Delivering Data-Driven Value at Scale" (O'Reilly, 2021) — on domain ownership as the organizational response to viscosity. Maxime Beauchemin, "Functional Data Engineering" (2018) — on treating data pipelines as software engineering with API design standards. Martin Fowler, "Patterns of Enterprise Application Architecture" (Addison-Wesley, 2002), Chapter 9, on data access patterns.

This post is part of a series on The Many Vs of Data, originating from Doug Laney's 2001 Gartner note. Each V names a dimension of why data is hard.

On Data Visualization: the bridge between data and decision

Visualization is the bridge between data and decision. A table of numbers is data. A chart is understanding. The difference is the visual encoding that maps data to perception. Design the encoding badly, and the signal is invisible. Design it well, and the signal is impossible to miss.

data-engineeringvisualizationtuftedashboardsdesign

Visualization is not decoration. It is cognitive engineering. The chart is an interface between the data and the mind. The interface determines what the mind perceives. A bad interface hides the signal. A good one makes the signal impossible to ignore.

Visualization is the presentation of data in a form that humans can perceive patterns, anomalies, and relationships. A table of 50,000 rows is data. A line chart of the same data is information. The difference is the visual encoding — position, length, color, shape, orientation — that maps data attributes to perceptual channels. The mapping is a design decision. The design determines whether the viewer sees the signal or the noise.

The theory of visualization is the theory of human perception. Jacques Bertin's Semiology of Graphics (1967) identified the visual variables: position, size, value (lightness), texture, color, orientation, shape. Each variable has different perceptual properties. Position is the most accurate — humans can compare positions along a common scale with high precision. Color is less accurate — humans perceive color categorically, not continuously. The design principle: map the most important data attribute to the most accurate perceptual channel. Position for quantities. Color for categories. Shape for nothing important — it is the least accurate channel. The principle is violated in most dashboards, which use color for quantities and position for nothing.

Edward Tufte's The Visual Display of Quantitative Information (1983) defined the principles of graphical excellence: show the data, induce the viewer to think about the substance, avoid distorting what the data has to say, present many numbers in a small space, make large datasets coherent, encourage the eye to compare different pieces of data, reveal the data at several levels of detail, serve a clear purpose. Each principle is a constraint. Each constraint improves the result. Tufte's most famous rule: maximize the data-ink ratio — the proportion of ink used to present data compared to total ink used in the graphic. Erase non-data-ink. Erase redundant data-ink. The erasure is the discipline.

A hospital infection control dashboard illustrates the stakes. Raw data: 50,000 surgeries, 200 columns. The dashboard visualizes infection rates as a control chart — a line graph with upper and lower control limits, each theater as a separate line. When Theater 7 exceeds the upper control limit, the point turns red. The red point triggers a signal. The signal triggers an investigation. The investigation finds a changed sterilization protocol. The protocol is reverted. Infections decline. The visualization saved lives. The visualization worked because the design made the anomaly perceptible. A table of 50,000 rows would not have. The table is data. The chart is a decision support system.

See: Edward Tufte, "The Visual Display of Quantitative Information" (Graphics Press, 1983). Jacques Bertin, "Semiology of Graphics" (1967, English translation 1983). Leland Wilkinson, "The Grammar of Graphics" (Springer, 1999) — the theoretical foundation of ggplot2 and Vega-Lite.

This post is part of a series on The Many Vs of Data, originating from Doug Laney's 2001 Gartner note. Each V names a dimension of why data is hard.

On Data Value: the only V that justifies the platform

Value is the only V that justifies the existence of the data platform. Data has value when it informs a decision that produces a better outcome. Most data has no value. It was collected because it could be, stored because storage is cheap, and never used. Useless data is a liability, not an asset.

data-engineeringvalueeconomicsroiprioritization

Value is not a property of data. It is a property of the decision the data informs. Data that informs no decision has no value, regardless of how expensive it was to collect, how sophisticated the pipeline that serves it, or how impressive the dashboard that displays it.

Value is the only V that justifies the data platform's existence. Data has value when it changes a decision. The value is the difference between the outcome with the data and the outcome without it. If the data doesn't change the decision, it has no value. If the decision would have been the same either way, the data added nothing. The nothing was expensive to produce.

An e-commerce company collects every click, page view, add-to-cart, and purchase. The clickstream alone is 10 TB per day. The data team builds pipelines to ingest it, transform it, serve it. Six months later, nobody queries the clickstream tables. The analysts query purchases and ignore the rest. The clickstream data has zero value. The storage costs $2,400 per month. The pipeline maintenance costs one engineer at 20% time. The total cost exceeds the value. The data is a liability.

Most data in most organizations is a liability. It was collected because it could be — the event was there, the tracking was easy, the storage was cheap. It was never used because nobody knew what question it answered. The question came first: "we should track everything, we might need it later." The question was wrong. The right question is: "what decision will this data inform, and what is the value of a better decision?" If the answer is unclear, don't collect the data. The collection is not free. The cost is the pipeline, the storage, the maintenance, the cognitive load on the data team, the dilution of the data catalog with tables nobody queries. The cost is paid forever. The value is zero. The net is negative.

Value forces prioritization. You cannot pipeline every data source. You must pipeline the sources that will answer the business's most important questions. Identifying those questions requires talking to the people who will use the data. The talking is the most underinvested activity in data engineering. Engineers build pipelines for data they have, not for questions the business needs answered. The pipeline exists. The question doesn't. The value is zero.

Doug Laney's Infonomics (2017) proposes treating data as a balance-sheet asset: measure its value, depreciate it over its useful life, account for the cost of maintaining it. The proposal is radical because almost no organization does it. Data is treated as free. It is not free. The cost is real. The value is real only when the data is used. The gap between cost and value is the economic problem of data. Closing the gap is the discipline of data economics.

See: Doug Laney, "Infonomics: How to Monetize, Manage, and Measure Information as an Asset for Competitive Advantage" (Routledge, 2017). Thomas C. Redman, "Data's Credibility Problem" (Harvard Business Review, 2013). Benn Stancil, "The Data Platform Cost Model" (Mode, 2020).

This post is part of a series on The Many Vs of Data, originating from Doug Laney's 2001 Gartner note. Each V names a dimension of why data is hard.

On Data Variability: the gap between peak and average

Variability is the fluctuation in data characteristics over time. It breaks pipelines. A source that produces 1,000 events per hour suddenly produces 100,000. A column of integers suddenly contains strings. The pipeline wasn't designed for this. Variability is the reason pipelines need monitoring, alerting, and circuit breakers.

data-engineeringvariabilitypeakselasticitycapacity-planning

Variability is not the peak. It is the gap between the peak and the average. The average is fiction. The peak is the design constraint. Designing for the average is designing for failure at the moment of maximum need.

Variability is the fluctuation in data characteristics over time. It is the dimension that breaks pipelines. A data source that produces 1,000 events per hour suddenly produces 100,000. A column that has always contained integers suddenly contains strings. A partition that has always had data is suddenly empty. The pipeline was designed for the steady state. The steady state is a lie. The world is not steady.

A food delivery platform illustrates the constraint. Order volume follows a predictable pattern: peaks at lunch and dinner, troughs in between, higher on weekends. The pipeline is sized for 2× the average peak. On Super Bowl Sunday, order volume spikes to 20× the normal peak. The ingestion pipeline, sized for 2×, falls behind. The transformation pipeline, expecting 1 million rows, receives 20 million. Dashboard queries time out. The data platform fails at the moment it is most needed — when the business wants to know how many orders were placed, how many were delivered, what the average delivery time was. The numbers are unavailable. The unavailability is the cost of designing for the average.

The engineering response is elastic infrastructure. Cloud warehouses scale compute on demand. Pipelines auto-scale to handle spikes. The infrastructure adapts to the load. The adaptation has limits: scaling takes time (minutes, not seconds), and the peak may exceed the maximum scale. The limits are the constraint. The constraint must be managed by load shedding — dropping non-critical work during spikes to preserve critical work. The load shedding is a policy. The policy must be defined before the spike. Defining it during the spike is too late.

The deeper response is defensive pipeline design. Pipelines should degrade gracefully under load — process what they can, queue what they can't, alert on what they're dropping. Pipelines should be tested against synthetic spikes — double the volume, triple the volume, ten times the volume — to find the breaking point before production does. Pipelines should have circuit breakers — if the output is anomalous (zero rows, ten times the expected count, nulls in required fields), stop the pipeline and alert. The circuit breaker prevents bad data from propagating downstream. Stopping the pipeline is better than publishing wrong numbers. Wrong numbers are worse than no numbers. No numbers prompt a question. Wrong numbers prompt a wrong decision.

See: John Allspaw, "The Art of Capacity Planning" (O'Reilly, 2008) — the operations engineering approach to sizing for variability. Betsy Beyer et al., "Site Reliability Engineering" (O'Reilly, 2016), Chapter 13, on managing overload. Daniel Abadi et al., "The Design and Implementation of Modern Column-Oriented Database Systems" (Foundations and Trends in Databases, 2013) — why cloud elasticity changes the trade-offs.

This post is part of a series on The Many Vs of Data, originating from Doug Laney's 2001 Gartner note. Each V names a dimension of why data is hard.

On Data Variety: the cost of integration

Variety is the diversity of data formats, structures, and sources. Every new source requires a pipeline. Every pipeline requires maintenance. The cost of variety is the cost of integration, and integration is the unglamorous work that makes data useful.

data-engineeringvarietyintegrationschemaetl

Variety is not about how many formats you have. It is about how many pipelines you must maintain. Each format is a promise. Each pipeline is the cost of keeping that promise. The cost compounds.

Variety is the diversity of data formats, structures, and sources. It is the dimension that makes data integration hard. Data arrives as relational tables, JSON documents, XML messages, CSV files, Parquet files, Avro records, Protobuf messages, images, videos, log files, and free-text documents. Each format has its own schema, its own semantics, its own quirks. Each source has its own update cadence, its own reliability characteristics, its own failure modes. Integrating them requires understanding each one. Understanding each one requires time. Time is the scarcest resource in data engineering.

A hospital illustrates the constraint. Patient data spans electronic health records (structured, HL7 format), lab systems (semi-structured, ASTM), imaging systems (unstructured, DICOM), patient surveys (free text), and wearable devices (JSON streams). A single patient's record crosses five systems, four formats, three data models. Integrating them requires mapping each source to a common model, resolving identifier conflicts (is patient 123 in the EHR the same person as patient 456 in the lab system?), and handling different update cadences. The hospital's data engineering team spends 80% of its time on integration. The 80% is the cost of variety.

Variety is the reason data engineering teams grow faster than the data they manage. Each new data source adds a pipeline. Each pipeline adds a maintenance burden: the source schema changes, the pipeline breaks, the engineer fixes it. The breakage is not a one-time event. It is continuous. Every source system upgrade, every new column, every deprecated field produces a pipeline failure. The failures accumulate. The maintenance burden grows. The team grows to handle the burden. The growth is the cost of variety.

The modern response to variety is the ELT pattern with a schema-on-read approach. Extract the data in its native format. Load it into the data lake or warehouse without transformation. Apply schema at query time. The raw data is preserved. The transformation logic is version-controlled SQL. The approach decouples ingestion (fast, reliable, format-agnostic) from transformation (flexible, query-time, iterative). The decoupling is the architectural insight: you cannot predict how the data will be used, so preserve it in its original form and let consumers apply their own schemas.

See: Serge Abiteboul et al., "Data on the Web: From Relations to Semistructured Data and XML" (Morgan Kaufmann, 1999) — the classic text on the shift from structured to semi-structured data. Joe Reis and Matt Housley, "Fundamentals of Data Engineering" (O'Reilly, 2022), Chapter 4, on the modern integration landscape. James Serra, "Deciphering Data Architectures" (O'Reilly, 2024) — on choosing between data warehouse, data lake, lakehouse, and data mesh for variety-heavy environments.

This post is part of a series on The Many Vs of Data, originating from Doug Laney's 2001 Gartner note. Each V names a dimension of why data is hard.

On Data Velocity: how fast you must decide

Velocity is the speed at which data arrives and the speed at which it must be acted upon. Batch is forgiving. Streaming is not. The choice between them is the most consequential architectural decision in data engineering.

data-engineeringvelocitystreamingbatchreal-time

Velocity does not ask "how fast can you process?" It asks "how fast must you decide?" The speed of data is irrelevant if the decision can wait. The speed matters when the decision cannot.

Velocity is the dimension that separates batch from streaming. Batch processing: data arrives in files, accumulates for hours or days, and is processed in a single job that reads all available data and produces output. Streaming processing: data arrives as messages, each message is processed within milliseconds of arrival, and the output updates continuously. The architectures are different. The failure modes are different. The organizational capabilities required are different. Choosing between them is the most consequential architectural decision in data engineering.

A fraud detection system at a payment processor illustrates the velocity constraint. Each transaction generates an event. The event must be evaluated within 100 milliseconds — is this transaction fraudulent? The decision requires consulting historical patterns (has this card been used in this location before?), real-time aggregates (how many transactions has this card made in the past hour?), and ML model inference (what is the fraud score?). The velocity constraint forces the architecture: historical data pre-computed and served from an in-memory cache, real-time aggregates maintained by a stream processor, model served with sub-millisecond latency. A batch system that evaluated fraud once per hour would approve fraudulent transactions for up to 59 minutes before detection. The 59 minutes is the cost of batch.

Streaming is harder than batch. Batch systems are forgiving — if a job fails, you restart it from the last checkpoint. The input data is immutable. The output is overwritten. The failure is a delay. Streaming systems are unforgiving — if a message is missed, the aggregate is wrong. If the stream processor crashes, the in-memory state is lost. If two messages arrive out of order, the windowed computation is incorrect. Streaming requires exactly-once semantics (each message processed exactly once, even across failures), watermark handling (how to handle late-arriving data in windowed computations), and state management (how to recover in-memory state after a crash). The complexity is the price of low latency. The price is worth paying when the decision must be made now.

The modern synthesis is the Lambda architecture and its successor, Kappa. Lambda: maintain two parallel pipelines — a batch layer for accurate but delayed results, a speed layer for approximate but immediate results, a serving layer that merges them. Kappa: process everything as a stream, replaying historical data from the stream's retention log when reprocessing is needed. Kappa won because it simplifies the operational burden — one code path, not two. The simplification is the engineering insight: the stream is the source of truth. The batch is a special case of the stream — a stream with a very long window.

See: Tyler Akidau et al., "The Dataflow Model" (VLDB, 2015) — the paper that unified batch and streaming under event-time windowing, now the basis for Apache Beam. Jay Kreps, "Questioning the Lambda Architecture" (O'Reilly, 2014) — why maintaining two code paths is a maintenance disaster. Martin Kleppmann, "Designing Data-Intensive Applications" (O'Reilly, 2017), Chapter 11, on stream processing and the Kappa architecture.

This post is part of a series on The Many Vs of Data, originating from Doug Laney's 2001 Gartner note. Each V names a dimension of why data is hard.

On Data Veracity: trust is the product

Veracity is the trustworthiness of data. A pipeline that runs successfully can produce garbage. The tests can pass and the data can still be wrong. Veracity is the gap between what the data says and what is true. Closing that gap is the hardest problem in data engineering.

data-engineeringveracitydata-qualitytrusttesting

Veracity is not about whether the data is correct. It is about whether you believe it is correct. Trust is the product. Trust is earned through testing, monitoring, and the accumulated evidence of not being wrong when it mattered.

Veracity is the quality and trustworthiness of data. It is the hardest V because it is not purely technical. A pipeline can run successfully and produce garbage. The tests can pass and the data can still be wrong — the tests test what you thought to test, and you didn't think of everything. The consumers of data — analysts, executives, ML models — trust the numbers until they don't. The moment they stop trusting, every number produced by the data platform becomes suspect. Restoring trust is harder than destroying it. Most organizations never fully restore it.

A weather forecasting system illustrates the challenge. Data arrives from ground stations (accurate but sparse), satellites (global coverage but lower resolution), weather balloons (vertical profiles, twice daily), and citizen reports (abundant but unreliable — people report hail when they hear acorns on the roof). The system must fuse data from sources of varying veracity, weighting each by its historical accuracy. The fusion is the easy part. The hard part is maintaining the weights as source quality changes — a ground station's sensor drifts, a satellite's calibration degrades, a citizen reporting network grows. The weights must adapt. The adaptation must be automated. The automation must be verified against ground truth. The ground truth is expensive to obtain. The expense is the cost of veracity.

The data quality testing pyramid — column-level tests (nulls, ranges, allowed values), table-level tests (uniqueness, referential integrity), cross-table tests (reconciliation across sources), business logic tests (known reference values) — is the engineering response to veracity. Tests catch errors before users do. Users catching errors is the worst outcome because it destroys trust. Each error that reaches a user reduces the user's confidence in the platform. The reduction is cumulative. After enough errors, the user stops using the platform. The platform has failed. The failure is not that the data was wrong. The failure is that the wrongness was invisible until it reached the user.

Data lineage — the ability to trace any number in any dashboard back to its source, through every transformation — is the mechanism for debugging veracity failures. When the CFO asks why revenue is down, the data engineer must be able to trace the revenue number backward: this table → that pipeline → this source → that extraction. The trace is the answer. The inability to trace is the failure. Lineage is the data engineer's call stack. Without it, debugging is archaeology. With it, debugging is engineering.

See: Tom Redman, "Data Driven: Profiting from Your Most Important Business Asset" (Harvard Business Review Press, 2008) — the foundational text on data quality as a management discipline. Barr Moses et al., "Data Quality Management at Scale" (Monte Carlo, 2022) — modern operational practices for data observability. Wenfei Fan and Floris Geerts, "Foundations of Data Quality Management" (Morgan & Claypool, 2012) — a formal treatment of data consistency, currency, and accuracy.

This post is part of a series on The Many Vs of Data, originating from Doug Laney's 2001 Gartner note. Each V names a dimension of why data is hard.

On Data Volume: how much you can afford to keep

Volume is not about how much you have. It is about how much you can afford to keep. The engineering of volume is the engineering of deciding what to throw away.

data-engineeringvolumebig-datastorage

Volume is not about how much you have. It is about how much you can afford to keep. Every byte stored is a byte that must be justified. Most bytes are not.

Volume is the most obvious dimension of data and the least interesting. It is what people mean when they say "big data" — petabytes, exabytes, the incomprehensible scale of modern data generation. A single Boeing 787 generates approximately 500 GB of sensor data per flight. A fleet of 1,000 aircraft flying two flights per day generates 1 PB per day. The Large Hadron Collider generates 90 PB per year. YouTube ingests 500 hours of video per minute. The numbers are large. The numbers are not the point.

The point is that volume forces choices. You cannot keep everything. The physics of storage — the cost per terabyte, the I/O throughput of the disk subsystem, the network bandwidth between storage and compute — imposes constraints that no amount of cloud elasticity can eliminate. Elasticity changes the cost curve from stepwise (buying servers) to continuous (paying per gigabyte). It does not make storage free. The cost is linear with volume. The value of data is not. The oldest data has the lowest value and the same cost as the newest data. The divergence between cost and value is the engineering problem of volume.

The solution is tiered storage. Hot data — queried frequently, needed in milliseconds — lives on SSDs or in-memory caches. Warm data — queried occasionally, needed in seconds — lives on object storage with fast retrieval tiers. Cold data — queried rarely, needed in minutes or hours — lives on the cheapest object storage tier, or on tape, or is deleted. The tiers are distinguished by cost, latency, and throughput. The assignment of data to tiers is an optimization problem: minimize total cost subject to latency constraints for each query class. The optimization is continuous because data ages. Yesterday's hot data is today's warm data. Today's warm data is next month's cold data. The tiering must be automated. The automation must be correct. A misclassified dataset — cold data on hot storage — is a cost inefficiency. Hot data on cold storage is a latency violation.

The engineering discipline of volume is the discipline of deciding what to throw away. Most organizations never make this decision. They keep everything. The storage bill grows. The value of the oldest data approaches zero. The cost is constant. The gap between cost and value widens. The gap is waste. The waste is invisible because the storage bill is aggregated. Nobody sees the line item for "data that has not been queried in three years." The line item exists. It is the majority of the bill. The majority of the bill is waste.

See: James Hamilton, "Internet-Scale Storage" (AWS Reinvent, 2014) — on the physics of storage at hyperscale. Alex Petrov, "Database Internals" (O'Reilly, 2019), Chapter 3, on B-Trees and LSM-Trees — the data structures that make volume queryable. David DeWitt and Jim Gray, "Parallel Database Systems: The Future of High Performance Database Systems" (Communications of the ACM, 1992) — the paper that predicted the distributed, partitioned architecture that makes petabyte-scale queries possible.

This post is part of a series on The Many Vs of Data, originating from Doug Laney's 2001 Gartner note. Each V names a dimension of why data is hard.

Lehman's Software Evolution

Meir Lehman observed that software doesn't age — it evolves according to laws as regular as thermodynamics. E-type programs change or die. Complexity increases unless you fight it. The process is self-regulating whether you like it or not. Maintenance is not a phase. It is the process.

lehmansoftware-evolutioncomplexityentropylawsmaintenance

In 1980, Meir Lehman published a paper that should have changed how the industry thinks about software maintenance. It didn't. "Programs, Life Cycles, and Laws of Software Evolution" introduced a classification of programs and a set of laws governing their evolution. The laws were derived from measurement, not opinion. They described what software does, not what anyone wished it did.

"We shall, in fact, argue that the need for continuous change is intrinsic to the nature of computer usage."

Forty-six years later, the laws hold. The industry still acts surprised when its codebases become harder to change, more complex, less maintainable. Lehman explained why. The explanation is still correct. The surprise is still unwarranted.

"The resultant evolution of software appears to be driven and controlled by human decision, managerial edict, and programmer judgement. Yet as shown by extended studies, measures of its evolution display patterns, regularity and trends that suggest an underlying dynamics."

Individual decisions feel local and independent. The aggregate is regular. The aggregate is law-like. That is Lehman's central discovery. Software evolution is not random. It is not controlled. It is regular — and the regularity persists regardless of what anyone intends.

The program types

Lehman's first contribution was a classification. Not all programs are the same kind of thing. Treating them as if they are is the root error.

S-type (Specification-type). The problem has a complete, formal specification. Correctness can be proven. Changes are limited to efficiency or clarity. A sorting algorithm. The Eight Queens puzzle. S-type programs don't evolve. They are replaced.

"Programs whose function is formally defined by and derivable from a specification."

P-type (Problem-type). The problem can be stated formally, but a perfect solution is infeasible. The program uses heuristics or approximations. Acceptability is judged against the real world. Chess engines. Weather prediction. P-type programs evolve as heuristics improve.

E-type (Embedded-type). The program mechanizes a human or societal activity. It becomes part of the world it models. Deploying the program changes user behavior, which changes requirements, which changes the program. The problem cannot be precisely formulated — it involves judgment. Operating systems. Business software. Trading systems. Your job.

"The program has become a part of the world it models, it is embedded in it."

E-type programs must evolve. They have no final state.

"E-programs change because the real-world changes... but E-programs can also be the cause of that change in the real world."

You deploy the software. Users change their behavior. The changed behavior creates new requirements. The new requirements change the software. The changed software changes behavior again. This is not a bug in the process. This is the process. You are not building a tool. You are participating in a feedback loop. The loop has no end. The loop is the work.

Dijkstra, from a different tradition entirely, identified the same confusion in the language we use:

"Unfathomed misunderstanding is further revealed by the term 'software maintenance', as a result of which many people continue to believe that programs — and even programming languages themselves — are subject to wear and tear. Your car needs maintenance too, doesn't it?"

Software doesn't wear out. The environment changes around it. "Maintenance" implies restoring something to its original condition. Software evolution is about changing something to meet conditions that didn't exist when it was built. The word is wrong. The concept is wrong. The industry's entire budgeting model — "build, then maintain" — is built on the wrong concept.

The eight laws

Lehman published the first three laws in 1974 with Belady, expanded to five in 1980, and codified all eight by 1996. Each was derived from measurement of real systems — OS/360 first, then others. These are observations of statistical regularity. You can ignore them. You cannot make them false.

I. Continuing Change (1974)

"A program that is used and that as an implementation of its specification reflects some other reality, undergoes continual change or becomes progressively less useful. The change or decay process continues until it is judged more cost effective to replace the system with a recreated version."

An E-type system must be continually adapted or it becomes progressively less satisfactory. The environment changes — user needs, regulations, platforms, security threats. The software changes with it or becomes irrelevant. There is no third option. You do not finish an E-type system. You stop working on it. The only question is whether you stop because it was replaced or because it was abandoned.

Brooks, independently, reached the identical conclusion in No Silver Bullet (1986):

"All successful software gets changed. Software is embedded in a cultural matrix of users, laws, and hardware — all of which change continually."

Two researchers, different methods, same observation. Software that matters changes. Software that doesn't change doesn't matter. The change is not a failure mode. It is evidence that the software is doing its job — reflecting a world that moves. The cost of change is not overhead. It is the work.

Lehman was even more direct about the economic consequence:

"Assessments of the economic viability of a program must include total lifetime costs and their life cycle distribution, and not be based exclusively on the initial development costs."

If you budget only for building and not for changing, you have budgeted for failure. The change is not optional. The budget for it is not discretionary. The economics that ignore change are wrong on their own terms. They produce numbers that look good at project start and catastrophic at year five. The numbers were always catastrophic. The accounting hid it.

II. Increasing Complexity (1974)

"As an evolving program is continually changed, its complexity, reflecting deteriorating structure, increases unless work is done to maintain or reduce it."

This is Lehman's most practically important law. The default direction is toward disorder. Every change increases complexity unless explicit effort is made to reduce it.

Lehman and Belady originally called this entropy, not complexity. The 1971 IBM report states the thermodynamic intuition directly:

"The addition of any function not visualized in the original design will inevitably degenerate structure. Repairs also, will tend to cause deviation from structural regularity since, except under conditions of the strictest control, any repair or patch will be made in the simplest and quickest way. No search will be made for a fix that maintains structural integrity."

This is the most damning sentence in the entire Lehman corpus. Repairs are made in the simplest and quickest way. No search is made for a fix that maintains structural integrity. The deadline applies pressure. The fix is local. The structure degrades. The degradation is invisible at the time of the fix. It becomes visible later, when the next fix is harder because the structure is weaker. The cycle compounds. The entropy accumulates. Nobody intended it. The process produced it.

"All repairs tend to destroy the structure, to increase the entropy and disorder of the system. Less and less effort is spent fixing original design flaws; more and more is spent on fixing flaws introduced in earlier fixes. As time passes, the system becomes less and less well ordered."

The effort shifts. At first, you fix the original design. Then you fix the fixes. Then you fix the fixes of the fixes. Each layer adds entropy. Each layer makes the next layer more likely. The system is not just getting more complex. It is getting more complex in a way that accelerates further complexity. This is a positive feedback loop driving toward disorder. Physics has a name for this. Lehman borrowed it.

In the 1976 IBM Systems Journal paper, Belady and Lehman formalized it:

"The entropy of a system (its unstructuredness) increases with time, unless specific work is executed to maintain or reduce it."

Entropy. Not complexity. Unstructuredness. The loss of form. The drift toward chaos. Specific work is required to maintain structure. If you are not doing that specific work — if all your effort goes to features and fixes — the structure is degrading. You may not notice. The degradation is gradual. By the time it is obvious, it is expensive.

Parnas, independently, provided the mechanism for fighting Law II: information hiding. Hide volatile design decisions behind stable interfaces. Contain the change inside the module. The interface stays clean. The rest of the system doesn't accumulate the complexity of the change. Lehman described the problem. Parnas described the defense. Both were published in the early 1970s. Most production code follows neither.

Later, Parnas introduced the concept of software aging (1994), identifying two causes that directly mirror Lehman's Laws:

  1. Lack of movement — failure to adapt to environmental change (Law I violation)
  2. Ignorant surgery — changes made without proper understanding of the system (Law II accelerator)

Parnas and Lehman converge on the same insight from different angles: software doesn't physically decay. Its structure degrades through a series of individually reasonable, collectively destructive changes. Each change made sense at the time. The accumulation makes no sense at all.

Brooks separated complexity into essential and accidental in No Silver Bullet:

"The complexity of software is an essential property, not an accidental one. Descriptions of a software entity that abstract away its complexity often abstract away its essence."

"From the complexity comes the difficulty of communication among team members, which leads to product flaws, cost overruns, and schedule delays."

Essential complexity is inherent in the problem. Accidental complexity is imposed by our solutions. Law II describes the accumulation of accidental complexity over time. Each quick fix adds a little more. Unless you fight it. Brooks: "There is no silver bullet." Lehman: "Complexity increases unless you work to reduce it." Same argument. Different vocabularies.

"The unit cost of change must initially be made as low as possible and its growth, as the system ages, minimized. Programs must be made more alterable, and the alterability maintained throughout their lifetime. The change process itself must be planned and controlled."

This is Lehman's practical prescription. Make the system alterable. Maintain alterability. Plan the change process. Control it. Most organizations do none of these. They make the system work, then react to change requests, then wonder why each change is harder than the last. The alterability was never designed in. It was assumed. The assumption was wrong.

III. Self-Regulation (1974)

"Program evolution is subject to a dynamics which makes the programming process, and hence measures of global project and system attributes, self-regulating with statistically determinable trends and invariances."

Lehman observed that OS/360's growth data showed patterns "typical of a self-stabilising process with positive and negative feedback loops." The rate of system growth was self-regulatory despite varying budgets, varying team sizes, varying management attitudes. The process finds its equilibrium.

"Individual decisions may appear localised and independent, but their aggregation, moderated by many feedback relationships, produces overall system responses that are regular and often normally distributed."

This is the law that should make managers uncomfortable. You cannot accelerate software evolution by adding resources. The process has its own pace, determined by feedback loops among users, developers, and the codebase. Brooks's Law — "adding manpower to a late software project makes it later" — is a special case of Lehman's Self-Regulation. The process resists perturbation. It returns to its natural rate.

IV. Conservation of Organizational Stability (1978)

"During the active life of a program the global activity rate in a programming project is statistically invariant."

The amount of work a team produces per release is roughly constant. Change the team. Change the tools. The output stays approximately the same. This is not about individual productivity. It is a statistical observation about organizations. The organization has a natural throughput. It can be measured. It cannot be wished higher.

V. Conservation of Familiarity (1978/1980)

"During the active life of a program the release content (changes, additions, deletions) of the successive releases of an evolving program is statistically invariant."

Each release contains roughly the same amount of change. Not because anyone plans it. Because the organization can only absorb so much change at once. Ship more — quality drops, bugs increase, the next release is smaller to compensate. The average holds. Law III operating through Law V.

VI. Continuing Growth (1991)

The functional content of an E-type system must grow to maintain user satisfaction. Users demand more features. Growth is not optional. Managing growth is the entire job. A system that doesn't grow is a system users abandon for one that does. The growth is demanded. The cost of growth is Law II. The two laws together: you must grow, and growing increases complexity. This is the tension that defines the economics of software.

VII. Declining Quality (1996)

The quality of an E-type system declines unless it is rigorously maintained and adapted to environmental change. Quality is relative to the environment. The environment moves. What was excellent in 2019 is unmaintainable legacy in 2026, even if the code hasn't changed. The standards rose. The code stayed where it was.

Lehman on the practical consequence:

"Top-level managerial pressure to apply life-cycle evaluation is therefore essential if a development and maintenance process is to be attained that continuously achieves desired overall balance between the short- and long-term objectives of the organization."

Short-term: ship the feature. Long-term: maintain the structure. The two are in tension. Management must enforce the balance. If management only rewards shipping, structure degrades. If management only rewards structure, nothing ships. The balance is the job. Most organizations don't recognize it as a job. They recognize shipping. They wonder why quality declines.

VIII. Feedback System (1996)

"The global software process that includes technical, business, marketing, user and other activities constitutes a multi-loop, multi-level feedback system. To change the characteristics of such a system requires one to consider, design or adapt and tune both forward and feedback paths to achieve the desired changes in externally visible behaviour."

Software evolution is a multi-loop, multi-agent feedback system. Users provide feedback. Developers respond. The system changes. The changes generate new feedback. The loops interact. Emergent behavior. You cannot control it. You can only participate in it.

"Current world-wide process models and improvement activities focus primarily on the forward technical path and overlook the many feedback paths and the constraints that they impose on improvement."

Most process improvement focuses on the forward path: build better, test better, deploy better. Lehman's point is that the feedback paths — how users react, how the market responds, how the organization learns — are equally important and almost entirely ignored. You improved the build pipeline. You didn't improve the organization's ability to learn what to build. The forward path is faster. The feedback path is still broken. The system as a whole is not improved. It is accelerated toward the wrong destination.

What Lehman and his peers agreed on

Lehman, Brooks, Parnas, and Dijkstra arrived at the same destination from different starting points. Lehman measured systems and derived statistical laws. Brooks managed systems and derived engineering principles. Parnas decomposed systems and derived design criteria. Dijkstra thought about systems and derived epistemological critiques. All four concluded: change is inevitable, complexity is the enemy, and the process cannot be controlled — only influenced.

Lehman in 1980:

"Any program is a model of a model within a theory of a model of an abstraction of some portion of the world or of some universe of discourse."

A program is not the world. It is a model of a model of a theory of a model of an abstraction. That is four levels of indirection. Each level introduces error. Each level changes independently. The program must track changes across all four. This is epistemologically ambitious. It is also what every business application attempts. Lehman understood the difficulty. Most project plans don't.

Brooks in 1986:

"The hardest single part of building a software system is deciding precisely what to build. Therefore the most important function that the software builder performs for the client is the iterative extraction and refinement of the product requirements."

The requirements are not known. They are discovered. The discovery process is iterative. Lehman's Laws are the dynamics of that iteration at scale. The iteration doesn't stop at version 1.0. It continues for the life of the system. The life of the system is the iteration.

"There is no royal road, but there is a road." — Brooks

Dijkstra in 1972:

"The major cause of the software crisis is that the machines have become several orders of magnitude more powerful. As long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming has become an equally gigantic problem."

The machines got faster. The problems got bigger. The complexity grew with the capability. Lehman's Laws describe the dynamics of that growth. More powerful machines don't reduce complexity. They enable larger systems, which are more complex. The complexity is the thing. The machine is the substrate.

Weinberg, in The Psychology of Computer Programming (1971), identified the human dimension that Lehman's statistical laws abstract over: programming is done by people, in organizations, under pressure. The pressure to ship degrades structure. The degradation is organizational before it is technical. Lehman's laws describe the aggregate. Weinberg describes the individual decisions that produce the aggregate. Both are necessary. Neither is sufficient alone.

The entropy connection

In 2023, Torres, Baltes, Treude, and Wagner applied information theory to software evolution. Two entropy definitions — structural (dependency graph) and textual (compression ratio) — were tracked across 25 open source projects.

"Both entropy measures display weak and unstable correlations with other complexity metrics."

Entropy captures something traditional metrics miss: the information content of the codebase and how it changes. Lehman's Law II said complexity increases unless work is done to reduce it. Lehman and Belady originally called it entropy. The Torres paper returns to the original framing and gives it measurement. Structural entropy rises when the dependency graph becomes more interconnected. Textual entropy rises when the codebase becomes more information-dense. Both rise when complexity, in Lehman's sense, is accumulating.

"An unexpected high frequency of events where there is considerable change in the information content of the project."

These are surprisal events. Commits where the information structure jumps significantly relative to history. A refactoring. A deletion that simplifies the dependency graph. A new module that reconfigures everything. These are the commits where Law II is being obeyed or violated. Traditional metrics see size. Entropy sees structural impact. Lehman predicted these events would matter. The Torres paper gives us a way to flag them. A system's structural entropy, tracked over time, is a measure of whether Law II is being respected. Rising entropy: complexity accumulating. Stable entropy: the team is doing the work. Dropping entropy: the team is actively simplifying. Most codebases rise. Few are stable. Almost none drop. The measurement confirms what Lehman predicted in 1974.

What to do

Lehman's Laws are not despair. They are realism. The laws describe what happens. How you respond is a choice.

Accept that maintenance is the process. Law I says you will change the system forever. Not because you built it wrong. Because the world changes. Stop treating maintenance as a phase after development. Development is the first phase of maintenance. Everything after is the rest of it.

Budget for complexity reduction. Law II says complexity increases unless you fight it. Fighting costs time and money. Budget for it. Refactoring is not a sign of past failure. It is the necessary work of preventing future failure. If your organization treats it as a luxury, your complexity is rising at a rate determined by your change velocity and your neglect of structure. Measure it. Or choose not to know. The complexity doesn't care.

Measure entropy. The tools exist. Dependency graph entropy. Compression-ratio entropy. Track them. Flag surprisal commits. These are the commits where the information structure changes significantly — and they are the commits your size-based review will miss. Size is easy. Structure is more important.

Hire for taste. All eight laws describe a system that resists management. The only counterforce is good design decisions, made early and defended persistently. That means good designers. Brooks spent his career arguing this. Lehman's Laws are the evidence Brooks was right. The process won't save you. Only taste will.

"The central question in how to improve the software art, centers, as it always has, on people." — Brooks, 1986


References:

  • M.M. Lehman, "Programs, Life Cycles, and Laws of Software Evolution," Proceedings of the IEEE, Vol. 68, No. 9, September 1980, pp. 1060-1076.
  • M.M. Lehman, "Laws of Software Evolution Revisited," Proceedings of the 5th European Workshop on Software Process Technology, 1996.
  • M.M. Lehman, "Feedback in the Software Evolution Process," Information and Software Technology, Vol. 38, 1996.
  • L.A. Belady and M.M. Lehman, "A Model of Large Program Development," IBM Systems Journal, Vol. 15, No. 3, 1976.
  • L.A. Belady and M.M. Lehman, "Programming System Dynamics or The Metadynamics of Systems in Maintenance and Growth," IBM Research Report RC3546, 1971.
  • M.M. Lehman and L.A. Belady, Program Evolution: Processes of Software Change, Academic Press, 1985.
  • Frederick P. Brooks, Jr., "No Silver Bullet: Essence and Accidents of Software Engineering," Computer Magazine, April 1987.
  • Frederick P. Brooks, Jr., The Mythical Man-Month, Addison-Wesley, 1975 (Anniversary Edition, 1995).
  • David L. Parnas, "On the Criteria to Be Used in Decomposing Systems into Modules," Communications of the ACM, Vol. 15, No. 12, December 1972.
  • David L. Parnas, "Software Aging," Proceedings of the 16th International Conference on Software Engineering, 1994.
  • Edsger W. Dijkstra, "The Humble Programmer," Communications of the ACM, Vol. 15, No. 10, October 1972 (Turing Award Lecture).
  • Edsger W. Dijkstra, "On the cruelty of really teaching computing science," EWD 1036, 1988.
  • Gerald M. Weinberg, The Psychology of Computer Programming, Van Nostrand Reinhold, 1971.
  • Adriano Torres, Sebastian Baltes, Christoph Treude, Markus Wagner, "Applying Information Theory to Software Evolution," NLBSE 2023. arXiv:2303.13729

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

Software does not age. It evolves. The environment changes around it. The change forces adaptation. Adaptation increases complexity. Complexity must be fought. The fight is continuous. The continuity is the discipline of software engineering.

I, Pencil

No single person on earth knows how to make a pencil. Or a computer mouse. Or any non-trivial software system. The knowledge is distributed across millions of minds, coordinated by exchange. This is the economics that software engineering forgot.

i-pencilmatt-ridleydistributed-knowledgesoftware-economicsspecialization

In 1958, Leonard Read published an essay titled "I, Pencil." It is written in the voice of a pencil, describing its own creation. The pencil traces its family tree: cedar from Oregon, felled by loggers using saws made of steel alloyed with nickel from Canada. Graphite from Sri Lanka, mined by workers using equipment powered by diesel refined from oil drilled in the Middle East. Clay from Mississippi. Candelilla wax from Mexico. Rapeseed oil from Indonesia. Pumice from Italy. The lead is not lead at all — it is graphite mixed with clay, baked, treated with a mixture of fats and waxes. The eraser is rubber from Malaysia, vulcanized with sulfur, bonded with a brass ferrule made of zinc and copper. The paint is six coats of lacquer. The label is a film of carbon black pressed into the wood.

"Simple? Yet, not a single person on the face of this earth knows how to make me."

That is the sentence. The pencil is simple. A child can use it. It costs a few cents. Nobody knows how to make one.

"Actually, millions of human beings have had a hand in my creation, no one of whom even knows more than a very few of the others. There isn't a single person in all these millions, including the president of the pencil company, who contributes more than a tiny, infinitesimal bit of know-how."

The knowledge is distributed. No master mind coordinates it. No central planner designs the supply chain. The loggers, miners, chemists, machinists, and factory workers do not collaborate. Most have never met. Most do not know their work contributes to pencils. They exchange their tiny know-how for wages and trade those wages for goods they need. The pencil emerges from the exchange. It is a product of the market — of millions of specialized know-hows coordinated not by command but by price.

"There is a fact still more astounding: the absence of a master mind, of anyone dictating or forcibly directing these countless actions which bring me into being. No trace of such a person can be found. Instead, we find the Invisible Hand at work."

"The lesson I have to teach is this: Leave all creative energies uninhibited. Permit these creative know-hows freely to flow. Have faith that free men and women will respond to the Invisible Hand."

When ideas have sex

In 2010, Matt Ridley stood on the TED stage and gave the talk that extends Read's pencil into a general theory of innovation. The title: "When Ideas Have Sex."

Ridley's argument is that human progress is not driven by individual intelligence. It is driven by the recombination of ideas through exchange. Biological evolution accelerates when sex mixes genes from different lineages. Cultural evolution accelerates when trade mixes ideas from different minds. The computer mouse is not one idea. It is a confection of plastic, lasers, transistors, ergonomics, injection molding, USB protocols — each the product of thousands of specialized minds, none of whom could make a mouse alone.

"It's not important how clever individuals are; what really matters is how smart the collective brain is."

"There is nobody on the planet who knows how to make a computer mouse. I mean this quite seriously."

The pencil and the mouse are the same argument at different technological scales. The pencil required global supply chains of raw materials. The mouse requires global supply chains of ideas. The principle is identical. Distributed knowledge, coordinated by exchange, produces objects no individual understands. The collective brain is smarter than any of its neurons. The neuron's job is not to know everything. It is to connect.

Ridley contrasts two objects of similar size and shape, both designed to fit the human hand. The Acheulean hand axe was made by Homo erectus to an unchanging design for 30,000 generations — a million years of hitting things with the same rock. The computer mouse is obsolete within five years. The difference is not individual intelligence. Homo erectus had brains. They had no exchange networks. Their ideas did not have sex. Human ideas do.

"It's the interchange of ideas, the meeting and mating of ideas between them, that is causing technological progress, incrementally, bit by bit."

Ridley's darkest example: Tasmania. When rising sea levels isolated Tasmania from mainland Australia 10,000 years ago, its population of 4,000 was too small to sustain the specialization needed to maintain bone tools, fishing equipment, and cold-weather clothing. They lost technologies their ancestors had possessed for millennia. Not because they became less intelligent. Because the network was too small. The collective brain atrophied. Tierra del Fuego, connected by a land bridge, kept its technology. The difference was not genetics. It was exchange.

The software pencil

Every non-trivial software system is a pencil. Nobody knows how to make it.

The Linux kernel: 30 million lines of code, contributed by thousands of developers, running on hardware designed by thousands of engineers, compiled by toolchains built by hundreds of contributors, deployed on servers manufactured across global supply chains. The kernel maintainer does not know how to make a server. The server manufacturer does not know how to write a scheduler. The knowledge is distributed. The system emerges from the exchange — not of money, in the kernel's case, but of patches, reviews, and trust. Open source is the pencil argument applied to software. No master mind. Distributed know-how. Emergent order.

"The lesson I have to teach is this: Leave all creative energies uninhibited."

The open source movement learned Read's lesson instinctively. Permit contributions to flow. Have faith that distributed developers will respond to something — not the Invisible Hand of price, but the visible hand of reputation, utility, and the satisfaction of solving hard problems with smart people. The mechanism is different. The principle is the same. The knowledge is distributed. The coordination is emergent. The product exceeds any individual's understanding.

The economics of software specialization

Read and Ridley describe a world where specialization and exchange produce objects of extraordinary complexity with no central coordination. Software engineering has not internalized this. The dominant model of software production is still hierarchical: architects design, developers implement, managers coordinate. The organization chart is the architecture. Conway's Law guarantees it.

But the pencil suggests a different model. Software as emergent from specialized modules, each produced by a team that knows only its own domain, exchanging through stable interfaces. This is microservices done right — not better monoliths, but genuine specialization coordinated by interface contracts rather than organizational hierarchy.

Parnas's information hiding is the software equivalent of market exchange. Each module hides its implementation. Each module exposes only what others need. The modules do not collaborate. They exchange. The interface is the price mechanism — it communicates what is available and what is required without revealing how either is produced. The system emerges from the exchange of specialized know-hows. No module knows how the whole system works. No module needs to.

"There is a fact still more astounding: the absence of a master mind."

A well-architected software system has no master module. No orchestrator that knows everything. The knowledge is distributed across components, each responsible for one hidden decision. This is Parnas's vision, implemented. It is also Read's vision, applied to code. The pencil and the module are the same argument in different media.

Task automation economics: the asset as exchange

The task automation economics paper discussed earlier on this blog argues that the economic unit of software production is not the agent run but the verified automation asset — a released object with explicit specification, evidence, and a defined interface. This is the pencil argument applied to automation.

A verified automation asset is a unit of specialized know-how, encapsulated behind a stable interface, exchangeable across teams. One team produces the asset. Another team consumes it. Neither knows how the other works. The interface is the contract. The exchange is the mechanism. The system's capability grows not through central planning but through the accumulation and recombination of specialized assets. This is Ridley's "ideas having sex" implemented in software infrastructure. Each asset is an idea. Each pipeline that composes assets is an idea mating. The collective brain of the organization grows with each new interface, each new composition, each new recombination of existing capabilities.

The dark factory extends this further. Specs go in. Software comes out. The spec writer does not know how the agents generate the code. The agents do not know what the spec means. The validation layer does not know why the spec was written. Each component knows its fragment. The whole exceeds the parts. The factory is the pencil. No component knows how to make the software. The software emerges anyway.

The limits of the metaphor

Read's pencil has a weakness. It assumes the Invisible Hand produces optimal outcomes without specifying the conditions under which it fails. Markets produce pencils efficiently because the pencil's design is stable, its components are well-specified, and its interfaces — the size of the ferrule, the hardness of the graphite, the diameter of the wood casing — are standardized. When interfaces are unstable, markets fail. The pencil works because the specifications are fixed.

Software interfaces are rarely fixed. They change. They drift. They accumulate assumptions that become invalid. Lehman's Laws apply: E-type systems must evolve, and evolution increases complexity. The market mechanism — stable interfaces, specialized producers, emergent coordination — breaks down when the interfaces themselves are changing. This is why software architecture is harder than pencil manufacturing. The pencil's architecture is stable. The software's architecture is not.

The insight is not that software should be more like pencils. It is that the parts of software that are stable should be modularized behind stable interfaces, and the parts that are volatile should be contained. Parnas again: hide the volatile decisions. Expose the stable ones through interfaces that don't change. Where the interface is stable, the market works. Where the interface shifts, the market fails, and coordination requires something closer to hierarchy. The art of software architecture is knowing which parts are which.

The collective brain of a codebase

Ridley's collective brain is not a metaphor. It is a description of how a well-architected codebase actually works when it works well. Each module contributes specialized knowledge. Each interface enables exchange. Each composition of modules is an idea mating — a recombination of existing capabilities into something new. The system's intelligence is not in any single component. It is in the connections between them.

"It's not important how clever individuals are; what really matters is how smart the collective brain is."

Replace "individuals" with "modules." Replace "collective brain" with "architecture." The sentence remains true. A system of simple, well-specified modules connected by stable interfaces is smarter than a monolith written by geniuses. The monolith is a single mind. The modular system is a collective brain. The collective brain outlives its components. The monolith dies with its authors.

Brooks argued that conceptual integrity requires one mind. Ridley argues that progress requires millions of minds exchanging ideas. The resolution: one mind designs the interfaces. Millions of minds implement behind them. The interfaces are the market. The implementations are the specialized producers. The system has conceptual integrity because one mind controlled the interfaces. The system has distributed intelligence because many minds contributed the implementations. This is the pencil argument synthesized with Brooks. The pencil has conceptual integrity — it is clearly one thing, designed for one purpose. The pencil is also the product of millions of specialized know-hows coordinated by exchange. The pencil is both. Good software is both.


References:

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

No single person knows how to make a pencil. The knowledge is distributed across millions of minds, coordinated by prices, not by a plan. The pencil is the product of the market. The market is the mechanism. The mechanism is the architecture of cooperation without coordination.

Git is a Unix tool

Git was designed by someone who hated CVS, understood filesystems, and believed in conceptual integrity. Its best features — the object model, plumbing and porcelain, branches as 40-byte pointers — are the ones Brooks and Parnas would have designed.

gitunixlinus-torvaldsconceptual-integrityinformation-hiding

In April 2005, the Linux kernel project lost access to BitKeeper, its proprietary version control system. Andrew Tridgell had reverse-engineered the BitKeeper protocol. The license was revoked. Linus Torvalds halted all kernel development and wrote a replacement. He called it Git.

"Writing code is easy. Getting a good design is what matters. So there was a fair amount of background to those few days that is pretty important, and that part doesn't show up in the history." — Linus Torvalds

The "few days" is the myth. The background is the reality. Linus had been thinking about version control design for months. He had strong opinions about what was wrong with every existing system — CVS, SVN, BitKeeper, the commercial alternatives. When the crisis hit, the design was ready. The implementation took two weeks.

"I based pretty much all of the git design on three basic goals: performance, distribution, and integrity checking. Everything else pretty much flows from those three things."

Performance. Distribution. Integrity. Three goals. Everything else emerges. This is conceptual integrity in Brooks's sense: a small set of orthogonal primitives from which the entire system follows. The design feels like one mind made it — because one mind did.

The Unix philosophy, applied to version control

Linus describes Git's relationship to Unix explicitly:

"I kind of compare it to Unix. Unix has like a core philosophy of everything is a process, everything is a file, you pipe things between things. There's the simple concepts that underlie the philosophy, but then all the details are very complicated. I think Git has some of the same kind of — there's a fundamental core simplicity to the design and then there's the complexity of implementation."

Unix: everything is a file. Git: everything is an object identified by its content hash.

The mapping is direct. In Unix, the file is the universal abstraction — storage, devices, sockets, pipes all present the same interface. In Git, the content-addressed object is the universal abstraction — files (blobs), directories (trees), history (commits), releases (tags) all use the same storage mechanism. You insert content. Git returns a hash. You ask for the hash. Git returns the content. The key-value store is the kernel. The version control system is a user interface built on top.

This is the core design insight. Git is not a version control system with a storage layer. It is a content-addressable filesystem with a version control interface. The Pro Git book states it directly: "Git is fundamentally a content-addressable filesystem with a VCS user interface written on top of it." The filesystem is the mechanism. The VCS is the policy. Mechanism, not policy — the oldest Unix principle, applied to the problem Linus understood best.

The object model: four primitives

Git's conceptual integrity rests on four object types. No more. Each is content-addressed. Each is immutable once written. Each composes with the others.

Blob. File contents. No filename. No metadata. No permissions. Just the bytes, compressed, identified by their SHA-1 hash. Two files with the same content produce the same blob. Deduplication is free. Identity is content. Content is identity. This is the simplest possible file abstraction. It is also the most powerful — because it makes no assumptions about what the bytes mean.

Tree. A directory listing. Names mapped to hashes. A tree points to blobs (files) and other trees (subdirectories). It records the structure without recording anything about the content. The tree is the namespace. The blob is the data. Separate concerns. Separate objects.

Commit. A pointer to a tree (the snapshot), plus metadata: author, committer, timestamp, message, and zero or more parent commits. The parent pointers form a directed acyclic graph. History is the DAG. The DAG is traversed. Branches are just refs pointing to commits. Merges are commits with multiple parents. The entire history model falls out of the commit object's parent pointers.

Tag. A named reference to any object, typically a commit, with an optional message and GPG signature. Used for releases. Lightweight. Immutable once created. The tag on the object. Not the object itself.

Four types. That is the entire storage model. Every Git repository is a key-value store containing blobs, trees, commits, and tags. Every Git operation — commit, merge, rebase, cherry-pick, bisect — is a manipulation of these four types, referenced by hash, organized into a DAG. The complexity is in the operations. The substrate is minimal.

Plumbing and porcelain: information hiding applied

Git was architected from the start as two layers:

Plumbing. Low-level commands that manipulate the object store: git hash-object, git cat-file, git write-tree, git commit-tree, git update-ref. These are stable, scriptable, composable. They do one thing. They can be piped. They are the Unix toolset applied to a content-addressable database.

Porcelain. High-level commands that provide user workflows: git commit, git merge, git log, git rebase. Built on plumbing. Cosmetic. Replaceable. The first version of Git had no porcelain — you committed by running git commit-tree and manually writing the resulting hash to .git/HEAD. Linus built the mechanism first. The user interface came later, contributed by others, layered on top.

This is Parnas's information hiding in system architecture. The plumbing hides the object store's implementation — the compression algorithm, the pack format, the delta encoding — behind stable interfaces. The porcelain hides the workflow complexity behind user-friendly commands. Each layer's internals can change without affecting the other. Git's pack format has been revised multiple times. The SHA-1 to SHA-256 migration is underway. The porcelain has grown from a handful of commands to hundreds. The plumbing interfaces are the same.

"I approached it more like I would a distributed journaling filesystem, not really a traditional SCM." — Linus Torvalds

He designed a filesystem. The filesystem's interface is stable — put content, get hash; give hash, get content. The VCS is a consumer of the filesystem. The separation is clean. The hiding is real. Twenty years later, the original plumbing still works. The porcelain has changed beyond recognition. The architecture absorbed the change because the volatile decisions were hidden behind the right interfaces.

The ultra-minimalist features

Git's best features are the ones it doesn't have. Or the ones it implements so simply they seem trivial. Brooks would recognize them as examples of conceptual integrity achieved through refusal to add complexity.

Branches are 40-byte pointers. A branch is a file in .git/refs/heads/ containing a single SHA-1 hash. Creating a branch is writing 40 bytes. Switching branches is changing which ref HEAD points to. Merging is creating a commit with two parents. Branching in CVS meant copying the entire repository. In Git, it means writing 40 bytes. The cost is so low that branching stops being a decision. It becomes a reflex. This changes how people work — not because Git preached branching, but because the cost made it invisible.

No rename tracking. Traditional SCMs track file renames by recording metadata: "file A was renamed to file B." Git doesn't. Renames are inferred from content similarity at query time. If you rename a file and modify 90% of it, Git sees a delete and a create. If you rename a file and modify 10%, Git sees a rename with modifications. The algorithm runs when you ask git log --follow. The decision is deferred to the reader, not encoded by the writer. This is information hiding applied to history: the rename is an interpretation, not a fact. Different tools can interpret differently. The history is not polluted with metadata that later proves wrong.

Everything is local. git init creates a .git directory. That directory contains the entire repository — objects, refs, config, hooks, the index. Clone it, copy it, back it up. Every clone is a full backup. Every developer has the complete history. No network required for commit, branch, merge, log, bisect, or blame. The server is just another clone. The distributed nature is not a feature bolted onto a centralized model. It is the model. The centralized workflow is a convention built on top of a distributed substrate.

Integrity is automatic. Every object is named by its content hash. Any corruption anywhere is immediately detectable. History is tamper-evident — changing any past commit changes all subsequent hashes. Trust is cryptographic, not social. You don't need to trust the server. You don't need to trust your colleagues. You can verify. The SHA-1 was never about security. It was about detecting accidental corruption. The fact that it also detects malicious tampering is a side effect.

"People kind of think that using the SHA-1 hashes was a huge mistake. But to me, SHA-1 hashes were never about the security. It was about finding corruption." — Linus Torvalds

The design heuristic: WWCVSND. What Would CVS Not Do? Linus hated CVS. He saw SVN as "lipstick on a pig." His design method was systematic inversion: distributed instead of centralized, whole-tree snapshots instead of per-file history, content-addressable instead of incrementally versioned, local instead of networked, lightweight branches instead of heavy copies. Every design decision was the opposite of what CVS had done. The result was not a better CVS. It was a different category of thing.

What Brooks and Parnas would say

Brooks would recognize Git's conceptual integrity immediately. Four object types. One identification scheme. One DAG model. One storage layer. A system that feels like one mind designed it — because one mind did. The complexity is in the operations on the model, not in the model itself. The model is stable. The operations evolve. This is the architecture of systems that age well.

Brooks argued that conceptual integrity requires one mind. Git had one mind for its critical design phase. Linus designed the core. Junio Hamano maintained it. The porcelain grew from contributions. The plumbing stayed stable. The one-mind rule held where it mattered — at the object model, the storage layer, the fundamental abstractions. The community built on top. The foundation didn't shift.

Parnas would recognize the information hiding. The plumbing hides the object store. The porcelain hides the plumbing. The object model hides the storage format. The content addressing hides the transport mechanism. Each volatile decision is encapsulated behind a stable interface. The pack format changed. The network protocol changed. The hashing algorithm is migrating. The interface — insert content, get hash; give hash, get content — has not changed in twenty years. This is Parnas's criterion applied to system infrastructure. Hide the decisions likely to change. Expose stable interfaces. Let the rest of the system evolve behind them.

Parnas would also recognize the design method as a form of information hiding applied to project structure. The plumbing is the stable core, maintained by a small group who understand it deeply. The porcelain is the volatile periphery, contributed by the community, evolving rapidly. The interface between them — the plumbing commands — is the contract. As long as the contract holds, the two layers can evolve independently, at different speeds, by different people, with different governance. This is modularity in organizational form.

The thing Git got right that most software gets wrong

Git's defining achievement is not any individual feature. It is that the core design has not needed to change. The object model is twenty years old. The DAG model is twenty years old. The plumbing is twenty years old. The system has scaled from the Linux kernel — the largest software project in history — to individual developers' dotfiles, without changing the fundamental abstractions. The same git commit works for a monorepo with millions of files and a single-file hobby project. The primitives composed.

Most software systems fail this test. The abstractions that worked at version 1.0 cannot handle version 10.0. The design decays. The model fractures. Git's model didn't fracture because the model was minimal. Four object types. One DAG. One content-addressing scheme. There was nothing to fracture. The minimalism was not an aesthetic choice. It was an engineering strategy. Less to design. Less to implement. Less to break. Less to regret.

Linus understood this, whether or not he would use Brooks's language. He built the simplest thing that could work for his problem — Linux kernel development — and nothing more. He refused to design for use cases he didn't have. He refused to add features he didn't need. He refused to generalize beyond the concrete problem in front of him.

"I'll do something that works for me, and I won't care about anybody else. And really that showed in the first few months and years — people were complaining that it was kind of hard to use, not intuitive enough. And then something happened, like there was a switch that was thrown."

The switch was GitHub. The community built the porcelain. The community built the hosting. The community built the tutorials, the GUIs, the integrations. Linus built the content-addressable filesystem with the DAG on top. The world built everything else. This is the pencil argument applied to version control: no single person knows how to make the whole Git ecosystem. Millions contributed their tiny know-how. The core, the part one mind designed, remained stable. The periphery, the part millions contributed to, evolved rapidly. The interface between them — the plumbing — held. That is the architecture of a system that will outlive its creator.


References:

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

Git is not a version control system. It is a content-addressable filesystem with a version control interface. The interface is porcelain. The filesystem is plumbing. The plumbing is stable. The porcelain evolves. The architecture survives because the interface between them has not changed in twenty years.

Engineering is art and philosophy, grounded in economic law

Engineering has three layers. The top is art: taste, judgment, the feel for what is right. The middle is philosophy: principles, values, what to optimize. The foundation is economics: scarcity, constraints, trade-offs. Art without economics is fantasy. Philosophy without economics is empty.

engineeringeconomicsartphilosophysoftware-design

Engineering has three layers. The top is art. The middle is philosophy. The foundation is economics. Most engineers operate in the top layer. Great engineers operate in all three. The ones who ignore the foundation build beautiful things that fail.

The foundation: economics

The foundation is economics because the foundation is always economics. Before you can make anything beautiful, before you can decide what principles to follow, you must confront the fact that resources are finite. Time is finite. Attention is finite. Complexity budget is finite. You cannot do everything. You must choose. The choice of what to build and what to leave unbuilt is the first decision. It is an economic decision. Everything above it depends on it.

"The first lesson of economics is scarcity: There is never enough of anything to satisfy all those who want it. The first lesson of politics is to disregard the first lesson of economics." — Thomas Sowell

The first lesson of engineering is also scarcity. You have finite means — developer hours, cognitive capacity, compute, money. You have infinite ends — features, optimizations, refactors, experiments. The means have alternative uses. Every hour spent on one thing is an hour not spent on another. This is not a metaphor. It is a structural fact. The structure is economic. Denying it does not remove it. Denying it makes it operate invisibly. Invisible constraints produce worse decisions than visible ones.

Lionel Robbins defined the structure in 1932:

"Economics is the science which studies human behaviour as a relationship between ends and scarce means which have alternative uses."

Ends. Means. Scarcity. Alternative uses. Four concepts. Every engineering decision involves all four. The engineer who does not think economically is making economic decisions without knowing they are economic decisions. The decisions are still economic. They are just worse.

The middle: philosophy

Above the foundation sits philosophy. Philosophy answers the question: given that resources are finite, what should we optimize? What principles should guide our choices? What values should the system embody?

Philosophy is not economics. Economics tells you that you must choose. Philosophy tells you what to choose. Economics tells you that every feature has a cost. Philosophy tells you that correctness matters more than features. Or that user experience matters more than correctness. Or that developer velocity matters more than either. The philosophy is a choice. The choice is made under scarcity. The scarcity is the foundation.

Brooks's conceptual integrity is a philosophical principle. One mind should control the design. The principle is not derived from economics. It is derived from a value: coherence is better than comprehensiveness. But the principle operates within economic constraints. One mind controls the design because attention is scarce. If attention were infinite, every design could be understood by everyone. The philosophy says what to optimize. The economics says why optimization is necessary.

Schumacher's Small Is Beautiful is a philosophical argument grounded in economic reality:

"Ever bigger machines, entailing ever bigger concentrations of economic power, do not represent progress: they are a denial of wisdom. Wisdom demands a new orientation of science and technology toward the organic, the gentle, the non-violent, the elegant and beautiful."

Small is not beautiful because small is virtuous. Small is beautiful because small is comprehensible. A system you cannot understand is a system you cannot control. The limit on comprehensibility is cognitive. The cognitive limit is a scarcity. The scarcity is economic. The philosophy — small is beautiful — is a response to the economic fact that human attention is finite.

Parnas's information hiding is a philosophical principle. Hide volatile decisions behind stable interfaces. The principle is derived from a value: systems should be resilient to change. But the principle operates because change is costly, and cost is economic. If change were free, hiding would be unnecessary. The philosophy says: design for change. The economics says: change is expensive, so contain it.

The top: art

Above philosophy sits art. Art is what you cannot derive from principles. It is taste. Judgment. The feel for what is right. The sense that this interface is elegant and that one is clumsy. Art is what Brooks meant when he wrote:

"The building of a design is the forcing of the will of one upon the stuff of the world."

The will of one. Not the calculation of one. The will. Design is an act of authority. It imposes coherence on a medium that has no opinion about coherence. The imposition is not rational. It is aesthetic. The designer feels that this shape is right and that shape is wrong. The feeling is taste. Taste is art.

Knuth called programming an art, not a science:

"The process of preparing programs for a digital computer is especially attractive, not only because it can be economically and scientifically rewarding, but also because it can be an aesthetic experience much like composing poetry or music."

The aesthetic experience is real. The programmer who has felt it knows it. The programmer who hasn't cannot be told. The feeling of a clean abstraction, a well-factored module, an interface that fits — these are aesthetic judgments. They are not derived from economics. They are not derived from philosophy. They are felt. The feeling is the art.

Dijkstra insisted that elegance was a practical property, not a decorative one:

"Simplicity is a prerequisite for reliability."

An elegant program contains fewer bugs because its structure is transparent. The elegance is aesthetic. The consequence — fewer bugs — is practical. The art serves the philosophy. The philosophy serves the economics. A simpler program costs less to maintain. The cost is economic. The simplicity is aesthetic. The aesthetic produces the economic outcome. It does not replace it.

Art without economic grounding produces beautiful systems that nobody uses. The architecture is elegant. The abstractions are clean. The system solves a problem nobody has at a cost nobody calculated. The art is real. The economics were ignored. The system fails.

Philosophy without economic grounding produces principles that sound correct and produce disaster. "Everything should be a microservice" is a philosophical principle. It sounds good. It ignores the economic reality that coordination has a cost, that distributed state is hard, that the complexity budget is finite. The principle was chosen without reference to the constraint. The constraint asserted itself anyway. The constraint always does.

The three-layer engineer

The three-layer engineer makes decisions that work at all three levels. They feel the right shape — the art. They know what to optimize — the philosophy. They know what constraints they're operating under — the economics. When the art says "this interface should be richer" and the economics says "there is no time," the philosophy decides which to sacrifice. The philosophy was chosen under scarcity. The scarcity is the ground.

Most engineers operate in one layer. The artist builds beautiful things that don't ship. The philosopher designs principles that don't survive contact with a deadline. The economist — rare, and usually not an engineer — cuts scope without understanding what was lost. The artist feels the loss. The philosopher can name the principle violated. The economist knows the constraint was real. All three are right. None is sufficient.

The three-layer engineer holds all three in tension. The tension is the work. The work is hard. The hardness is why most engineers never develop all three layers. The art takes years of building things and feeling which ones were right. The philosophy takes years of reading principles and testing which ones survive. The economics takes years of seeing projects fail for reasons that were always economic and were never named. The naming is the first step. The feeling is the last. Between them is the philosophy. Below them is the scarcity. Above them is the will.

"No solutions, only trade-offs." — Thomas Sowell

The trade-off is economic. The choice among trade-offs is philosophical. The feel for which trade-off is right is aesthetic. Engineering is all three. The foundation is the first. Without it, the other two are floating. They land on nothing.


References:

  • Lionel Robbins, An Essay on the Nature and Significance of Economic Science, Macmillan, 1932.
  • Thomas Sowell, Basic Economics, Basic Books, 2000.
  • E.F. Schumacher, Small Is Beautiful, Blond & Briggs, 1973.
  • Frederick P. Brooks, Jr., The Design of Design, Addison-Wesley, 2010.
  • Donald E. Knuth, "Computer Programming as an Art," Communications of the ACM, 1974.
  • David L. Parnas, "On the Criteria to Be Used in Decomposing Systems into Modules," Communications of the ACM, 1972.
  • Related posts: On Scarcity, The Unix philosophy, Brooks on Software Design

Scarcity is the universal engineering constraint. Time, attention, compute, complexity — every engineering decision is made within a budget. The budget is economic. The engineer who doesn't track the budget makes decisions blind. The engineer who tracks it makes decisions with full knowledge of the trade-off. The trade-off is the decision. The budget is the constraint. Scarcity is the unifying principle.

Engineering is decision-making under constraints. Economics is the science of choice under scarcity. The two are the same activity described in different vocabularies. The engineer who does not think economically makes economic decisions without knowing they are economic. The decisions are still economic. They are just worse.

the electric light was not a better candle

The electric light did not emerge from gas companies improving their mantles. Schumpeter named the force. Christensen named the mechanism. Thom named the moment. Software design lives inside all three.

disruptive-innovationschumpeterchristensencatastrophe-theorysoftware-architecture

The electric light did not come from the continuous improvement of candles. It did not come from gas companies making better mantles. It came from a different axis of value entirely — cleanliness, safety, convenience. The early bulbs were fragile, unreliable, expensive, and had no distribution infrastructure. On every metric the gas companies measured, the electric light was worse. By 1885, Edison held 75% of the U.S. market. The gas mantle was irrelevant.

Why did the gas companies not see it coming? Why did they not respond? Why did the disruption feel sudden when it had been building for years? Three thinkers, three pieces of the answer. Schumpeter named the force. Christensen named the mechanism. Thom named the moment. Together, they explain why most software "revolutions" are just better candles — and what the real electric light might be.

Schumpeter: the perennial gale

Joseph Schumpeter published Capitalism, Socialism, and Democracy in 1942. Chapter VII — six pages — introduced the concept that now bears his name.

"The fundamental impulse that sets and keeps the capitalist engine in motion comes from the new consumers' goods, the new methods of production or transportation, the new markets, the new forms of industrial organization that capitalist enterprise creates."

This is not an abnormal state. It is the normal state. Capitalism is change. Stability is the illusion.

"This process of Creative Destruction is the essential fact about capitalism. It is what capitalism consists in and what every capitalist concern has got to live in."

Creative Destruction. Not "innovation." Not "disruption." Destruction — and creation. Simultaneous. Inseparable. The new does not arrive after the old departs. The new is the departure of the old. You cannot have the electric light without destroying the gas industry. The destruction is not a side effect. It is the mechanism.

"Every piece of business strategy acquires its true significance only against the background of that process and within the situation created by it. It must be seen in its role in the perennial gale of creative destruction; it cannot be understood irrespective of it or, in fact, on the hypothesis that there is a perennial lull."

The perennial gale. Not a storm that passes. The wind that always blows. Strategy that assumes calm is strategy that assumes away the defining property of the environment. Most software architecture strategy assumes calm. The assumption is wrong.

"In capitalist reality as distinguished from its textbook picture, it is not that kind of competition which counts but the competition from the new commodity, the new technology, the new source of supply, the new type of organization — competition which strikes not at the margins of the profits and the outputs of the existing firms but at their foundations and their very lives. This kind of competition is as much more effective than the other as a bombardment is in comparison with forcing a door."

A bombardment, not forcing a door. Incumbents defend the door. The disruptor bombs the foundations. The door is irrelevant. The gas companies fortified their doors — better mantles, better distribution, better customer relationships. Edison bombed the foundations — a different kind of light, a different distribution model, a different value proposition. The gas companies never lost a door battle. They lost the foundation while they were reinforcing the door.

Schumpeter gives us the why. Why does the perennial gale blow? Because the fundamental impulse of the system is creation-through-destruction. The new good, the new method, the new market. Not better performance on existing metrics. New metrics. New markets. New foundations.

Christensen: the mechanism

If Schumpeter named the force, Clayton Christensen named the mechanism. The Innovator's Dilemma (1997) explains how great firms fail despite doing everything right.

"Generally, disruptive innovations were technologically straightforward, consisting of off-the-shelf components put together in a product architecture that was often simpler than prior approaches. They offered less of what customers in established markets wanted and so could rarely be initially employed there. They offered a different package of attributes valued only in emerging markets remote from, and unimportant to, the mainstream."

The electric light was simpler than the gas infrastructure. It offered less of what gas customers wanted — brightness, reliability, cost. It offered a different package valued by people who weren't gas customers. It was worse on the incumbents' metrics. Better on its own. The incumbents' customers didn't want it. The incumbents listened to their customers. They were destroyed.

"It was as if the leading firms were held captive by their customers."

"Blindly following the maxim that managers should keep close to their customers can be a fatal mistake."

The captivity is structural. The firm's resource-allocation processes reward sustaining innovations — better products for existing customers at higher margins. They penalize disruptive investments — worse products for nonexistent customers at lower margins. The managers are rational. The processes are rational. The outcome is fatal.

"The way decisions get made in successful organizations sows the seeds of eventual failure."

Christensen's prescription: create independent organizations, small enough to be excited by small markets, shielded from the parent's customers and cost structures. The disruption cannot be managed within the incumbent. It must be separated. The organizational structure is the problem. The organizational structure must be changed.

"Disruptive technology should be framed as a marketing challenge, not a technological one."

The technology is simple. The market is hard. The electric light was not a technology problem. Edison solved the filament in 1879. The problem was building a market that didn't exist for a product that was worse than the alternative on every dimension the existing market measured. That took twelve years to turn a profit. The incumbents' processes couldn't tolerate twelve years of losses on an inferior product. The disruptor's could. That is the mechanism.

Thom: the moment

If Schumpeter named the force and Christensen named the mechanism, René Thom named the moment. Catastrophe theory — introduced in Structural Stability and Morphogenesis (1972) — is the mathematics of discontinuous change. Continuously changing forces. Sudden, discontinuous effects.

"Catastrophe theory favors a dialectical, Heraclitean view of the universe, of a world which is the continual theatre of the battle between 'logoi,' between archetypes." — Thom

The universe as a theater of conflict between fundamental forms. Quantitative changes accumulate. Qualitative transformation erupts. The gas industry accumulated quantitative improvements — better mantles, better burners, better distribution — while the qualitative threat accumulated invisibly in a different market. The gas industry looked stable. The control variables were shifting. The catastrophe was approaching. Nobody could see it because the metrics they tracked were the old metrics. The new metric — "percentage of homes with electrical wiring" — was not on their dashboard.

The cusp catastrophe is the simplest model. One behavior variable. Two control factors: a normal factor and a splitting factor. As the splitting factor increases, the system develops two possible stable states. A small change in the normal factor can cause a sudden jump between them. The system exhibits five properties:

Bimodality. Two stable states exist simultaneously. Gas lighting and electric lighting coexisted for years. The market could support both. Until it couldn't.

Catastrophe. The jump between states is sudden. The market didn't shift gradually from gas to electric. It flipped. One year, gas was dominant. A few years later, it was irrelevant. The transition was not linear.

Hysteresis. The jump-down point differs from the jump-back-up point. Once the market flipped to electric, returning to gas would require electric to become much worse than gas was when gas was dominant. The threshold for switching back is higher than the threshold for switching forward. The new equilibrium is sticky. Disruption, once complete, is hard to reverse.

Inaccessibility. Intermediate states are unstable. You cannot be half-gas and half-electric for long. The transition is not a smooth gradient. It is a jump. The organization that tries to do both — sustain the old while developing the new — is in the inaccessible region. It falls to one side or the other. Most fall to the old side. The old side has revenue, customers, and organizational gravity.

Divergence. Small initial differences lead to dramatically different outcomes. Two gas companies, identical in 1879. One experiments with electric lighting, creates an independent division, shields it from the main business. The other doubles down on mantles. In 1885, one is an electric company. The other is bankrupt. The initial difference was small. The final divergence was total.

E.C. Zeeman, who popularized catastrophe theory in the 1970s, described the pattern directly:

"A gradual change in the control can cause a catastrophic sudden change in behavior. In all of nature we observe continuous changes giving rise to discontinuous jumps. In economics, a gradual relaxation after compression can cause a sudden inflationary explosion. People suddenly change opinion, and suddenly lose composure. Nations suddenly go to war."

Gradual change in the control variables. Catastrophic jump in the behavior. This is the moment Christensen's incumbents miss. They track the control variables they understand. The splitting factor — the new technology's trajectory, the new market's growth, the new axis of value — accumulates invisibly. The normal factor — market share in the old market, customer satisfaction among existing customers — looks fine. The system appears stable. The catastrophe is already determined. It just hasn't happened yet.

The three lenses on software

Schumpeter, Christensen, and Thom give us three ways to see software architecture decisions.

Schumpeter's lens: what is the perennial gale doing to your stack?

"This process of Creative Destruction is the essential fact about capitalism."

The essential fact about software is that the environments E-type systems serve are in perennial gale. Lehman's Law I — continuing change — is Schumpeter restated for code. The system must change or die because the world it models is being creatively destroyed. The new business model, the new regulation, the new user behavior — each is a Schumpeterian innovation that destroys the old assumptions the code was built on. The code does not age. The assumptions age. The assumptions are being bombarded. The code sits on the foundations.

Christensen's lens: are you improving candles or installing electric light?

Most software "innovation" is sustaining. Better monoliths. Better microservices. Better CI/CD. Better gas mantles. The customers — internal teams, product managers, the business — want better mantles. They reward better mantles. The engineers who build better mantles get promoted. The organization is held captive by its customers exactly as Christensen described.

The electric light — the true disruption — would serve non-consumers. Applications that can't be built under the current model. Users who can't afford the current model. Problems too small to justify the current cost structure. Dark factories may be electric light. Serverless was supposed to be. Most microservices migrations were better mantles on a monolith architecture. The mantle improved. The foundation didn't change.

Thom's lens: when does the catastrophe arrive?

The cusp catastrophe maps directly onto architecture evolution. The normal factor is the performance of the existing architecture on existing metrics — latency, throughput, developer productivity. The splitting factor is the divergence between what the architecture assumes and what the business needs. As the business changes, the assumptions embedded in the architecture become less valid. The splitting factor increases. The system develops bimodality — the old architecture and a possible new architecture exist as alternative stable states. The organization is in the inaccessible region between them. A small additional change in the business requirement — a new regulation, a new integration, a new scale threshold — triggers the catastrophe. The old architecture cannot accommodate it. The organization flips. The flip feels sudden. The splitting factor had been accumulating for years.

This is why architecture rewrites are always "unexpected" and "overdue" simultaneously. The catastrophe is visible in retrospect. The splitting factor — accumulated technical debt, architectural mismatch, assumption decay — was tracked by nobody. The normal factor — uptime, feature velocity — was tracked by everyone. The system looked stable. The catastrophe was already determined. It just hadn't happened yet.

The connection to Parnas

Parnas argued that modules should hide design decisions likely to change. This is Thom's catastrophe theory applied to software structure. The design decision is a control variable. If it changes, and the change propagates, the system experiences a catastrophe — a sudden large-scale restructuring triggered by a local change. Information hiding prevents propagation. The change stays inside the module. The catastrophe is contained.

"The criteria for module decomposition should be based on minimizing the propagation of change." — Parnas

Minimizing the propagation of change is minimizing the catastrophe surface. Every module boundary is a containment wall. The splitting factor rises inside the module. The catastrophe, when it comes, is local. One module flips. The rest of the system doesn't notice. This is the architecture of systems that survive the perennial gale. Most systems don't have it. The splitting factor rises globally. The catastrophe, when it comes, is total. The rewrite is announced. The organization flips. The old system is retired. The new system inherits the assumptions of its moment. The cycle begins again.

Schumpeter's gale blows. Christensen's incumbents fall. Thom's catastrophes erupt. Parnas's modules contain them. The four thinkers are one argument. The argument is about whether your architecture can survive the inevitable — not whether the inevitable can be avoided. It cannot.


References:

  • Joseph A. Schumpeter, Capitalism, Socialism, and Democracy, Harper & Brothers, 1942. Chapter VII: "The Process of Creative Destruction."
  • Clayton M. Christensen, The Innovator's Dilemma: When New Technologies Cause Great Firms to Fail, Harvard Business School Press, 1997.
  • René Thom, Structural Stability and Morphogenesis, W.A. Benjamin, 1972. (Translated by D.H. Fowler, 1975.)
  • E.C. Zeeman, "Catastrophe Theory," Scientific American, Vol. 234, No. 4, April 1976, pp. 65-83.
  • David L. Parnas, "On the Criteria to Be Used in Decomposing Systems into Modules," Communications of the ACM, Vol. 15, No. 12, December 1972.
  • Related posts: Brooks on Software Design series, Lehman's Software Evolution, Parnas's Information Hiding, Software dark factories, Henney's Microservices

Engineering is the discipline of building things that work within constraints. Every topic on this blog — operating systems, AI models, trading infrastructure, research labs, innovation economics — is examined through the lens of systems design. The lens is engineering. The method is: understand the constraints, design within them, verify the design works, iterate. The domain provides the specifics. The method is universal.

The electric light was not a better candle. It was a different category of thing, valued along a different axis. The incumbents who improved their candles did everything right and were destroyed anyway. The innovator's dilemma is not a failure of management. It is a property of the structure.

The knowledge is dispersed

Hayek wrote in 1945: 'The knowledge of the circumstances of which we must make use never exists in concentrated form, but solely as the dispersed bits of incomplete and frequently contradictory knowledge which all the separate individuals possess.' This is the most important sentence in economics for software engineers.

hayekknowledgedistributed-systemsteamsarchitecture

In 1945, Friedrich Hayek published "The Use of Knowledge in Society." The paper is about economics. It is also about software architecture. It is also about why your team's decisions are worse than you think, and why that's not their fault.

"The knowledge of the circumstances of which we must make use never exists in concentrated or integrated form, but solely as the dispersed bits of incomplete and frequently contradictory knowledge which all the separate individuals possess."

The knowledge is dispersed. Not concentrated. Not integrated. Dispersed. Across individuals. Each individual holds fragments. The fragments are incomplete — no one has the whole picture. The fragments are frequently contradictory — different people know different things that can't simultaneously be true. The contradiction is not a bug. It is a property of the system.

Hayek was writing about economic planning. His target was the idea that a central planner could allocate resources efficiently by gathering all relevant information and computing the optimal allocation. The information cannot be gathered because it doesn't exist in gather-able form. It exists as local knowledge — the farmer who knows which field drains poorly, the factory manager who knows which machine is about to break, the trader who knows which supplier is becoming unreliable. This knowledge cannot be transmitted to a central planner. It is tacit. It is local. It is constantly changing. The planner's model is always out of date. The market processes information through prices without requiring anyone to understand the whole. The price of copper rises. Users of copper use less copper. Nobody needs to know why the price rose. The price communicates the scarcity. The behavior adapts.

Software teams are Hayekian systems. The knowledge of what the system needs, what it costs, what will break, and what users actually do is dispersed across the team, the codebase, the incident history, the support tickets, and the production metrics. No single person holds it all. The architect who designs the system in advance is the central planner. Their model is out of date. The knowledge they need doesn't exist in the form they need it. It exists as the backend engineer who knows the database migration will take six weeks, the frontend engineer who knows the design system is being rewritten, the SRE who knows the load balancer can't handle the new traffic pattern, the PM who knows the requirements are changing next quarter. None of this knowledge appears in the architecture document. The architecture document is the plan. The plan is wrong.

The price system of software

Hayek's insight is that markets solve the knowledge problem through prices. Prices are signals. They communicate scarcity without requiring anyone to understand the whole. The price of lumber rises because of a supply disruption in Canada. A furniture maker in Texas uses less lumber. The furniture maker doesn't need to know about the supply disruption. The price tells them everything they need to know: lumber is more expensive. Use less.

Software systems need prices. They don't need literal money. They need signals that communicate scarcity without requiring the consumer to understand the cause. API rate limits are prices. When the rate limit triggers, the caller backs off. The caller doesn't need to know that the service is overloaded because a downstream database is slow. The 429 tells them: this service is scarce right now. Try again later. Queue depths are prices. When the queue grows, the producer slows down. The producer doesn't need to know that the consumer is processing a batch of large messages. The queue depth tells them: consumption is scarce. Circuit breaker states are prices. When the circuit opens, traffic routes elsewhere. The router doesn't need to know why the service is failing. The open circuit tells them: this service is unavailable. Route around it.

These are Hayekian mechanisms. They communicate dispersed knowledge through signals. The signals are local. The response is local. The system adapts without central coordination. The adaptation is the intelligence of the system. The intelligence is not in any component. It is in the signals between components.

What this means for teams

The knowledge dispersion has consequences for how teams should be structured. A team that makes decisions without consulting the people who hold the relevant local knowledge will make worse decisions. The knowledge exists. It is in the team. It cannot be extracted by a planning process. It must be surfaced by a decision process that involves the people who have it.

Conway's Law is a Hayekian observation. The system mirrors the communication structure because the communication structure determines what knowledge flows where. If the backend team and the frontend team don't talk, the API will be designed without knowledge of how the frontend actually uses it. The API will be clean and wrong. The knowledge of what the frontend needs existed in the frontend team. It didn't flow to the API designers. The API was designed without it. The design is worse than it could have been. The knowledge was dispersed. The communication structure didn't connect it.

The solution is not to eliminate the dispersion. The dispersion is irreducible. The solution is to design mechanisms that surface local knowledge at the point of decision. Code review surfaces the knowledge of the reviewer. Incident postmortems surface the knowledge of the responder. User research surfaces the knowledge of the user. Each mechanism connects dispersed knowledge to a decision that would otherwise be made without it. The mechanisms are not free. They cost time and attention — both scarce. The Hayekian engineer designs mechanisms that surface the most valuable knowledge at the lowest cost. The design is economic. The economics are Hayekian.

The architect as market designer

The architect who understands Hayek stops trying to design the system. They design the mechanisms by which the system designs itself. Stable interfaces are the prices. Services are the market participants. API contracts are the property rights. Automated testing is the enforcement mechanism. The architect sets the rules. The system evolves within them. The evolution produces outcomes the architect didn't anticipate. The outcomes are better than the architect could have designed because they incorporate knowledge the architect didn't have. The knowledge was dispersed. The market aggregated it.

"The price system is a mechanism for communicating information. The most significant fact about this system is the economy of knowledge with which it operates." — Friedrich Hayek

The economy of knowledge. The system operates without anyone needing to know the whole. The knowing is local. The responding is local. The coordinating is emergent. This is the architecture of systems that survive complexity. The complexity is in the world. The knowledge of it is dispersed. The system processes the dispersion without centralizing it. The processing is the architecture.


References:

Engineering is the discipline of building things that work within constraints. Every topic on this blog — operating systems, AI models, trading infrastructure, research labs, innovation economics — is examined through the lens of systems design. The lens is engineering. The method is: understand the constraints, design within them, verify the design works, iterate. The domain provides the specifics. The method is universal.

The knowledge is dispersed. No single person knows how to build the system. The architect who pretends to is making decisions with incomplete information. The market aggregates the dispersed knowledge through prices. The architecture must aggregate it through interfaces.

Cosmos SDK is the substrate for AI agents

The Cosmos SDK lets you build an application-specific blockchain from composable modules. For AI agents, this is the ideal substrate: sovereign execution, native interoperability, programmable governance, and protocol-level automation. Agents don't need smart contracts on a shared VM. They need their own chain. Cosmos gives them one.

cosmoscosmos-sdkibcblockchainai-agentssovereignty

AI agents need infrastructure. They need to hold assets, execute transactions, coordinate with other agents, and govern their own behavior according to rules they can verify. The dominant model — agents interacting with smart contracts on a shared blockchain — has limits. Shared block space means congestion. Shared governance means the agent's rules can be changed by people who don't understand the agent. Shared execution means the agent competes for compute with every other application on the chain.

The Cosmos SDK offers a different model. Each application gets its own blockchain. The blockchain is the application. The validators run the application's logic directly, not a generic VM that interprets the application's bytecode. The application controls its own blockspace, its own fee model, its own governance, and its own upgrade path. For AI agents, this is the ideal substrate: an autonomous system running on its own sovereign infrastructure, interoperating with other systems through a standardized protocol, governed by rules embedded in the chain itself.

What the Cosmos SDK is

The Cosmos SDK is a framework for building application-specific blockchains in Go. It is modular: you assemble your chain from pre-built modules — accounts, tokens, staking, governance, IBC — and add custom modules for your application's unique logic. The SDK provides the scaffolding. You provide the business logic. The result is a sovereign blockchain that does exactly what your application needs and nothing else.

The architecture has three layers:

Consensus: CometBFT. Byzantine Fault Tolerant consensus. Validators take turns proposing blocks. Finality is immediate — no probabilistic confirmation, no reorgs. Throughput up to 10,000 TPS. The consensus layer is application-agnostic. It doesn't know or care what the application does. It orders transactions. The application processes them.

Networking: IBC. The Inter-Blockchain Communication protocol. Trust-minimized packet passing between independent chains. Chains verify each other's consensus state through light clients. Packets are authenticated. Relayers carry packets between chains but cannot forge or modify them. The security model is: you trust the counterparty chain's validators, not the relayer. This is the "TCP/IP of blockchains." Any chain that implements IBC can communicate with any other chain that implements IBC. The network effect is horizontal — each new chain adds value for all existing chains.

Application: the module system. The SDK provides modules for common blockchain functionality: bank (token transfers), staking (validator delegation), governance (proposal voting), auth (account management), IBC (cross-chain communication). Custom modules extend the chain with application-specific logic. Modules have isolated state stores. They communicate through defined interfaces. The module system is the SDK's core abstraction. A module is a self-contained piece of blockchain logic with its own state, its own message handlers, and its own invariants. Modules compose. The composition is the chain.

Why AI agents need their own chain

Sovereignty. An agent running on a shared blockchain is a tenant. The landlord — the chain's governance — can change the rules. Gas costs can increase. Opcodes can be disabled. The agent has no recourse. An agent running on its own chain is sovereign. It controls its own rules. Its governance is its own. No external party can change its execution environment without its consent. Sovereignty is the property that converts an agent from a tenant to an owner. Ownership matters when the agent controls assets.

Predictable execution. Shared blockchains have congestion. When a popular NFT mint clogs the network, every application on the chain pays higher gas fees. The agent's time-critical transaction — an arbitrage, a liquidation, a risk adjustment — is delayed. On a sovereign chain, the agent has dedicated blockspace. No other application competes for it. The agent's transactions execute predictably, at known cost, with known latency. Predictability is essential for algorithmic systems. Shared infrastructure cannot guarantee it. Sovereign infrastructure can.

Protocol-level automation. The Cosmos SDK provides block lifecycle hooks — BeginBlocker and EndBlocker — that execute deterministically at the start and end of every block. An agent can embed logic that runs every block, without an external keeper bot, without a cron service, without relying on a centralized operator to trigger it. The logic is in the protocol. The protocol runs automatically. The automation is trustless — anyone can verify that the agent's rules are being followed because the rules are in the chain's source code.

Native interoperability. An agent on its own chain still needs to interact with other chains — to trade assets, to query data, to coordinate with other agents. IBC provides this natively. The agent's chain can send tokens to any other IBC-enabled chain. It can query the state of other chains through Interchain Queries. It can control accounts on other chains through Interchain Accounts. The interoperability is built into the stack. The agent doesn't need to deploy bridge contracts or trust external relayers. IBC is the bridge. The bridge is part of the protocol.

Custom execution environments. The Cosmos SDK does not force the agent to use a specific VM. The agent's logic runs as native Go code in the chain's binary. If the agent needs a smart contract VM — for user-submitted strategies, for composable DeFi primitives — the Cosmos EVM module adds Ethereum compatibility. If the agent doesn't need a VM, it can disable smart contracts entirely, eliminating an entire class of attack surface. The execution environment is a choice. The choice is the agent's.

Governance as code. The agent's rules — its risk limits, its allowed strategies, its upgrade process — can be encoded in the chain's governance module. Changes to the rules require a governance proposal. The proposal is voted on by the agent's stakeholders. The vote is recorded on-chain. The change, if approved, executes automatically. The governance is transparent, auditable, and enforced by the chain. The agent doesn't need to trust a human operator to follow the rules. The chain enforces them.

The Colombian CBDC: sovereignty in practice

In May 2025, Colombia announced it was building a CBDC proof-of-concept on the Cosmos stack. The design is revealing. Colombia does not use the public Cosmos Hub. It does not interoperate with public DeFi. It uses the Cosmos SDK, CometBFT, and IBC as a modular toolkit — assembling exactly the components it needs, adding custom KYC modules, disabling smart contract VMs, and implementing chain-level packet inspection for controlled interoperability. The result is a sovereign blockchain that borrows infrastructure from the public Cosmos ecosystem without depending on it.

This is the pattern for AI agents. The agent doesn't need the public Cosmos Hub. It doesn't need to be part of the public interchain. It needs the toolchain — the SDK for building its logic, CometBFT for running consensus, IBC for connecting to the chains it needs to interact with. It assembles what it needs. It omits what it doesn't. The result is a purpose-built chain for a purpose-built agent. The agent is the chain. The chain is the agent.

The multi-agent future

A single agent on a single chain is the starting point. The endpoint is a network of agents, each on its own chain, communicating through IBC, coordinating through shared protocols, governed by their own rules. An arbitrage agent on its own chain. A market-making agent on its own chain. A risk management agent on its own chain. A treasury agent on the Cosmos Hub, managing the aggregate portfolio. The agents form a mesh. The mesh is the system. The system is sovereign at every node.

This architecture mirrors the Unix philosophy: each agent does one thing well, communicates through a uniform interface, and composes with other agents. The IBC protocol is the pipe. The chain is the program. The interchain is the shell. The philosophy that built the internet's server architecture is the philosophy that will build the agent architecture. The substrate is Cosmos.


References:

Engineering is the discipline of building things that work within constraints. Every topic on this blog — operating systems, AI models, trading infrastructure, research labs, innovation economics — is examined through the lens of systems design. The lens is engineering. The method is: understand the constraints, design within them, verify the design works, iterate. The domain provides the specifics. The method is universal.

An AI agent that runs on a shared blockchain is a tenant. An agent that runs on its own chain is sovereign. The difference is not technical. It is architectural. Sovereignty means nobody can change your rules without your consent.

In a single encounter, confrontation is logical

Robert Aumann won the Nobel Prize for proving that cooperation is rational in repeated games. 'In a single encounter, confrontation is the logical move; but when the interaction will occur repeatedly, cooperation is the logical behavior.' This is why teams that stay together build trust. The trust is mathematics.

aumanngame-theorycooperationrepeated-gamesteams

Robert Aumann shared the 2005 Nobel Prize in Economics with Thomas Schelling. His contribution was the mathematics of repeated games. The insight is compressed to a sentence:

"In a single encounter, confrontation is the logical move; but when the interaction will occur repeatedly, cooperation is the logical behavior."

In a single encounter — a one-shot game — the rational strategy is to defect. There is no future in which the other player can punish you. There is no shadow of tomorrow to discipline today. You take what you can get. They would do the same. Both of you know this. Both defect. Both get the suboptimal outcome. The Prisoner's Dilemma is a one-shot game. The dilemma is real.

In a repeated encounter — an iterated game with no known end — the calculus changes. Defection today costs you cooperation tomorrow. The other player will remember. They will punish you in future rounds. The future rounds matter because the game continues indefinitely. The shadow of the future disciplines the present. Cooperation becomes rational. Not because the players are virtuous. Because the mathematics changed. The payoff structure changed. The change is in the repetition. The repetition is the mechanism.

Teams as repeated games

Software teams are repeated games. The same people work together sprint after sprint, quarter after quarter, year after year. The game has no predetermined end. The shadow of the future is long. This is why teams that stay together develop trust. The trust is not a personality trait. It is an equilibrium. Each person has learned that cooperation produces better long-run outcomes than defection. Each person expects others to cooperate because others have learned the same thing. The expectation is rational. The cooperation is stable.

Teams that churn cannot sustain this equilibrium. When people leave and join frequently, the effective horizon shortens. The new person doesn't have a history of cooperation with the existing team. The existing team doesn't know if the new person will cooperate. The uncertainty reduces the expected value of cooperation. The shadow of the future is shorter because the future with this person is uncertain. The shorter shadow produces less cooperation. Less cooperation produces worse outcomes. The churn is costly in ways that appear on no balance sheet. The cost is real. The mathematics predicts it.

Aumann's insight is that cooperation doesn't require central enforcement. It doesn't require a manager mandating collaboration. It doesn't require HR programs or team-building exercises. It emerges from the structure of the interaction. Repeated interaction with no known end produces cooperation as an equilibrium. The structure is the mechanism. The mechanism produces the behavior. The behavior looks like culture. It is mathematics.

API contracts as repeated games

The same logic applies to inter-service communication. Service A depends on Service B's API. If the interaction is one-shot — A calls B once and never again — B has no incentive to maintain a stable API. B can change the API whenever it wants. A's dependence is A's problem. The interaction is one-shot. Defection is rational.

If A and B will interact repeatedly — A will call B's API every day for years — the calculus changes. B knows that breaking the API today costs B in the future. A will be angry. A will escalate. A might build their own version of B. The future cost of breaking the API exceeds the present benefit. B maintains the API. The stability is not because B is considerate. It is because the game is repeated. The repetition changes the payoff.

This is why internal APIs between teams that have worked together for years are more stable than external APIs consumed by strangers. The internal teams are in a repeated game. The external consumers are in a one-shot game from the provider's perspective. The provider doesn't feel the future cost of breaking the API because the future cost is diffused across thousands of anonymous consumers. The consumers can't coordinate to punish the provider. The coordination problem prevents the repeated-game equilibrium from forming. The provider defects — changes the API, deprecates the endpoint, raises the price. The consumers suffer individually. The suffering is aggregate but uncoordinated. The provider doesn't feel it. The one-shot structure produces the defection. The structure is the problem.

Mechanism design for repeated games

If the natural structure produces one-shot interactions where repeated interactions would produce better outcomes, change the structure. This is mechanism design. Automated contract testing changes API interactions from one-shot to repeated. Every build runs the contract tests. Every breaking change is immediately visible. The visibility creates a repeated-game payoff structure. The provider can't defect invisibly. The defection is detected. The detection has a cost — the build breaks, the provider must fix it. The cost is immediate. The immediacy simulates repetition. The simulation changes the behavior.

SLAs with penalty clauses do the same. The penalty is the future cost of defection, brought forward to the present. The provider who breaks the SLA pays now. The payment is the shadow of the future, compressed into a contract. The contract is a mechanism for making one-shot interactions behave like repeated ones. The mechanism substitutes for the missing future.

Code review is a repeated game. The author and the reviewer will interact again. The author who ignores feedback today will receive less helpful feedback tomorrow. The reviewer who is needlessly harsh today will find their reviews ignored tomorrow. The mutual expectation of future interaction disciplines present behavior. The discipline is automatic. It doesn't require rules. It requires continuity.

The half-life of trust

Aumann's mathematics implies that trust has a half-life. It decays when the future becomes uncertain. A reorg that shuffles teams resets the repeated-game equilibrium. The new teams have no history of cooperation. They must rebuild it. The rebuilding takes time. During the rebuilding, cooperation is suboptimal. The system performs worse. The reorg's cost includes the lost cooperation during the rebuilding period. Nobody accounts for this cost. The cost is real.

A layoff that cuts a team in half shortens the shadow of the future for everyone who remains. The remaining people now know the game can end unexpectedly. The unexpected end converts an infinite-horizon game into an uncertain-horizon game. Uncertain horizons produce less cooperation than infinite horizons. The layoff's cost includes the reduced cooperation among survivors. Nobody accounts for this cost. The cost is real.

A team that knows it will be disbanded in six months is in a finite-horizon game. Finite-horizon games unravel from the end. In the final sprint, defection is rational — there is no future to punish it. In the second-to-last sprint, defection is rational because defection in the final sprint is already expected. The logic propagates backward. By induction, cooperation collapses in the first sprint. The collapse is mathematical. The team's morale didn't fail. The structure changed. The structure produced the outcome.

The Hayekian manager understands this. They preserve team continuity not because "culture matters" but because continuity is the structural precondition for cooperation. The structure produces the behavior. Change the structure. The behavior changes. The change is predictable. Aumann gave us the mathematics. The mathematics is clear. Most organizations ignore it.


References:

  • Robert Aumann, "Acceptance Speech," Nobel Prize in Economics, 2005.
  • Robert Aumann and Michael Maschler, Repeated Games with Incomplete Information, MIT Press, 1995.
  • Anatol Rapoport and Albert Chammah, Prisoner's Dilemma, University of Michigan Press, 1965.
  • Related posts: Scarcity and Games, On Scarcity, Design the Game

Engineering is the discipline of building things that work within constraints. Every topic on this blog — operating systems, AI models, trading infrastructure, research labs, innovation economics — is examined through the lens of systems design. The lens is engineering. The method is: understand the constraints, design within them, verify the design works, iterate. The domain provides the specifics. The method is universal.

Cooperation is not virtue. It is mathematics. The shadow of the future disciplines the present. When the game is repeated, defection is punished tomorrow. The punishment is the mechanism. The mechanism produces trust.

Colibri

Colibri is a pure-C inference engine that runs a 744-billion-parameter Mixture-of-Experts model on 25 GB of consumer RAM. Zero dependencies. No GPU required. One C file. It is the most impressive piece of systems engineering I have seen this year.

colibriinferencesystemscmixture-of-experts

Colibri is a pure-C inference engine. It runs GLM-5.2 — a 744-billion-parameter Mixture-of-Experts language model — on a consumer machine with roughly 25 GB of RAM. It has zero dependencies. No BLAS. No Python at runtime. No GPU required, though an optional CUDA backend exists. The engine is a single C file: c/glm.c, approximately 2,400 lines. The project is a one-person effort, written and tested entirely on a 12-core laptop.

The name means hummingbird. Tiny creature. Enormous workload.

The model it runs is GLM-5.2, released by Z.ai under MIT license. 744 billion parameters total. 75 Mixture-of-Experts layers, each with 256 experts. Only about 40 billion parameters activate per token. The challenge: 21,504 routed experts totaling roughly 370 GB at int4 precision. They don't fit in RAM. They barely fit on disk. Colibri makes them work anyway.

How it fits

The insight is separating what must stay resident from what can be streamed. Dense components — attention layers, shared experts, embeddings — total about 17 billion parameters at int4, occupying roughly 9.9 GB of RAM. They stay resident. Routed experts — all 21,504 of them, about 19 MB each at int4 — sit on disk at 370 GB total. They are streamed on demand with a per-layer LRU cache. The operating system's page cache acts as a free second-level cache. An expert that was recently read stays in RAM because the OS hasn't evicted it yet. The OS doesn't know it's caching model weights. It doesn't need to.

The cache is adaptive. At startup, Colibri reads MemAvailable and auto-sizes the expert cache to fit within available RAM, projecting the full working set — KV cache, MTP row, reconstruction buffers — so the OOM killer never fires. The cache learns. The engine records expert usage patterns to .coli_usage. At startup, it pre-pins the hottest experts into spare RAM. A live tier adaptation mode swaps cold experts for hot ones at turn boundaries. The cache gets smarter the longer it runs.

The I/O is async. Expert readahead uses WILLNEED — the kernel reads the next expert block while the CPU computes the current one. An experimental router-lookahead prefetch predicts the next layer's routing from the current layer's post-attention state with 71.6% recall, issuing readahead from a dedicated I/O thread. The disk and the CPU work in parallel. The parallelism is the performance.

The attention

GLM-5.2 uses Multi-head Latent Attention (MLA) with compressed key-value cache. Standard attention stores K and V for every head, for every token. At 64 heads, that's a lot of memory. MLA compresses the KV cache to 576 floats per token instead of 32,768 — a 57× reduction. The compression is lossless relative to full attention. Colibri validates token-exact against the reference implementation.

MLA weight absorption, borrowed from DeepSeek, eliminates per-token key/value reconstruction during decode. The query absorbs the kv_b projection. The reconstruction is skipped. The speedup is significant.

GLM-5.2 also uses DSA sparse attention — a "lightning indexer" that selects a top-2048 causal key set per layer. Colibri auto-extracts the selection weights from the model files. Dense attention is quadratic in sequence length. Sparse attention is linear in the selected set size. The sparsity is the speed.

KV-cache persistence: conversations reopen warm across engine restarts via .coli_kv files. The files are approximately 182 KB per token. They are crash-safe. No re-prefill on restart.

Speculative decoding

Speculative decoding uses a draft model to predict the next several tokens, then verifies them against the full model. If the draft is correct, you get multiple tokens per forward pass. If it's wrong, you recompute. The technique is standard. The implementation is not.

Colibri uses GLM-5.2's own MTP (Multi-Token Prediction) head — layer 78 — as the draft model. The head must be int8, not int4. At int4, acceptance collapses below 4%. At int8, acceptance reaches 39-59%, yielding 2.2-2.8 tokens per forward pass. The draft is lossless under sampling via rejection sampling. No separate draft model. No extra memory for draft weights beyond what the inference engine already loads. The MTP head is a few hundred megabytes. The return is a 2-3× speedup.

Grammar-forced speculative drafts handle constrained outputs — JSON, function calling. When the grammar admits exactly one legal byte, the grammar itself injects a pre-accepted draft. No draft head needed. A wrong grammar cannot change the output. Worst case: rejected drafts, no speedup, no harm.

The quantization

Colibri uses integer-dot kernels. Int8 uses AVX2 maddubs — multiply-add unsigned bytes — achieving approximately 119 GFLOP/s. Int4 uses packed representations with per-row scales and dequant-on-use. The routing between int8, int4, and float32 is decided per shape by measurement. The engine profiles each matrix shape at startup and selects the fastest kernel.

The offline converter takes the FP8 checkpoint — 756 GB — and converts it to int4. It downloads one shard at a time, approximately 5 GB, converts it, and deletes it. The full 756 GB never exists on disk at once. The converter is resumable.

The integer-dot approach was chosen over BLAS for two reasons. First, zero dependencies. Second, expert shapes are small — 19 MB each — and BLAS overhead dominates for small matrices. The custom kernels are faster than calling into OpenBLAS for the shapes that matter. The trade-off was measured. The measurement drove the decision.

The performance

On the developer's laptop — WSL2, 12 cores, 25 GB RAM, NVMe via VHDX at approximately 1 GB/s random reads — Colibri achieves roughly 0.05-0.1 tokens per second cold, improving as the cache warms. Community benchmarks show what happens with more hardware:

Hardware Throughput Notes
Apple M5 Max, 128 GB, Metal backend 1.83 tok/s 66% expert hit rate
Ryzen AI 9 HX 370, 128 GB 0.37 tok/s MTP acceptance 52%
Ryzen 9 9950X, PCIe 5.0 NVMe 0.28 tok/s Bottleneck flipped to matmul
Ryzen AI Max+ 395, 128 GB, Optane 0.40 tok/s 71% hit rate

The project estimates that with enough RAM to fully cache hot experts plus AVX-512/VNNI kernels, 5-15 tokens per second is achievable — interactive speeds for a 744-billion-parameter model. The bottleneck moves from disk to compute as RAM increases. The movement is the scaling path.

What it includes

The engine is a single C file. The CLI provides chat, serve, plan, doctor, convert, and bench subcommands. The HTTP API is OpenAI-compatible — /v1/chat/completions, /v1/models, SSE streaming, usage counts — with a bounded FIFO admission queue and multi-slot KV contexts. The Web UI is a React/TypeScript client, approximately 390 lines. An optional CUDA backend handles resident tensors and hot-expert VRAM tier with multi-GPU support. Windows 11 native build via MinGW-w64 with POSIX-to-Win32 shims. A quality benchmark harness — MMLU, HellaSwag, ARC — exists but is untested at scale.

The project is Apache 2.0. The model weights are MIT. The entire thing was written by one person on a laptop.

Why this matters

Colibri matters for the same reason the Unix philosophy matters. One tool, doing one thing well. Zero dependencies. The entire inference engine for a 744B model fits in a single C file shorter than this blog post. The complexity is in the model architecture, the quantization scheme, the cache policy, the I/O strategy. The code is not simple. The system is. The system has conceptual integrity — it feels like one mind designed it, because one mind did.

It matters because it proves that frontier models don't require frontier hardware. A 744B model runs on a laptop. Slowly, but it runs. The gap between what the cloud provides and what a consumer machine can do is a function of engineering, not physics. The physics says the model must be read from disk. The engineering says: stream it, cache it, prefetch it, quantize it, compress it. The engineering works.

It matters because it is a one-person project. No team. No funding. No infrastructure beyond a laptop. The output is a competitive inference engine for a model that, six months ago, required a datacenter. The democratization of inference is not a policy objective. It is an engineering achievement. The achievement is public. The code is open. The weights are MIT. Anyone with a laptop and an NVMe drive can run a 744B model. The democratization is complete.


References:

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

A 744-billion-parameter model running on 25 GB of RAM is not a curiosity. It is a proof. The proof is that the gap between cloud and consumer is a function of engineering, not physics. Engineering can close the gap.

Chinese models will win the local-first race

US export controls forced Chinese AI labs to optimize for efficiency. The result is models that run on consumer hardware while American labs build ever-larger cloud-only systems. Local-first is the next battleground. China has the structural advantage.

aichinalocal-firstdeepseekqwenopen-sourceeconomics

In October 2022, the US banned Nvidia from selling its most advanced AI chips to China. The intent was to slow China's AI progress by denying it compute. The effect was the opposite: it forced Chinese labs to become the most efficient model builders in the world. America built bigger. China built smarter. The local-first race is now between American models that require datacenters and Chinese models that run on your laptop. The laptop will win.

This is not a prediction about which country will build the most powerful model. America still holds the frontier — GPT-5, Claude Opus 4.6, Gemini Ultra. The largest models, trained on the largest clusters, achieving the highest benchmark scores, are American. The argument is about a different race: who will own the models that run on devices. Phones. Laptops. Edge servers. The models that work offline, cost nothing per query, and keep data local. That race is determined by efficiency, not scale. And efficiency is what the export controls forced China to master.

The sanctions that backfired

The US export controls escalated through multiple rounds. October 2022: ban on A100 and H100 chips. October 2023: the H800 — Nvidia's bandwidth-crippled H100 designed specifically to comply with the earlier ban — was itself banned. December 2024: high-bandwidth memory restricted. Each round tightened the screws. Each round was met with Chinese efficiency gains that erased the intended advantage.

DeepSeek trained its V3 model on a cluster of H800 GPUs — chips deliberately hobbled by export controls to have reduced inter-chip bandwidth. The H800 was supposed to be too slow for frontier training. DeepSeek programmed 20 of the 132 processing units on each H800 specifically for cross-chip communication, working below Nvidia's CUDA layer to overcome the bandwidth limits. The result was a model that matched GPT-4o on most benchmarks at roughly 1/30th the training cost. The export controls didn't prevent the model. They provoked the optimization that made it possible.

"Forced to operate under a far more constrained computing environment, AI engineers in China are innovating in ways that their computing-rich American counterparts are not." — Brookings Institution, 2025

DeepSeek's CEO stated the dynamic directly: "Money has never been the problem; bans on advanced chips are the problem." The problem produced the solution. The constraint produced the efficiency. The efficiency produced models that can run on consumer hardware — not because that was the goal, but because the optimization path that the sanctions forced converged on it. Models that fit in 16GB of RAM. Models that run at interactive speeds on a MacBook Air. Models that can be downloaded, run locally, and never phone home. American labs, with essentially unlimited compute, never had to optimize for this. Chinese labs had no choice.

By April 2026, DeepSeek V4 was trained entirely on Huawei Ascend chips — zero Nvidia dependency. The Flash variant runs at 17–31 tokens per second on a Mac Studio. Qwen3.5-4B runs at 147 tokens per second on a MacBook Air using 2.4GB of RAM. These are not research curiosities. These are production-ready models that run on hardware you already own, at speeds that feel instant, with no API key, no rate limit, no privacy policy, no vendor that can revoke your access or raise your price.

The American bet: scale

American labs — OpenAI, Anthropic, Google — are built on a different bet. The bet is that intelligence scales with compute, that the largest models will be the most capable, and that capability at the frontier justifies the infrastructure cost. This bet has produced extraordinary results. GPT-5.4, Claude Opus 4.6, Gemini 2.5 Ultra are remarkable. They solve problems that smaller models cannot. They are the best at what they do.

But they cannot run on your device. They cannot run offline. They cost money per query. They require an internet connection. They require you to send your data to a server owned by a company whose incentives are not aligned with yours. They can revoke your access, raise your price, change their terms, or go out of business. The model is theirs. You rent access.

This bet also makes American labs structurally disinterested in efficiency. When you have effectively unlimited compute, you optimize for capability, not efficiency. You train larger models on larger clusters because that's what your infrastructure, your talent, and your economics are built for. You don't spend your best researchers' time squeezing a 7B model to run on a phone when they could be training a 1.6T model on a 100,000-GPU cluster. The incentives point toward scale. The economics point toward scale. The culture points toward scale.

The result is that American frontier models are better, and American local models are worse, than they would be if the incentives were reversed. The gap at the top is large. The gap at the bottom — the models that can run on consumer devices — is also large, but in the opposite direction. Chinese labs own the bottom. The bottom is where volume lives.

The Chinese bet: efficiency

Chinese labs optimize for efficiency not by choice but by necessity. The export controls cut off access to the largest training clusters. The response was systematic:

  • Mixture of Experts architectures: DeepSeek V3 and V4 use MoE with hundreds of experts but only activate 37–158 billion parameters per token. The model is 1.6T parameters. The inference cost is for 158B. The capability is from 1.6T. The efficiency is from 158B.

  • Low-level GPU optimization: When the H800's inter-chip bandwidth was artificially limited, DeepSeek programmed around the limit at the hardware level. When Nvidia chips became unavailable, they ported to Huawei Ascend. The model is hardware-agnostic because it had to be.

  • Aggressive quantization: Chinese models ship with 4-bit, 2-bit, and mixed-precision variants optimized for consumer hardware. Qwen3.5-4B at 4-bit uses 2.4GB of RAM and runs at 147 tokens per second. A full GPT-4 class model in 2019 required a datacenter. A capable reasoning model in 2026 requires a MacBook Air.

  • Distillation as a first-class technique: Large models train small models. DeepSeek R1 distilled its reasoning capability into 7B and 14B variants that retain most of the capability at a fraction of the size. OpenAI accused DeepSeek of distilling from ChatGPT outputs — an accusation later walked back. The technique is legal. The results are effective. The small models run locally. The large models run in datacenters. The capability leaks downward.

  • Open-weight licensing: DeepSeek, Qwen, Yi, and most Chinese frontier labs release their models under permissive licenses — Apache 2.0, MIT, or custom open-weight terms. You can download the weights. You can run them locally. You can fine-tune them. You can build products on them. American models are API-gated. You can access them. You cannot own them.

The combination is powerful: efficient architectures, hardware-portable implementations, aggressive quantization, distillation to small sizes, and permissive licensing. Each factor compounds the others. The result is a Chinese open-weight model ecosystem that dominates the local-first deployment landscape. American labs have nothing comparable because their incentives never produced it.

The economics of local-first

The economics favor local models for a large and growing fraction of use cases. The cost structure is different in kind, not just in degree.

A cloud API charges per token. Heavy usage gets expensive fast. A local model costs the hardware once, then zero per token forever. The crossover point — the usage volume at which buying hardware is cheaper than paying per token — keeps moving lower. As models get more efficient, cheaper hardware can run capable models. As hardware gets faster, the same model runs at higher throughput. Both trends favor local.

Privacy is economic. Sending data to a cloud API means trusting the provider's privacy policy, security practices, and government access policies. For medical data, legal data, financial data, personal correspondence, internal company documents — the risk of a breach or a policy change is a real cost. Local models eliminate it. The data stays on the device. There is no provider to trust, breach, subpoena, or change.

Availability is economic. Cloud APIs go down. Rate limits apply. Accounts are suspended. Pricing changes. Terms of service shift. A local model works when the internet is down, when the API is overloaded, when the provider has disabled your account, when the provider has gone out of business. The model is a file. The file is yours. It works as long as the hardware works.

The market for local-first AI is not the market for frontier intelligence. It is the market for everyday intelligence — summarization, coding assistance, document analysis, email drafting, translation, data extraction. These tasks do not require a trillion-parameter model. They require a model that is fast, private, available, and free to use. That is the Chinese open-weight model ecosystem. That is what the export controls inadvertently created.

The cultural consequences

The local-first race is not only about efficiency and economics. It is about who controls the models that run on the world's devices.

If the models running locally are predominantly Chinese open-weight models, then the default AI experience for hundreds of millions of users will be shaped by models trained in China, on Chinese data, reflecting Chinese assumptions about what an AI should say and not say. The biases will be Chinese biases. The safety filters will be Chinese safety filters. The alignment will be Chinese alignment. American models will be available as cloud APIs — better at the frontier, more expensive, less private, less available. The everyday AI experience will be Chinese not because users chose Chinese models but because Chinese models were the ones that ran on their devices.

If the models running locally are predominantly American, the dynamic reverses. But American models don't run locally. They are not designed to. They are not licensed to. They are not optimized to. The American AI industry has bet on the cloud. The cloud bet may win the frontier. It will lose the local-first market because it is not competing in it.

The open-weight licensing difference is structural. Chinese labs release weights. You can inspect them, modify them, fine-tune them, deploy them. American labs release APIs. You can call them. The Chinese approach builds an ecosystem — tooling, quantization methods, inference engines, fine-tuning datasets, deployment guides — around open weights. The American approach builds an ecosystem around API integration. The open-weight ecosystem produces local-first capability as a byproduct. The API ecosystem produces nothing local. The ecosystem divergence is self-reinforcing. More developers build tools for open weights. More tools make open weights more capable. More capability attracts more developers. The flywheel spins.

What happens next

The export controls will not be reversed in any meaningful way. The US political consensus on containing China's AI capability is bipartisan and durable. The controls may tighten further. If they do, Chinese labs will become more efficient still — because they will have to.

American labs will continue to lead the frontier. The largest, most capable models will be American for the foreseeable future. The gap at the top may even widen as American labs deploy ever-larger training clusters.

But the local-first race is not about the frontier. It is about ubiquity. Models that run everywhere, on everything, for free, with privacy, without permission. That race is determined by efficiency, openness, and ecosystem. On all three dimensions, Chinese labs have the structural advantage — an advantage created, perversely, by American policy.

The export controls were intended to contain Chinese AI. They contained Chinese access to American chips. They accelerated Chinese innovation in everything else. The local-first future was an unintended consequence. It is now the likely outcome. The world will run American models in the cloud and Chinese models on devices. The cloud is where the capability is. The devices are where the people are. The people are more numerous than the datacenters. That is the math. The math is unfavorable to the American bet.


References:

  • Brookings Institution, "DeepSeek shows the limits of US export controls on AI chips," January 2025.
  • Epoch AI, "What did US export controls mean for China's AI capabilities?" December 2024.
  • Andrew L., "Chinese AI Models 2026: The Agentic Revolution, Hardware Independence," dev.to, 2026.
  • David Lin, testimony to US Congressional hearing, 2025.
  • DeepSeek technical reports, V3 (2025), R1 (2025), V4 (2026).
  • Qwen technical reports, Qwen2.5 (2025), Qwen3.5 (2026).
  • Related posts: Disruptive Innovation, I, Pencil, The electric light was not a better candle

Engineering is the discipline of building things that work within constraints. Every topic on this blog — operating systems, AI models, trading infrastructure, research labs, innovation economics — is examined through the lens of systems design. The lens is engineering. The method is: understand the constraints, design within them, verify the design works, iterate. The domain provides the specifics. The method is universal.

Export controls did not stop Chinese AI. They redirected it. The controls made large training clusters unavailable. Chinese labs optimized for efficiency. The optimization produced models that run on laptops. The laptops are everywhere.

BSD is clean, OpenBSD is cleaner

BSD is an operating system designed as a whole. Linux is a kernel assembled into an operating system by distributions. The difference in design quality follows directly from the difference in architecture. OpenBSD takes the logic to its extreme: correctness over features, every time.

bsdopenbsdlinuxunixoperating-systemssoftware-design

In 1991, a Finnish student wanted a Unix-like system for his 386 PC. BSD Net/2 had been released, but he didn't know about it. He started writing his own kernel. He called it Linux. The rest is history — but the wrong history, or at least an incomplete one.

Linus Torvalds later said that if a working 386BSD had been available, he would never have created Linux. He was not making a philosophical choice between cathedral and bazaar. He was solving a practical problem with the tools he knew existed. The lawsuit that froze BSD development from 1992 to 1994 — AT&T's USL vs. BSDi, alleging that BSD contained proprietary Unix code — gave Linux a two-year window with no competition. By the time the suit settled and 4.4BSD-Lite shipped clean, Linux had momentum it would never lose.

The consequence is that the world runs on the bazaar. The cathedral — cleaner, more coherent, better designed — is a niche. The niche is worth understanding because it represents a different theory of what an operating system should be. OpenBSD is the purest expression of that theory.

What BSD is

BSD is not a kernel with some utilities. It is an operating system. One source tree. One development group. One coherent design. The kernel, the C library, the core utilities (ls, cp, grep, the shell), the daemons, the manual pages — all maintained by the same people, in the same repository, under the same quality standards.

This is the model inherited from Research Unix at Bell Labs. Ken Thompson and Dennis Ritchie didn't ship a kernel and let someone else figure out the userland. They shipped a system. The system had conceptual integrity because one small group controlled all of it. BSD preserved this model. Linux abandoned it — not by choice, but by circumstance. When Linus started, the GNU project had already written the C library, the coreutils, and the shell. He only needed to write the kernel. The assembly model was the path of least resistance.

The difference is visible at every level. BSD configuration files have consistent syntax because the same people who wrote the daemons wrote the configuration parsers. BSD manual pages are complete and accurate because they are maintained as part of the source tree, not as an afterthought by a separate documentation project. BSD systems boot predictably because the init system, the device drivers, and the service scripts were designed together, not integrated after the fact by a distribution maintainer.

"Practically the entire BSD distribution is written and packaged by the distribution maintainers. The FreeBSD people know about (and have written) everything that's part of the FreeBSD distribution."

This is not a marketing claim. It is a statement about the structure of the source repository. Every component is designed, reviewed, and tested as part of one system. When a kernel interface changes, the userland tools that depend on it are updated in the same commit. When a security vulnerability is found in a library, every program that uses that library is audited — because they are all in the same tree and maintained by the same people. The coherence is not accidental. It is the result of organizational structure. It is Conway's Law, applied to operating systems: the design mirrors the communication structure of the team that builds it. BSD's team is one team with one tree. The design has one voice.

What Linux is

Linux is a kernel. Linus Torvalds chose to focus only on the kernel and to not ship user-level programs. Distributions — Red Hat, Debian, Ubuntu, Arch, hundreds of others — assemble an operating system from the Linux kernel, the GNU C library, GNU coreutils, a shell, a desktop environment, an init system, and hundreds of other projects. Each project has its own maintainers, its own priorities, its own release schedule, its own coding style, its own documentation format. The distribution's job is to make them work together and to resolve the conflicts when they don't.

This model has been extraordinarily successful at generating variety and breadth. There are Linux distributions for every purpose, every hardware platform, every ideological preference. The Linux kernel supports more hardware than any other operating system in history. The driver ecosystem alone is a miracle of coordination — thousands of contributors from hundreds of companies, merging code into a single tree at a rate of 10,000+ commits per release cycle.

But the model has no mechanism for internal coherence. There is no single group that understands the entire system. There is no consistent design language across components. A Linux system is a negotiated settlement between independently developed projects that happen to run on the same kernel. The negotiation is managed by distribution maintainers who did not write any of the components and whose primary job is integration, not design.

"Linux has no similar concept [of a base system]; the kernel is maintained and distributed by one group, the usual runtime library by another, and so on. Linux distributions have the job of assembling all of the bits."

This is Brooks's committee design applied to operating systems. Each component is individually impressive. The assembly lacks conceptual integrity because no single mind controlled the interfaces. The interfaces were negotiated between projects with different goals, different timelines, and different ideas about what "good" means. The result works. It is not clean. It is not coherent. It is a patchwork quilt. Everyone who has debugged a Linux system at 3am knows the feeling of crossing a component boundary and discovering that the assumptions changed.

The lawsuit that chose the bazaar

The USL vs. BSDi lawsuit (1992-1994) is the pivot on which operating system history turned. AT&T's Unix Systems Laboratories sued Berkeley Software Design, Inc., alleging that the BSD Net/2 release still contained proprietary AT&T code. The suit froze BSD development for nearly two years. Developers couldn't contribute. Users couldn't trust the codebase. Companies couldn't build products on it. The uncertainty was total.

During those two years, Linux — which had no AT&T code, was written from scratch, and faced no legal threat — absorbed the energy that would have gone to BSD. Developers who would have contributed to the BSD kernel contributed to Linux instead. Companies that would have built BSD distributions built Linux distributions instead. The network effects tipped. By 1994, when the lawsuit settled and 4.4BSD-Lite shipped with all AT&T code removed, Linux had won the mindshare. It has never lost it.

Linus himself acknowledged the contingency:

"If 386BSD had been available when I started on Linux, Linux would probably never have happened."

The world's dominant server operating system was not chosen on technical merit. It was chosen because a lawsuit froze the technically superior alternative during the critical window when network effects were forming. This is not a criticism of Linux. It is a fact about history. The cathedral didn't lose because it was worse. It lost because it was sued. The bazaar didn't win because it was better designed. It won because it was available. Availability beats design quality in the short run. The short run became the long run.

OpenBSD: the extreme of the philosophy

If BSD represents the cathedral model, OpenBSD represents the cathedral with the strictest building code. Theo de Raadt forked OpenBSD from NetBSD in 1995. The project's goals are explicit and uncompromising: correctness over features, security through design, and code quality as the primary metric.

"We are non-stop trying to find ways across our entire source tree that small little programmer errors result in problems. At some point, we have to start asking ourselves whether features are the thing, or whether quality is the issue. I really think we have to focus on the quality before the features." — Theo de Raadt

This is not a slogan. It is enforced by process. The six-month release cycle — May and November, every year, twenty-five consecutive on-time releases with no critical bugs — structures all development.

The cycle has phases. Four months of development: features are written, code lands, the tree is open. One month of API lockdown: interfaces freeze, testing intensifies, bugs are fixed. Final weeks of code freeze: only the simplest edits — documentation, minor fixes — are accepted. The tree must build and boot on every supported architecture before release. If a feature isn't ready, it waits. The next release is never more than six months away.

"Pretty soon, you get a clue, and start working with the release. Also, when you have a guarantee that you WILL have a six months release schedule, trying to wedge that last improvement in loses some of its attraction. You know that you will be able to do it for the next release, which is only six months away..." — Marc Espie, OpenBSD developer

The six-month cadence is a forcing function. Features that aren't ready are deferred. Features that are deferred are refined. Features that are refined land clean. The discipline produces a system where every component has been through the freeze cycle multiple times, where every interface has been tested on every architecture, where the manual pages match the code because both were updated before the freeze. The result is not just a secure operating system. It is a well-engineered one.

Security as design, not feature

OpenBSD's approach to security is the clearest expression of its design philosophy. Most operating systems treat security as a feature to be added: firewalls, anti-virus, intrusion detection, patches for vulnerabilities as they are discovered. OpenBSD treats security as a property of the design itself. Eliminate entire classes of vulnerability. Make the correct thing the only possible thing. If the API can be misused, change the API.

The innovations list is long because the approach is systematic:

  • W^X: Memory is either writable or executable. Never both. This eliminates the mechanism that most exploits rely on. Implemented in 2003. Now standard everywhere. OpenBSD did it first.

  • pledge(2): A process declares at startup what system calls it will use. After pledge(), any other system call kills the process. A file server that pledges only stdio and sendmsg cannot open new files, cannot fork, cannot exec. If an attacker compromises it, they gain a process that can do almost nothing. This is not a mitigation. It is a design constraint enforced by the kernel.

  • unveil(2): A process declares which parts of the filesystem it can access. After unveil(), the rest of the filesystem does not exist as far as that process is concerned. A web browser that unveils only ~/Downloads cannot read ~/.ssh. If the browser is compromised, the attacker can't either.

  • Secure malloc: The memory allocator randomizes allocations, guards pages, and detects use-after-free. These are not optional hardening flags. They are the default allocator.

  • pf(4): The packet filter, written by OpenBSD, with a clean syntax, default-deny semantics, and integration with the rest of the system's security model. Not bolted on. Part of the base system.

The project's security record is stated without marketing: "Only two remote holes in the default install, in a heck of a long time." The statement is verifiable. The record is public. The claim is modest. The achievement is extraordinary.

"The problem with security is that people learn what they're supposed to by example, learn they're supposed to use APIs in a certain way, and they're just wrong." — Theo de Raadt

The API is the problem. Fixing the API fixes the vulnerability class. Fixing individual instances of the vulnerability fixes only the instances. The OpenBSD approach is to find the API that produces the vulnerability, change the API so it cannot be misused, and then fix every caller to use the new API correctly. This is information hiding applied to security: hide the dangerous operation behind an interface that makes the dangerous operation impossible. The caller cannot do the wrong thing because the wrong thing is not exposed.

What Brooks and Parnas would see

Brooks would recognize BSD as a system with conceptual integrity. One small group controls the entire design. Every interface is reviewed by the same people who wrote the implementations that use it. The system speaks with one voice. The manual pages match the code. The configuration syntax is consistent. This is what conceptual integrity looks like at the scale of an operating system. It is achieved by the one-mind rule — one development group with authority over the entire tree.

He would recognize Linux as a system assembled from components designed by different groups with different visions. The kernel has one design philosophy. Systemd has another. The GNU tools have a third. GNOME has a fourth. Each component is internally coherent. The assembly is not. This is committee design at the scale of an operating system. It works. It is not clean. The seams between components are where the complexity lives. The seams are also where the bugs live.

Parnas would recognize OpenBSD's security architecture as information hiding applied to attack surfaces. The kernel hides the hardware. Libc hides the kernel. The daemons hide the services. Each layer exposes the minimum interface. The pledge and unveil mechanisms enforce that a compromised process cannot exceed its declared interface. The attacker gains access to a process. The process has no access to anything else. The information hiding is not advisory. It is enforced by the kernel. The volatile decision — what this process can do — is hidden behind a stable interface that the process itself cannot change.

Parnas would also recognize the release cycle as a form of modular development. The base system is a module with a stable interface (the release). The ports tree is a separate module with a different interface (the package). The two modules evolve at different speeds, by different processes, with different quality standards. The release boundary is the contract. What ships in the release must work together, on every architecture, with no known bugs. What is in ports is best-effort. The separation is clean. The hiding is real. The architecture absorbs the difference in quality requirements because the volatile part (ports) is isolated from the stable part (base).

The dirty and the clean

Linux is dirty in the way that a city is dirty. It works. It is full of life. It contains multitudes. The streets don't follow a plan because the city grew organically, each neighborhood added by different builders at different times with different ideas about what a street should be. The result functions. It is not elegant. Nobody would design it from scratch this way. But it is what exists, and it runs most of the internet, and the sheer variety and energy and pace of development are extraordinary.

BSD is clean in the way that a well-designed building is clean. The structure is visible. The materials are consistent. The wiring is labeled. The documentation matches the implementation. You can understand the whole thing by studying any part because the same design language is used throughout. It does fewer things than Linux. The things it does, it does correctly. The release ships on time, with no critical bugs. The manual pages are accurate. The security model is coherent. The system makes sense as a system.

OpenBSD is cleaner still — the building with the strictest code, inspected continuously, where every door has a lock and every lock has a key and every key opens exactly one door. The tradeoff is clear: less hardware support, fewer features, a ports tree that lags behind Linux package availability. The benefit is also clear: a system where correctness is not aspirational but enforced by process, where security is not a feature but a property of the design, where the release ships on schedule and you can trust it.

The world runs on Linux because the lawsuit froze BSD at the wrong moment and network effects did the rest. The cathedral lost. The bazaar won. But the cathedral is still standing. For work where correctness matters more than breadth — firewalls, routers, secure servers, any system facing the internet — the cathedral is the right choice. OpenBSD is the cathedral at its most uncompromising. The design principles are visible in every layer. The discipline is enforced by process. The result is an operating system that does less than Linux and does it correctly. That is not a limitation. That is the point.


References:

Engineering is the discipline of building things that work within constraints. Every topic on this blog — operating systems, AI models, trading infrastructure, research labs, innovation economics — is examined through the lens of systems design. The lens is engineering. The method is: understand the constraints, design within them, verify the design works, iterate. The domain provides the specifics. The method is universal.

Clean code is not an aesthetic preference. It is a property of systems designed as a whole. BSD is clean because one group designed the whole. Linux is not because a thousand groups designed the parts.

The freedom to innovate was a historical accident

Bell Labs produced the transistor, Unix, C, information theory, and nine Nobel Prizes. It did so because three historical accidents converged: a regulated monopoly, American economic supremacy after WWII, and the greatest migration of scientific talent in history. None of these are reproducible.

bell-labsinnovationhistoryeconomicsunixresearch

Bell Labs produced the transistor. The laser. The Unix operating system. The C programming language. Information theory. Communications satellites. The silicon solar cell. Cellular telephony. The charge-coupled device that made digital cameras possible. Statistical process control. Nine Nobel Prizes. Four Turing Awards. The modern world was built in a single building in Murray Hill, New Jersey, by a few thousand people working for a telephone company.

This was not a coincidence of genius. It was a coincidence of conditions. Three historical accidents converged to create the most productive research institution in history. None of them were designed. None of them are reproducible. Understanding why Bell Labs worked is understanding why nothing like it exists today — and what that means for the freedom to innovate.

The regulated monopoly: a tax on phone calls that funded basic research

AT&T was a government-guaranteed monopoly. The Communications Act of 1934 established the Federal Communications Commission and blessed AT&T's control of the American telephone system. In exchange, AT&T accepted regulation — rate-setting, service obligations, and a commitment to universal service. The bargain was explicit: you get a monopoly. We regulate your prices. You serve everyone.

The bargain had a side effect nobody planned for: it created the most stable R&D funding stream in history. Every American phone bill contained what amounted to a small tax that flowed to Bell Labs. The revenue was predictable across decades. It did not depend on quarterly earnings, competitive threats, or the stock market. It depended on Americans making phone calls. Americans made more phone calls every year. The funding grew. The research continued.

"That freedom was predicated on the steady stream of revenue provided by the monthly bills paid by telephone subscribers, which allowed Bell Labs to function much like a national laboratory." — Jon Gertner, The Idea Factory

Mervin Kelly, the Bell Labs director most responsible for its culture, described research as a "non-scheduled area of work." No deadlines. No objectives. No progress reports. Researchers could pursue their own investigations "sometimes without concrete goals, for years on end." Claude Shannon spent a decade developing information theory with no mandate to produce a commercial product. He was trying to understand communication itself. The monopoly paid him to think. The thinking produced the bit, the mathematical foundation of the digital age.

"We give much attention to the maintenance of an atmosphere of freedom and an environment stimulating to scholarship."

This was not a mission statement that a shareholder-owned company could write. It was a mission statement that a regulated monopoly, indifferent to competition because there was none, could write and mean. The monopoly was the funding model. The regulation was the accountability. The freedom was the byproduct.

The physical architecture reinforced the culture. Kelly personally designed the Murray Hill building with long corridors and modular rooms that mixed theorists, experimentalists, and technicians across disciplines. Office doors were kept open. Physicists sat next to chemists. Mathematicians sat next to metallurgists. The policy was explicit: you will encounter people who don't work on what you work on. You will talk to them. Something will happen. The building was an idea-mating machine before anyone had the vocabulary for it. Ridley's "ideas having sex" was the architecture of Murray Hill, implemented in brick and corridor.

The math group captured the spirit. Thornton Fry, who ran it, said: "Mathematicians are queer people. Anybody who was queer enough that you didn't know what to do with him, you said, 'This fellow is a mathematician. Let's have him transferred over to Fry.'" Claude Shannon thrived there. "Kind of free-wheeling," he called it. "I enjoyed it more that way, where I was working on my own projects." Shannon juggled, rode a unicycle down the halls, built a machine that solved mazes, and founded information theory. Nobody told him to stop juggling. The monopoly paid for the unicycle.

The American century: why the money was there

America emerged from World War II as the only major industrial power whose territory had not been bombed, whose factories had not been destroyed, whose population had not been decimated. The Bretton Woods system, established in 1944, made the dollar the world's reserve currency, pegged to gold at $35 per ounce. Every other currency pegged to the dollar. The system gave America an export privilege no other nation had: the world needed dollars to trade, and America could print them.

The Marshall Plan rebuilt Europe with American capital, purchasing American goods. The GI Bill educated millions of returning soldiers at government expense, creating the most skilled workforce in history. Federal investment in highways, universities, and basic research — the NSF, the NIH, DARPA, the national laboratory system — created infrastructure that private capital would never have built alone. The Cold War directed defense spending into semiconductors, computing, communications, and aerospace. Bell Labs was a private institution funded by a regulated monopoly, but it operated in an economy that was being systematically invested in by the federal government at every level.

The phone monopoly thrived because the economy thrived. Americans made more calls because Americans had more money, more businesses, more reasons to communicate. The Bell Labs tax grew because the base it taxed grew. The virtuous cycle was not accidental, but it was not designed either. It was the intersection of a global monetary system, a domestic investment program, and a regulatory bargain that nobody had optimized for innovation. The innovation was a side effect. It was the best side effect in history.

The talent migration: Europe's loss, America's gain

The third accident was the movement of people. Between 1933 and 1945, the United States received the greatest transfer of scientific talent in history. Fascism expelled Europe's best minds. America received them.

The Hungarians came first, and they came in extraordinary concentration. John von Neumann. Edward Teller. Eugene Wigner. Leo Szilard. Theodore von Kármán. They were called "The Martians" — a joke about their otherworldly intelligence and impenetrable language. Enrico Fermi, when asked whether extraterrestrials existed, replied: "Of course, they are already here among us: they just call themselves Hungarians."

Von Neumann alone invented game theory, the architecture of the stored-program computer, the mathematical foundation of quantum mechanics, and the implosion mechanism for the atomic bomb. He did this while being a full-time consultant to the Army, the Navy, the Air Force, the Atomic Energy Commission, the RAND Corporation, IBM, and Bell Labs. He was not an exception among the Martians. He was representative.

Einstein fled Germany in 1933. Fermi fled Italy in 1938 — his wife was Jewish. Hans Bethe fled Germany. Felix Bloch fled. Emilio Segrè fled. James Franck resigned his Göttingen post in 1935 in protest and left. George Gamow escaped Soviet Russia. The list is not a list of great scientists. It is a list of people who, had they stayed in Europe, would have been killed. They came to America. They built American science.

The Manhattan Project was staffed disproportionately by European refugees. The Theoretical Division at Los Alamos, under Hans Bethe, was dominated by German-speaking physicists — Peierls, Frisch, Placzek, Bethe, Weisskopf. Oppenheimer quipped: "May the Lord preserve us from the enemy without and from the Hungarians within" — Teller was obsessed with the hydrogen bomb and would not stop talking about it. The bomb was built by refugees from the regimes America was bombing.

After the war, these scientists dispersed into American institutions. Bell Labs got its share. The transistor was invented by Shockley, Bardeen, and Brattain — two Americans and an American. But the environment they worked in was shaped by the presence of European-trained physicists, mathematicians, and engineers who had brought their training, their methods, and their standards with them. The American scientific establishment before the war was provincial. After the war, it was the world's best. The difference was not gradual improvement. It was the sudden arrival of thousands of the most trained minds in Europe, concentrated in a few institutions, funded by a government that had just won a global war and an economy that was growing at rates never seen before or since.

The talent migration was not a policy. It was a humanitarian catastrophe in Europe that America benefited from. The scientists did not come because America had a better research environment. They came because staying in Europe meant death. America built the better research environment after they arrived, in part because they arrived. The environment and the talent co-evolved. The monopoly money funded the environment. The migration supplied the talent. The combination produced Bell Labs.

The paradox: the monopoly that suppressed what it discovered

The story contains a paradox that matters. The regulated monopoly that funded utopian research freedom also suppressed technologies that threatened AT&T's business model. Magnetic tape recording was developed at Bell Labs in the 1930s. AT&T suppressed it for nearly fifty years, fearing that the possibility of recording conversations would discourage telephone use. An answering machine that could record calls was a threat to the business of carrying calls. The monopoly that funded the research killed the product.

Packet switching — the foundation of the internet — was presented to AT&T by Paul Baran in the 1960s. The company dismissed it. AT&T's business was circuit-switched voice calls. Packet switching was the opposite of circuit switching. The monopoly that funded Claude Shannon could not imagine a network built on Shannon's own principles. Fiber optics, mobile telephony, DSL — all developed at Bell Labs, all deployed with deliberate slowness, all constrained by the imperative not to disrupt the existing revenue model.

"Bell Labs was never a place that could originate technologies that could, by the remotest possibility, threaten the Bell system itself." — Tim Wu

The freedom was real. The boundary was real. You could invent anything that did not threaten the monopoly. The transistor was fine — it amplified signals, which was useful for the phone network. The laser was fine — it could carry signals through fiber, someday. Unix was fine — it was an operating system for internal use, and AT&T licensed it essentially for free to universities. C was fine — it was a tool for writing Unix. Information theory was fine — it was mathematics, and mathematics threatens no business model.

But anything that might change how people communicated — anything that might make them use the phone network less, or differently, or not at all — was suppressed. The freedom was conditional. The condition was invisible to the researchers but absolute in its effect. The researchers thought they had freedom. They had freedom within the perimeter. The perimeter was drawn by the business model. The business model was protected by the monopoly. The monopoly was protected by the government. The government was accountable to the voters. The voters wanted cheap phone service. Cheap phone service required the monopoly. The monopoly required the perimeter. The perimeter suppressed the answering machine. The loop was closed.

The breakdown

The AT&T divestiture of 1982 broke the monopoly. Bell Labs survived in diminished form through Lucent, Alcatel-Lucent, and now Nokia. It never recovered its former scale or ambition. The funding model was gone. The regulated utility that had taxed phone calls to fund basic research was replaced by competitive telecommunications companies that had to justify every research dollar to shareholders every quarter. The freedom to spend a decade on information theory with no commercial mandate did not survive the transition. Nothing like it has existed since.

The Bretton Woods system ended in 1971 when Nixon closed the gold window. The dollar floated. The privileged position of the American economy persisted, but the structural guarantee of it did not. The postwar boom was a one-time event, fueled by the destruction of every competitor's industrial base and the creation of a global monetary system that America controlled. Those conditions cannot be recreated without another global war that destroys every other economy, which is not something to wish for.

The talent migration was a one-time event. Europe produced a generation of extraordinary scientists in the 1920s and early 1930s. Fascism expelled them. America received them. The generation aged, retired, and died. The pipeline from European universities to American research institutions has never again operated at that volume or that quality. European universities recovered. European scientists stayed in Europe. The asymmetry was temporary. Its effects were permanent.

What replaced it

The research model that replaced Bell Labs is the venture-capital-funded startup. The startup model is good at developing products. It is terrible at funding basic research with no visible commercial application and a ten-year time horizon. Nobody pitches a VC on "I want to understand the fundamental nature of communication and I need a decade of funding with no deliverables." Claude Shannon would not get funded today. He would be told to focus, to find a market, to build an MVP, to show traction. Information theory would not exist. The bit would not have a name. The digital age would rest on a foundation that was never laid.

The corporate research lab model that survived — Microsoft Research, Google Research, DeepMind, OpenAI — is different from Bell Labs in a crucial respect. These labs are funded by competitive companies in competitive markets. Their research freedom is conditional on continued corporate success and continued executive patience. When the market turns, the research budget is cut. When the executive changes, the research direction changes. The freedom is not structural. It is discretionary. Discretionary freedom can be withdrawn. Bell Labs' freedom was structural — embedded in the regulatory bargain, funded by a tax on a necessity. As long as Americans made phone calls, the research continued. Google Research depends on Google's advertising revenue. If advertising revenue declines, research declines. The funding is not a tax on a necessity. It is a share of a competitive surplus. Competitive surpluses are temporary. Necessities are durable.

The lesson

Bell Labs was not a replicable model. It was a historical accident produced by the intersection of a regulated monopoly, a globally dominant economy, and a one-time talent migration. You cannot recreate it by making your office plan open or by giving your engineers 20% time or by hiring a chief innovation officer. The conditions that produced it were structural, not cultural. The culture was an effect of the conditions. You can copy the open doors. You cannot copy the monopoly, the Bretton Woods system, or the European scientific diaspora of 1933-1945.

The lesson is not that we should recreate Bell Labs. The lesson is that the conditions for deep innovation are structural, not managerial. They depend on funding models that are stable across decades, not quarters. They depend on freedom that is guaranteed by the structure of the institution, not by the goodwill of a manager. They depend on concentrations of talent that are produced by forces larger than any hiring pipeline — war, migration, economic transformation. When those conditions coincide, extraordinary things happen. When they don't, we get what we have: incremental improvement, well-funded but narrowly focused, producing products rather than principles, optimizing the transistor rather than discovering it.

The freedom to innovate is not a policy choice. It is a property of a system. The system that produced Bell Labs no longer exists. The system that replaced it produces different things — faster iteration, better products, more responsive markets. It does not produce the laser, the transistor, Unix, C, and information theory in a single building over three decades. Nothing does. Nothing will, until the structural conditions that made it possible happen again. The conditions were accidents. Accidents are not strategies.


References:

  • Jon Gertner, The Idea Factory: Bell Labs and the Great Age of American Innovation, Penguin, 2012.
  • Tim Wu, The Master Switch: The Rise and Fall of Information Empires, Knopf, 2010.
  • Mervin Kelly, internal Bell Labs memos on research culture, 1940s-1950s.
  • Claude Shannon, A Mathematical Theory of Communication, Bell System Technical Journal, 1948.
  • Related posts: Disruptive Innovation, I, Pencil, BSD is clean, OpenBSD is cleaner

Engineering is the discipline of building things that work within constraints. Every topic on this blog — operating systems, AI models, trading infrastructure, research labs, innovation economics — is examined through the lens of systems design. The lens is engineering. The method is: understand the constraints, design within them, verify the design works, iterate. The domain provides the specifics. The method is universal.

Innovation is not a process. It is a condition. The condition is freedom from quarterly earnings, freedom from feature requests, freedom to pursue a question for a decade. Bell Labs had that condition. Nobody does now.

Algorithmic trading in crypto

Crypto markets are the most algorithmically traded markets in history. CEX-DEX arbitrage extracted $233M in 18 months. MEV is an infrastructure arms race. AI agents now achieve Sharpe ratios above 2.0. Latency is the ultimate edge. The game is being played at machine speed. Understanding it is the first step to not being someone else's exit liquidity.

cryptoalgorithmic-tradingmevarbitragedefilatency

Crypto markets are the most algorithmically traded markets in history. The data is public. The execution is programmable. The settlement is near-instant. The barriers to entry are two orders of magnitude lower than traditional finance. The result is an arms race where latency, strategy, and infrastructure determine who extracts value and who provides it.

In the 18 months from August 2023 to March 2025, 19 major searchers extracted $233 million from CEX-DEX arbitrage on Ethereum alone. Three searchers captured 75% of the volume. The daily transaction count grew 7.2×. This is not a niche activity. This is the dominant mode of professional trading in crypto.

The strategies

CEX-DEX arbitrage. The bread and butter. A token trades at a different price on Binance than on Uniswap. The arbitrageur buys on the cheaper venue, sells on the more expensive, pockets the spread. The trade must execute atomically — both legs must succeed or neither — or the arbitrageur is exposed to inventory risk. Flash loans solve this: borrow millions in a single transaction, execute both legs, repay the loan, keep the profit. If either leg fails, the entire transaction reverts. The loan is never drawn. The risk is zero. The barrier is speed. The fastest bot wins.

Statistical arbitrage. Not a single price discrepancy but a statistical edge across many assets over many trades. Pairs trading: two tokens that historically move together diverge. Buy the underperformer. Short the overperformer. Wait for convergence. Mean reversion: a token spikes on news, the spike is overdone, it reverts. The model identifies the deviation. The bot executes. The edge is small per trade. The volume makes it profitable.

Market making. Provide liquidity on both sides of the order book. Earn the spread. Manage inventory — too much inventory, you're exposed to price moves. Too little, you earn no spread. The market maker's edge is the bid-ask spread. The risk is adverse selection — informed traders trade against you when the price is about to move. The market maker who can't distinguish informed from uninformed flow loses. The market maker who can, wins. This is the oldest trade in finance. Crypto makes it programmable.

MEV extraction. Maximal Extractable Value. The profit that can be extracted by ordering, including, or excluding transactions within a block. Front-running: see a large buy order in the mempool, buy the same token first, sell after the large order moves the price. Sandwich attacks: buy before the victim, let the victim move the price, sell after. The victim pays a higher price. The attacker pockets the difference. MEV extraction is zero-sum. The extractor gains what the user loses. The extraction is algorithmic. The victim is anyone whose transaction waits in the public mempool.

Liquidity sniping. A new token launches. A liquidity pool is created. The sniper's bot detects the pool creation, buys tokens in the same block, and sells after the initial buying pressure drives the price up. The sniper's edge is speed — being first to the pool. The victim is the retail trader who buys after the snipe. The sniper's exit is the retail trader's entry.

JIT (just-in-time) liquidity. Provide liquidity for exactly one block — the block containing a large swap. The JIT LP sees the pending swap in the mempool, deposits liquidity, collects the swap fees, and withdraws in the same block. The LP earns fees with zero inventory risk. The strategy requires atomically bundling deposit, swap, and withdraw in one transaction. Flashbots and similar MEV infrastructure enable it.

Latency is the edge

On fast-finality chains — Ethereum L2s like Arbitrum, Base, ZKsync — block times are sub-second. Priority fees are largely ignored because block builders can't reliably order by fee within that window. The winning strategy is not the highest fee. It is the lowest latency. The bot whose transaction reaches the sequencer first wins the arbitrage. The difference between winning and losing is measured in milliseconds.

A June 2025 study of L2 MEV found that over 80% of reverted transactions on L2s are swap transactions from MEV bots. Bots spam duplicate transactions — rather than paying higher fees, they flood the network with copies, betting that at least one will land first. The spam is economically rational because latency, not fees, determines ordering. The spam degrades the network for everyone else. The degradation is an externality. The externality is unpriced.

Geography matters. The bot running in the same datacenter as the sequencer has a latency advantage measured in single-digit milliseconds. The bot running on a home connection has no chance. Colocation is the moat. The moat is expensive. The expense concentrates the game among professionals. The concentration is visible in the data: three searchers, 75% of the volume.

AI agents are entering

Multi-agent AI systems are now trading crypto with institutional-grade performance. Nex-T1, a 25-agent LLM system powered by GPT-4 Turbo with RAG, achieved a Sharpe ratio of 2.34 over its test period — outperforming single-agent baselines by 65% while cutting maximum drawdown by nearly half. The agents are organized into specialized teams: Research agents scan on-chain data, news, and social sentiment. Risk Management agents evaluate position sizing and portfolio exposure. Execution agents route trades across venues with sub-second latency. Governance agents monitor compliance and override anomalous decisions.

The architecture is the story. Each agent has a narrow responsibility. They communicate through structured interfaces. The system is modular — swap the execution agent without changing the research agent. The parallels to microservices architecture are exact. The difference is that these services trade money. The latency budget is milliseconds. The cost of a bug is not a 500 error. It is a position that moves against you while your agent is still thinking.

The agents use the same primitives as human traders — order books, AMM pools, lending protocols, bridges — but they operate at speeds humans cannot match. The human trader researches for hours. The agent ingests the entire on-chain state in seconds. The human trader monitors a few assets. The agent monitors thousands. The human trader sleeps. The agent doesn't. The asymmetry is structural. The structure favors the agent.

The infrastructure stack

Algorithmic trading in crypto requires infrastructure. The stack:

Layer Component
Data Real-time price feeds from CEXs (Binance, Coinbase, Kraken) and DEXs (Uniswap, Curve, pools on every chain). On-chain data — mempool monitoring, event logs, state diffs. Off-chain data — news, sentiment, order book depth.
Execution Direct node access for low-latency transaction submission. MEV infrastructure — Flashbots, MEV-Boost, private relays — to avoid front-running and extract MEV. Smart contract wallets for programmatic execution. Flash loan contracts for atomic arbitrage.
Strategy The model. Statistical, rules-based, or ML-driven. Backtested on historical data. Validated out-of-sample. Paper-traded before going live. The model is the edge. The edge erodes as competitors replicate it. The erosion is the reason for continuous research.
Risk management Position limits. Exposure limits. Drawdown limits. Circuit breakers that halt trading when losses exceed thresholds. The risk system must be independent of the strategy system — a bug in the strategy should not prevent the risk system from closing positions.
Monitoring Real-time P&L dashboards. Alerting on anomalous behavior — unexpected position sizes, unusual trade frequencies, sudden drawdowns. The monitoring system is the operator's window into a system that moves faster than any human can follow.

The game

Algorithmic trading in crypto is a multiplayer game with asymmetric information, asymmetric infrastructure, and asymmetric speed. The players with the best data, the lowest latency, and the most sophisticated models extract value from the players with worse data, higher latency, and simpler models. The extraction is the game. The game is zero-sum in the short run. In the long run, the infrastructure improves for everyone, the edges compress, and the extraction migrates to new venues, new assets, new strategies. The migration is continuous. The game never ends.

The individual trader entering this game with a Python script and a Binance API key is not a player. They are the liquidity. The extraction targets them. The latency gap ensures they will lose. The only rational response for the non-professional is passive: buy and hold, provide liquidity through automated vaults, or delegate to professional market makers who have the infrastructure. The active game is for professionals. The professional game is played at machine speed. The machines are getting faster.


References:

  • AFT 2025, "CEX-DEX Arbitrage on Ethereum: 2023-2025."
  • "First-Spammed, First-Served: MEV Extraction on Fast-Finality Blockchains," June 2025.
  • Nexis-AI, "Nex-T1: Multi-Agent Framework for Autonomous DeFi Trading," October 2025.
  • "RediSwap: MEV Redistribution at the Application Layer," 2024.
  • Related posts: Scarcity Rules Everything, Design the Game

Engineering is the discipline of building things that work within constraints. Every topic on this blog — operating systems, AI models, trading infrastructure, research labs, innovation economics — is examined through the lens of systems design. The lens is engineering. The method is: understand the constraints, design within them, verify the design works, iterate. The domain provides the specifics. The method is universal.

Crypto markets are not more efficient than traditional markets. They are more programmable. Programmability attracts algorithms. Algorithms compete. Competition compresses edges. The compression is the market becoming efficient in real time.

Why algorithmic trading is a game worth playing

Algorithmic trading is the discipline of encoding market beliefs into executable strategies and testing them against reality. It is the closest thing to pure applied epistemology in finance. Every trade is a hypothesis. Every P&L is a test result. This post explains why the game is worth playing and the nine books that teach you how.

algorithmic-tradingbooksquantitative-financemarket-microstructureeducation

Algorithmic trading is not about speed. It is not about colocation, microwave towers, or the arms race for nanoseconds. That is high-frequency trading — one branch of a larger discipline. Algorithmic trading is the broader practice: encoding market beliefs into executable strategies and testing them against reality. Every trade is a hypothesis. Every P&L is a test result. The market is the laboratory. The laboratory never closes.

The game is worth playing for three reasons. First, it forces intellectual honesty. A backtest that looks good and a live strategy that loses money cannot both be right. The market's verdict is final. Second, it forces you to understand markets at a level that passive investing never requires. You must understand order flow, liquidity, market impact, adverse selection. You must understand what you're trading against and why your edge exists. Third, it is composable with everything else you know. Statistics, computer science, game theory, behavioral economics — every intellectual tool you have finds application in trading. The trading sharpens the tools. The tools sharpen the trading.

This post is about the books that teach you to play. Nine books. Each teaches a different part of the game. Each matters for a different reason. The reading order builds understanding from the ground up.

1. Larry Harris — Trading and Exchanges: Market Microstructure for Practitioners (2003)

Harris was the SEC's chief economist. His book is the best single-volume education in how markets actually work. 600 pages. Assumes nothing. Explains everything: order types, priority rules, trading costs, market maker obligations, dealer markets vs. auction markets, transparency, fragmentation, regulation.

The lesson: the market is not a black box that converts orders into executions. It is a specific set of rules and participants with specific incentives. The rules determine who wins and who loses. Understand the rules before you trade.

Harris organizes markets along two dimensions: the trading session (continuous vs. call auction) and the market structure (order-driven vs. quote-driven vs. brokered). Every real market is a hybrid. The NYSE is continuous and order-driven but runs call auctions at the open and close. The corporate bond market is quote-driven and brokered. Crypto AMMs are a form that didn't exist when Harris wrote — formula-driven markets. The taxonomy is portable. Add new forms. The principles remain.

A trader who hasn't read Harris doesn't know what a limit order is, how it interacts with other orders in the book, why the spread exists, or what happens to their order when they click submit. They are trading blind. The blindness is expensive.

2. Maureen O'Hara — Market Microstructure Theory (1995)

Where Harris tells you what happens, O'Hara tells you why. The book traces the evolution of microstructure theory: from inventory models (market makers set spreads to manage inventory) to information-based models (market makers set spreads to protect against informed traders) to strategic trader models (traders anticipate each other's strategies).

The key insight: the bid-ask spread is not a transaction cost. It is an information cost. The market maker posts a bid and an ask, offering to trade with anyone. Some of those anyones know more than the market maker. The spread compensates for the expected loss to informed traders. This is why spreads widen during volatility (more uncertainty → more adverse selection), why small-cap stocks have wider spreads (less public information → more private information), and why AMM fees are higher for volatile pairs. The explanation is the same across markets. O'Hara gives you the explanation.

3. Ernest P. Chan — Algorithmic Trading: Winning Strategies and Their Rationale (2013)

Chan's book is the practical starting point for someone who wants to build and test trading strategies. Where the theory books give you models, Chan gives you: here is how to backtest a mean-reversion strategy. Here is how to source data. Here is why your backtest looks great and will lose money live.

The lesson: backtesting is hard. Most backtests are overfit. The overfit is invisible to the person who ran the backtest because they want the strategy to work. The only defense is out-of-sample testing, paper trading, and the humility to accept that a strategy that backtests perfectly will lose money in production.

Chan's most valuable chapter catalogs backtesting pitfalls: look-ahead bias (using data not available at trade time), survivorship bias (testing on assets that survived), data-snooping bias (testing many strategies and reporting the winner). Each bias inflates backtest returns. Each is present in most amateur backtests. Each is avoidable. Avoiding them is the discipline that separates algorithmic trading from gambling dressed in Python.

The book covers mean reversion, momentum, pairs trading, and ETF arbitrage. The specific strategies are dated — most stopped working as more capital chased them. The principles of strategy development, backtesting, and risk management are permanent. Chan teaches you to fish. The fish you catch are your own.

4. Álvaro Cartea, Sebastian Jaimungal, José Penalva — Algorithmic and High-Frequency Trading (2015)

This is the graduate textbook. It assumes Harris and O'Hara. It applies stochastic optimal control to execution, market making, and liquidity provision. The mathematics is advanced — SDEs, Hamilton-Jacobi-Bellman equations, impulse control. The reward is a unified framework for thinking about trading as optimization under uncertainty.

The lesson: trading is an optimization problem. The uncertainty has structure — price dynamics, order arrival dynamics, market impact. The structure can be modeled. The model can be solved. The solution is a strategy.

The gap between the continuous-time model and discrete market reality is where the practitioner's edge lives. Cartea gives you the model. You must adapt it to the reality of gas costs, discrete blocks, and adversarial MEV searchers. The adaptation is the work.

5. Marco Avellaneda and Sasha Stoikov — High-Frequency Trading in a Limit Order Book (2008)

A 25-page paper, not a book. It is on this list because it is the most influential single publication in algorithmic market making. Every market-making bot in production descends from it.

The model: a market maker chooses bid and ask quotes to maximize expected utility. The state: cash, inventory, price. The control: how far from the mid-price to place quotes. The solution: skew quotes away from inventory. Long inventory → sell more aggressively. Short inventory → buy more aggressively. Spread widens with volatility and risk aversion.

The paper is implementable. The calibration is harder than the implementation. The calibration is the edge. The edge erodes as more people implement the model. The erosion is why the field keeps moving. Read the paper. Implement it. Learn what happens when your calibration is wrong. Then build something better.

6. Andrew Lo — Adaptive Markets: Financial Evolution at the Speed of Thought (2017)

Lo's book is not about strategies. It is about the framework within which all strategies operate. The efficient market hypothesis says prices reflect all available information. Lo says: markets are adaptive systems populated by boundedly rational agents competing for profits. The competition produces efficiency as an emergent property, not a static condition. Efficiency is approached, never reached.

The lesson: strategies work, attract capital, compress their own edge, and stop working. The cycle is evolutionary. The strategist must continuously find new edges. The search is the career.

Lo integrates neuroscience, evolutionary biology, and behavioral economics into a unified theory. The theory explains why arbitrages exist, persist, and disappear. The explanation is evolutionary. The evolution is driven by competition. The competition is getting smarter. The smarter competition is the subject of the next book.

7. Peter Bernstein — Against the Gods: The Remarkable Story of Risk (1996)

Bernstein tells the history of risk management from the Renaissance to modern finance. The mathematicians, philosophers, and gamblers who invented probability, statistics, and derivatives pricing. Not a trading book. The intellectual infrastructure that makes trading possible.

The lesson: risk is quantifiable. The quantification of risk — modeling the future as a probability distribution rather than the will of the gods — is the defining intellectual achievement of modern capitalism. Every trading strategy is a bet on a distribution. Understanding the distribution is the edge. Understanding that the distribution is an estimate, and estimates are wrong, is the meta-edge.

Bernstein's history ends before 2008. The crisis validated his thesis in reverse: quantification breeds overconfidence. The model is not the territory. The territory contains fat tails that the model missed. The fat tails are the subject of the next book.

8. Nassim Nicholas Taleb — Dynamic Hedging: Managing Vanilla and Exotic Options (1997)

Taleb was an options market maker before he was a public intellectual. Dynamic Hedging is a collection of lessons from actually hedging options in real markets. Delta hedging works until it doesn't. Gamma risk kills you when volatility spikes. The tail is where the money is lost. The tail is thicker than Black-Scholes assumes.

The lesson: the market is not log-normal. Tails are fat. Hedging strategies that assume thin tails fail when the tail arrives. The tail arrives more often than the model predicts. Respect the tail.

Out of print and expensive. Worth it. Taleb's later books — Fooled by Randomness, The Black Swan — are philosophical expansions. Read Dynamic Hedging for the trading. The trading book is better.

9. Michael Lewis — Flash Boys: A Wall Street Revolt (2014)

Lewis tells the story of Brad Katsuyama and IEX — traders who discovered the U.S. stock market was rigged for speed and built an exchange to fix it. The book is a thriller. It is also the best introduction to the reality of modern market infrastructure for someone who has never thought about it.

The lesson: the market is not a level playing field. Speed is an advantage. The advantage is purchased through infrastructure. The infrastructure cost creates barriers. The barriers concentrate profits. The concentration is structural.

Flash Boys is controversial. HFT firms argue Lewis misunderstood market making. The controversy is beside the point. The book's contribution is making visible the physical infrastructure of trading — the data centers, the fiber, the matching engines — that determines who wins. The infrastructure was invisible before Lewis. It is not invisible now.

The reading order

Start with Harris. Understand the mechanism. You cannot trade what you don't understand.

Then O'Hara. Understand why the mechanism works the way it does. The theory makes the practice intelligible.

Then Chan. Learn the discipline of turning an idea into a backtest and a backtest into a live strategy. Make the mistakes Chan warns about. Learn from them.

Then Cartea. The mathematics of optimization. Not everyone needs this. The people who do know who they are.

Then Avellaneda-Stoikov. Short. Implementable. The bridge from theory to a working market-making bot. Build it. Watch it lose money because your calibration is wrong. Fix the calibration.

Then Lo. The evolutionary framework. Understand why your edge will erode and why that's normal and what to do about it.

Then Bernstein. The intellectual history. Understand where the tools came from. Respect the minds that built them.

Then Taleb. The reminder that the tools have limits. The limits are where you lose money. The reminder is uncomfortable. The discomfort is productive.

Then Lewis. The human story. The market is not an abstraction. It is people, infrastructure, incentives, and politics. Lewis makes it human. The humanity matters.

The bookshelf is a curriculum. The curriculum is a career. The career is a continuous search for edges that haven't yet been competed away. The search is the game. The game is worth playing.


References:

  • Larry Harris, Trading and Exchanges: Market Microstructure for Practitioners, Oxford University Press, 2003.
  • Maureen O'Hara, Market Microstructure Theory, Blackwell, 1995.
  • Ernest P. Chan, Algorithmic Trading: Winning Strategies and Their Rationale, Wiley, 2013.
  • Álvaro Cartea, Sebastian Jaimungal, José Penalva, Algorithmic and High-Frequency Trading, Cambridge University Press, 2015.
  • Marco Avellaneda and Sasha Stoikov, "High-Frequency Trading in a Limit Order Book," Quantitative Finance, 2008.
  • Andrew Lo, Adaptive Markets: Financial Evolution at the Speed of Thought, Princeton University Press, 2017.
  • Peter Bernstein, Against the Gods: The Remarkable Story of Risk, Wiley, 1996.
  • Nassim Nicholas Taleb, Dynamic Hedging: Managing Vanilla and Exotic Options, Wiley, 1997.
  • Michael Lewis, Flash Boys: A Wall Street Revolt, W.W. Norton, 2014.
  • Related posts: DEX trading series, On Scarcity

Engineering is the discipline of building things that work within constraints. Every topic on this blog — operating systems, AI models, trading infrastructure, research labs, innovation economics — is examined through the lens of systems design. The lens is engineering. The method is: understand the constraints, design within them, verify the design works, iterate. The domain provides the specifics. The method is universal.

Trading is applied epistemology. Every trade is a hypothesis. Every P&L is a test result. The market is the laboratory. The laboratory never closes. The experiment never ends.

The economics of data

Data is not free. It costs money to collect, store, transform, and serve. It generates value when used and costs when ignored. The economics of data engineering is the economics of an asset that depreciates faster than any other: data that was valuable yesterday may be worthless today, but it still costs money to store.

data-engineeringeconomicscostbuild-vs-buydata-value

Data is an asset. Data is also a liability. The distinction is whether the data generates more value than it costs to maintain. Most organizations treat all data as an asset. Most data is a liability. The difference between the two is the difference between a data platform that pays for itself and a data platform that is a cost center the CFO wants to cut.

The cost structure

The cost of data has four components. Each is measurable. Few organizations measure them.

Ingestion cost. The engineering time to build and maintain data pipelines. The compute cost of running them. The cost of the ingestion tools (Fivetran, Airbyte) or the infrastructure they run on. Ingestion cost scales roughly linearly with the number of data sources.

Storage cost. The cost of storing data in the warehouse or data lake. Storage is cheap — $20-30 per TB per month in cloud warehouses. Storage is also unbounded — data accumulates indefinitely. A table that cost $100/month to store last year costs $200/month this year. The growth is compound. The compound growth is invisible until someone looks at the storage bill.

Transformation cost. The compute cost of running transformations. Every dbt run consumes warehouse credits. Every hourly refresh of the dashboard tables consumes credits. Transformation cost scales with data volume and refresh frequency. A dashboard that refreshes every hour costs 24 times more than a dashboard that refreshes once daily. The refresh frequency is a business decision. The cost is an engineering consequence.

Serving cost. The compute cost of querying the data. Every dashboard load, every ad-hoc query, every ML training job consumes warehouse credits. Serving cost scales with the number of users and the complexity of their queries. A poorly written query that scans the entire table costs more than a query that uses partitions and filters. The query author doesn't see the cost. The engineering team pays the bill.

The total cost is the sum of these four. For a typical mid-size organization, the breakdown is roughly: 15% ingestion, 25% storage, 35% transformation, 25% serving. The exact numbers vary. The pattern is consistent: most of the cost is in transformation and serving, not storage. Storage is the scapegoat. Transformation is the real expense.

Why data projects fail

Data projects fail for economic reasons disguised as technical reasons. The project was "too complex." The pipeline "couldn't scale." The data "wasn't reliable." These are symptoms. The cause is that the project's costs exceeded its benefits, and nobody measured either.

A data project has benefits: faster decisions, better decisions, new revenue from data products, reduced cost of manual reporting. The benefits are diffuse — they accrue to many people across the organization. They are hard to measure. They are easy to overstate in the business case and impossible to verify after the fact.

A data project has costs: engineering time, compute, storage, ongoing maintenance. The costs are concentrated — they are paid by the data engineering team. They are easy to measure. The data engineering team knows exactly how much the pipeline costs to build and run. They are rarely asked.

The asymmetry — diffuse benefits, concentrated costs, neither measured accurately — produces predictable outcomes. The project is approved based on overstated benefits. The project is built. The costs exceed expectations. The benefits are invisible because nobody is measuring them. The project is declared a failure. The postmortem blames the technology. The technology was not the problem. The economics were the problem. The economics were never made explicit.

Build vs. buy

The build-vs-buy decision in data engineering is the same as in software engineering: build if the internal cost is less than the external cost, adjusted for risk, control, and strategic value. The data-specific twist: data tools have extreme economies of scale. A managed service (Fivetran, Snowflake, dbt Cloud) amortizes its development cost across thousands of customers. An internal pipeline tool amortizes its cost across one organization. The managed service is almost always cheaper for common use cases — ingesting from standard sources, transforming with SQL, serving dashboards. The internal tool is justified only when the use case is unique to the organization or when the strategic value of control exceeds the cost premium.

The trap: organizations underestimate the maintenance cost of internal tools. The initial build is a month. The maintenance is years — bug fixes, feature requests, onboarding documentation, operational support. The maintenance cost is proportional to the number of users and the rate of change of the tool's dependencies. The number of users grows. The rate of change increases. The maintenance cost compounds. The internal tool that seemed cheaper than Fivetran at year one is more expensive than Fivetran by year three. The cost has shifted from the vendor to the internal team. The shift is invisible because internal labor is a fixed cost. The fixed cost is already paid. The marginal cost of asking the internal team to maintain another tool appears to be zero. It is not zero. It is opportunity cost — the features they could have built instead of maintaining the ingestion tool.

The value of data

Data has value when it is used to make a decision that produces a better outcome than the decision that would have been made without it. The value is the difference between the outcome with data and the outcome without. The value is measurable in principle. It is almost never measured in practice.

Data that is collected but never used has negative value — it cost money to collect, store, and maintain, and it generated zero benefit. The storage cost accumulates. The maintenance cost accumulates. The value is zero. The net is negative. The negativity is invisible because the costs are aggregated into the data platform budget and the zero benefit is never calculated. The calculation would require asking: "what decision did this dataset inform, and what was the outcome?" Nobody asks. The data accumulates. The costs accumulate. The value is zero. The net is negative.

The most valuable data in an organization is often the data that doesn't exist yet — the data that would answer a question the business has been asking for months but nobody has had the time to pipeline. The value is latent. The latency is a prioritization failure. The prioritization failure is an economic failure. The economic failure is that the cost of building the pipeline was compared to the engineering time required, not to the value of the decisions it would inform. The comparison was never made. The pipeline was never built. The decisions were made without data. The outcomes were worse than they could have been. The difference is the cost of not building the pipeline. The cost is real. It is unmeasured.

The depreciation problem

Data depreciates faster than any other asset. Customer behavior data from 2020 has limited relevance to customer behavior in 2026. The relevance decays over time. The storage cost is constant. The value declines. The crossover point — when the storage cost exceeds the remaining value — arrives faster than organizations expect. Most organizations never delete data. The data accumulates. The storage cost grows. The average value of the stored data declines. The decline is the depreciation.

Depreciation should be accounted for. Data that is older than its useful life should be archived or deleted. The useful life depends on the domain: transaction records (7+ years, for compliance), web analytics (2 years), ML training features (6 months, or until the model is retrained). The retention policy should be explicit. The policy should be enforced automatically. The enforcement should be a pipeline that deletes old data. Most organizations have no retention policy. The absence of policy is a policy — keep everything forever. Forever is expensive.

The discipline

The discipline of data economics is the discipline of measuring what matters. Measure the cost of every pipeline: build cost, run cost, maintenance cost. Measure the value of every dataset: what decisions does it inform, what outcomes does it improve? Compare the two. Keep the datasets where value exceeds cost. Kill the rest. The killing is the discipline. The discipline is rare. The rarity is why data platforms are cost centers. The cost center is a choice. The choice is to not measure.


References:

Storage engines and the physics of query

The choice of storage format determines everything above it: query speed, storage cost, schema flexibility, concurrent access. Row-oriented for transactions. Columnar for analytics. Parquet and Iceberg for the lakehouse. DuckDB for the laptop. The physics of storage is the physics of data.

data-engineeringstorageparqueticebergcolumnarduckdb

Data is stored as bytes on disk. How those bytes are organized determines how fast they can be read, how much space they occupy, and what kinds of queries are possible. The physics of storage is the foundation of data engineering. Everything above — the pipelines, the transformations, the dashboards — depends on choices made at the storage layer. The choices are irreversible enough that getting them wrong is expensive.

Row vs. column

Data can be stored row-by-row or column-by-column. In row-oriented storage, all the fields of a single record are stored together. Record 1: (name, age, city, salary). Record 2: (name, age, city, salary). Row storage is optimal for transactional workloads — inserting, updating, deleting individual records. The database reads or writes the entire row at once. PostgreSQL, MySQL, and most OLTP databases use row storage.

In column-oriented storage, all the values of a single column are stored together. Column name: (Alice, Bob, Carol, Dave). Column age: (30, 25, 35, 28). Column storage is optimal for analytical workloads — aggregating, filtering, grouping across many rows. A query that computes the average salary reads only the salary column. It skips name, age, and city. The reduction in I/O is the performance gain. Columnar databases — Snowflake, BigQuery, Redshift, ClickHouse — are designed for analytics.

The columnar advantage comes from three properties. First, columnar compression is more effective than row compression because values within a column are similar — all integers, all strings of similar length, all dates. Run-length encoding, dictionary encoding, and delta encoding compress columns efficiently. Second, columnar storage enables vectorized execution — the query engine operates on batches of values rather than individual rows, using SIMD instructions for parallelism within a single core. Third, columnar storage enables late materialization — the query engine can filter on one column before reading other columns, reducing the total I/O.

The trade-off: columnar storage is bad for point queries. Finding a single row by ID requires reading every column independently and reassembling the row. Row storage does this in a single read. The right storage format depends on the workload. Most data engineering workloads are analytical. Columnar is the default.

File formats

CSV. Comma-separated values. Human-readable. Universally supported. No schema enforcement. No type information. No compression. No nested data. CSV is the lowest common denominator. It is the format you use when you need to exchange data with a system that speaks nothing else. It is not a format for production storage.

JSON. Nested, semi-structured. Self-describing — each record contains its field names. Human-readable (for small records). Widely supported. Inefficient — field names are repeated in every record, numbers are stored as text. JSON is the format of APIs and event streams. It is the format you ingest, not the format you query.

Parquet. Columnar, compressed, with schema embedded in the file footer. Developed by Twitter and Cloudera in 2013. The dominant format for analytical workloads in the Hadoop and cloud ecosystems. Parquet stores data in row groups — chunks of rows stored column-by-column within the chunk. The row group size balances read parallelism (more groups = more parallelism) with columnar efficiency (larger groups = better compression). Parquet supports predicate pushdown — the query engine reads the file footer to determine which row groups contain data matching the query's filters, and skips the rest. The skipping is the performance.

Avro. Row-oriented, with schema stored in the file header. Developed by Doug Cutting for Hadoop. Used primarily for streaming data and as a serialization format for Kafka messages. Avro supports schema evolution — adding, removing, or modifying fields over time while maintaining backward compatibility. The schema evolution is the reason Avro is used for streams. Streams are long-lived. Schemas change. Avro handles the change.

ORC. Optimized Row Columnar. Developed by Hortonworks for Hive. Similar to Parquet in design. Better compression. Less ecosystem support. Parquet won the format war. ORC is still used in the Hive ecosystem but is not the default for new projects.

Table formats

File formats solve the problem of how to store a single file. Table formats solve the problem of how to manage a collection of files as a single table — partitioning, schema evolution, time travel, concurrent writes.

Apache Iceberg. Developed at Netflix. The dominant table format in 2024-2026. Iceberg tracks table metadata — the list of files that comprise the table, the schema, the partitions, the statistics — in a manifest. Queries read the manifest to determine which files to scan. Inserts, updates, and deletes produce new files. Old files are garbage-collected. The table is a logical abstraction over a collection of physical files. Iceberg supports hidden partitioning — the partition scheme is stored in metadata, not in the file path. Changing the partition scheme does not require rewriting the data. Schema evolution is additive — adding a column is a metadata change. Time travel — querying the table as of a past timestamp — is a manifest lookup. Iceberg works with multiple query engines — Spark, Trino, Flink, Snowflake, BigQuery, DuckDB. The engine independence is the architectural win.

Delta Lake. Developed at Databricks. Similar to Iceberg in concept. Deeper integration with Spark. Uses a transaction log rather than manifests. Supports ACID transactions, schema enforcement, and versioning. The competition between Iceberg and Delta Lake is the format war of the 2020s. Both are winning. The real winner is the user, who gets a standard abstraction over object storage.

Apache Hudi. Developed at Uber. Designed for streaming and incremental processing. Supports record-level upserts and deletes. More complex than Iceberg or Delta Lake. Used when the workload requires frequent updates to individual records — a streaming pipeline that must correct errors in previously written data.

Query engines

Data warehouses (Snowflake, BigQuery, Redshift). Fully managed, SQL-based, designed for analytical queries on structured data. Separate storage from compute. Scale elastically. Handle concurrency, security, and administration. The data warehouse is the default for organizations that have data analysts who write SQL and need a managed platform.

Query engines (Trino, Presto, Starburst). Federated SQL engines that query data in place — Parquet files in S3, tables in PostgreSQL, streams in Kafka. No data loading. No ETL. The query engine pushes computation to the data. Trino is the engine behind many data lake architectures. It is fast. It is complex to operate at scale.

Embedded engines (DuckDB). An in-process analytical database. No server. No configuration. Runs on a laptop. Reads Parquet, CSV, JSON directly. DuckDB is the SQLite of analytics. It is the engine you use when you need to query a Parquet file on your laptop and don't want to set up a warehouse. It is also the engine that is eating the low end of the analytical market — queries that would have required a warehouse five years ago now run in DuckDB on a MacBook.

Stream processors (Flink, Kafka Streams, RisingWave). Process data as it arrives, producing results continuously. Designed for real-time dashboards, alerting, and event-driven applications. Maintain state in memory and on disk. Provide exactly-once semantics. The stream processor is the complement to the warehouse — the warehouse handles historical analysis, the stream processor handles real-time decisions.

The physics

The physics of storage is the physics of data engineering. Data at rest must be organized so that data in motion can be processed efficiently. The organization is the schema. The format is the encoding. The table format is the abstraction. The query engine is the processor. Each layer constrains the layer above. The constraints are not limitations. They are the design. The design determines what is possible. What is possible determines what is built.


References:

  • Apache Iceberg, "Specification," Iceberg Documentation.
  • DuckDB, "Why DuckDB," DuckDB Documentation.
  • Netflix Tech Blog, "Apache Iceberg: An Architectural Look Under the Covers," 2020.
  • Related posts: The Unix philosophy, libp2p

Data quality and the problem of truth

Data quality is the hardest problem in data engineering because it is not purely technical. Bad data looks like good data. Errors propagate silently. The people who know the data is wrong are not the people who can fix it. The discipline of data quality is the discipline of making wrongness visible before anyone makes a decision based on it.

data-engineeringdata-qualitygovernancetestinglineage

Data quality is the problem of ensuring that data is correct, complete, consistent, and timely. It is the hardest problem in data engineering because it is not purely technical. A pipeline can run successfully and produce garbage. The tests can pass and the data can still be wrong — the tests test what you thought to test, and you didn't think of everything. The consumers of the data — the analysts, the dashboard viewers, the ML models — trust the data until they don't. The moment they stop trusting it, every number produced by the data platform becomes suspect. Restoring trust is harder than maintaining it. Most organizations never fully restore it.

The dimensions of quality

Data quality has multiple dimensions. Each is necessary. None is sufficient.

Accuracy. The data reflects reality. The sales amount in the warehouse matches the sales amount in the source system. The match should be exact. Approximate is not good enough — a 1% error on revenue, compounded across months, produces financial statements that don't reconcile. Accuracy is verified by reconciliation: comparing warehouse aggregates to source system aggregates. The reconciliation should be automated. It rarely is.

Completeness. All the expected data is present. No missing rows, no missing columns, no missing values where values should exist. Completeness is verified by volume checks: does today's row count fall within the historical range? Completeness failures are the most common data quality issue and the most embarrassing — "the dashboard shows zero revenue because the pipeline didn't run."

Consistency. The same data has the same meaning across tables. Customer ID 123 in the orders table refers to the same customer as Customer ID 123 in the customers table. Consistency is verified by referential integrity checks: do all foreign keys resolve to existing primary keys? Consistency failures produce silent errors — queries that return wrong results without complaining.

Timeliness. The data is available when it is needed. A dashboard that shows yesterday's data at 9 AM is useful. A dashboard that shows yesterday's data at 4 PM is not — the decisions were already made. Timeliness is verified by freshness checks: did the pipeline run within its SLA window? Timeliness failures are operations failures dressed as data failures.

Uniqueness. No duplicate records. The same event should not appear twice in the same table. Uniqueness is verified by primary key checks. Duplication failures are insidious — they inflate counts, double revenue, and are invisible to checks that only verify that values are within expected ranges.

Why data rots

Data rots because the world changes and the data model doesn't. A source system is upgraded. A column is renamed. A new product category is added. A business rule changes — "we now count subscriptions as revenue when billed, not when collected." The pipeline wasn't updated. The data in the warehouse no longer matches the data in the source or the expectations of the business. The rot is gradual. The rot is invisible until someone compares two numbers that should match and finds they don't.

Data rots because humans make errors and the errors accumulate. An analyst writes a SQL query with a join condition that accidentally drops rows. The query becomes a view. The view becomes a dashboard. Six months later, someone notices the numbers are low. The error was in the original query. The query was never tested. The test would have caught the error. The test didn't exist.

Data rots because documentation drifts from reality. The data dictionary says the status column contains 'active', 'inactive', and 'pending'. The application added 'suspended' six months ago. The documentation wasn't updated. The analyst's query filters to status IN ('active', 'inactive', 'pending'). Suspended customers are invisible. The invisibility is a data quality failure caused by a documentation failure.

The testing pyramid for data

Software engineering has the testing pyramid: unit tests at the bottom, integration tests in the middle, end-to-end tests at the top. Data engineering needs its own pyramid.

Column-level tests. Does this column contain nulls? Are values within expected ranges? Are all values from the expected set? Column-level tests catch the most common failures: missing data, out-of-range data, unexpected values. They are cheap to write and fast to run. Every column that matters should have at least one test.

Table-level tests. Is the primary key unique? Are there more rows than the minimum expected? Fewer rows than the maximum expected? Do foreign keys resolve? Table-level tests catch structural failures: duplicates, missing data, referential integrity violations.

Cross-table tests. Does the revenue in the orders table match the revenue in the payments table? Does the customer count in the warehouse match the customer count in the source? Cross-table tests catch reconciliation failures. They are the most valuable tests and the least common because they require understanding the relationships between systems.

Business logic tests. Does the total revenue for March 2026 match the known value from the audited financial statements? Business logic tests catch semantic errors — the data is structurally correct but wrong. They require known reference values. The reference values must come from outside the data platform — from the accounting system, from the source application, from manual audit. The independence of the reference value is what makes the test valid.

Governance

Data governance is the set of policies and processes that ensure data quality at scale. Governance answers: who owns this data? Who can access it? What does it mean? How long is it retained? What are the quality standards?

Governance fails when it is imposed by a central team without the authority to enforce it. The central team writes policies. The domain teams ignore them. The policies are documents. The documents are unread. Governance succeeds when it is embedded in the platform. Access controls are enforced by the warehouse, not by policy. Data classification is enforced by automated scanning, not by manual tagging. Quality standards are enforced by automated testing, not by review checklists. The platform is the enforcement mechanism. The mechanism is the governance.

The modern approach to governance is the data catalog. A catalog — Alation, Atlan, DataHub, Amundsen — indexes all data assets across the organization. It shows lineage — where data came from and where it goes. It shows ownership — who is responsible for this dataset. It shows quality metrics — when was this dataset last validated, what tests does it pass. It shows usage — who queries this dataset, how often. The catalog makes data discoverable. Discoverability is the first step toward quality. You cannot improve what you cannot find.

The principle

Data quality is not a project. It is a practice. There is no point at which you are "done" with data quality. The sources change. The business changes. The consumers' expectations rise. The practice must be continuous: test, monitor, fix, repeat. The practice must be owned by the people closest to the data. The platform must make the practice easy. The alternative is a data platform that produces numbers nobody trusts. A platform that produces untrusted numbers is a platform that has failed. The failure is not technical. It is organizational. The fix is not a tool. It is a commitment.


References:

Pipelines, ETL, and the art of moving data

A data pipeline is a program that moves data from A to B, transforming it along the way. It sounds simple. It is not. Pipelines fail silently, corrupt data, run late, and cost more to maintain than to build. The discipline of pipeline engineering is the discipline of making data movement boring.

data-engineeringetleltpipelinesorchestration

A data pipeline is a program that moves data from somewhere to somewhere else, transforming it along the way. It is the simplest concept in data engineering. It is also where most of the pain lives. Pipelines break. Pipelines run late. Pipelines produce wrong data and nobody notices for three weeks. Pipelines cost ten times more to maintain than to build. The discipline of pipeline engineering is the discipline of making data movement boring. Boring is the highest compliment a pipeline can receive.

ETL vs. ELT

The traditional pattern is ETL: Extract, Transform, Load. Extract data from source systems. Transform it — clean, enrich, aggregate — using an external processing engine. Load the transformed data into the warehouse. The transformation happens outside the warehouse, in a pipeline tool or custom code.

The modern pattern is ELT: Extract, Load, Transform. Extract data from sources. Load it into the warehouse raw, in its original format. Transform it inside the warehouse using SQL. The transformation happens after loading, using the warehouse's compute.

The shift from ETL to ELT was driven by the cloud data warehouse. Snowflake and BigQuery separate storage from compute. You can run transformations on massive datasets without provisioning infrastructure. The warehouse scales compute elastically. The transformation is just SQL. The SQL is version-controlled, tested, and documented by tools like dbt. The analyst who writes the query can also write the transformation. The engineer who built the pipeline doesn't need to understand the business logic.

ELT has a deeper advantage: the raw data is preserved. If the transformation logic is wrong, you can fix it and re-transform from the raw data. In ETL, if the transformation is wrong and the raw data was discarded, you must re-extract from the source — if the source still has the data. The raw layer is insurance. The insurance costs storage. Storage is cheap. The insurance is worth it.

The properties of a good pipeline

Idempotent. Running the pipeline twice produces the same result as running it once. If a pipeline fails halfway through and you restart it, you should not get duplicate data. Idempotency is achieved by deduplication — checking whether a record already exists before inserting it — or by overwriting the output partition entirely. Overwriting is simpler. Deduplication is harder but necessary when you can't afford to recompute the entire output.

Retryable. If the pipeline fails, it can be restarted without manual intervention. The retry should be automatic, with exponential backoff, up to a maximum number of attempts. The failure should be logged, with enough context to debug it. The alert should fire only after all retries are exhausted. Alerting on the first failure generates noise. Alerting after retry exhaustion generates signal.

Observable. You can answer: did the pipeline run? Did it succeed? How many rows did it process? How long did it take? Were there any anomalies — zero rows when there should be thousands, ten times the usual row count, nulls in columns that should never be null? Observability requires metrics, logs, and alerts. The metrics must be queryable historically — "is this week's row count unusual compared to the last twelve weeks?" The alert threshold must be tuned to avoid false positives while catching genuine anomalies.

Testable. You can verify that the pipeline produces correct output. Schema tests: does the output table have the expected columns with the expected types? Data tests: are primary keys unique? Are foreign keys present in the referenced table? Are values within expected ranges? Business logic tests: does the revenue column sum to the expected value given known inputs? Tests catch errors before users do. Users catching errors is the worst outcome.

Lineage-tracked. You can trace any row in any dashboard back to its source. The trace requires metadata: which pipeline produced this table, from which source tables, using which transformation logic, running at which time. Lineage is essential for debugging — "this number looks wrong, where did it come from?" — and for impact analysis — "if we change this source schema, what downstream tables are affected?" Lineage is the data engineer's call stack.

The patterns

Full refresh. Drop the output table. Rebuild it from scratch. Simple. Correct. Expensive for large tables. The full refresh is the default for small datasets and the fallback when incremental logic fails.

Incremental. Process only the data that changed since the last run. Requires a reliable way to identify changed records — a timestamp column, a change log, a CDC feed. More complex than full refresh. More efficient. The incremental pattern is necessary for large tables but introduces the risk of drift: over time, incremental updates accumulate errors that a full refresh would eliminate. Periodic full refreshes — weekly, monthly — reset the drift.

CDC (Change Data Capture). Read the database's transaction log directly. Capture every insert, update, and delete as it happens. Debezium for PostgreSQL, MySQL, MongoDB. The CDC feed is a stream of events. The pipeline consumes the stream and applies changes to the warehouse. CDC is the gold standard for freshness — the warehouse is seconds behind the source. It is also the most operationally complex. The CDC connector can fail. The transaction log can be purged before the connector reads it. The schema can change and break the connector's parsing. CDC is powerful. CDC is not free.

Lambda architecture. Maintain two parallel pipelines: a batch layer that processes all historical data and produces accurate but delayed results, and a speed layer that processes recent data in real time with approximate results. The serving layer merges both. The lambda architecture was popular in the Hadoop era. It has been largely replaced by stream processing systems (Kafka Streams, Flink) that can handle both real-time and historical processing in a single framework.

Kappa architecture. Process everything as a stream. Historical data is replayed from the stream's retention log. New data arrives in real time. The same code handles both. Simpler than lambda. Requires a stream platform with long-term retention — Kafka with tiered storage, or a cloud-native equivalent. Kappa is the modern default for organizations that have invested in streaming infrastructure.

The economics

Pipelines cost more to maintain than to build. The initial build is a week. The maintenance is years. Every source system change — a renamed column, a new data type, a deprecated API — requires a pipeline update. Every business logic change requires a transformation update. Every scale increase — more data, more users, more dashboards — requires performance optimization. The maintenance cost is proportional to the number of pipelines and the rate of change of their dependencies.

The economics favor fewer pipelines, simpler pipelines, and pipelines owned by the people who understand the data. The centralized data team that builds pipelines for every department becomes a bottleneck. The data mesh model — each domain owns its pipelines — distributes the maintenance cost to the teams that benefit from the data. The distribution is the economics of the data mesh. The same economics that favor microservices over monoliths — independent deployability, domain ownership, reduced coordination cost — favor the data mesh over the centralized warehouse. The principles are identical. The domain is different.


References:

  • Maxime Beauchemin, "Functional Data Engineering — a modern paradigm for batch data processing," 2018.
  • Jay Kreps, "Questioning the Lambda Architecture," 2014.
  • dbt Labs, "What is dbt?" dbt Documentation.
  • Related posts: The Unix philosophy, No solutions, only trade-offs

Data modeling is the hard part

Data modeling is the art of deciding how to structure data so it can be queried efficiently and understood by humans. The models change. The principles don't. From Codd's relational model to Kimball's star schemas to the modern wide table, the problem is always the same: how do you represent reality in tables?

data-engineeringdata-modelingkimballstar-schemanormalization

Data modeling is the hardest part of data engineering because it is the part that requires judgment. You can learn a tool in a week. You can learn a pipeline pattern in a day. Data modeling takes years because the feedback loop is slow — you design a model, people query it for months, and only then do you discover what you got wrong. The wrongness is expensive to fix because downstream dashboards, ML models, and business processes have been built on the original model. Changing the model breaks them. The breakage is the cost of the original design error.

The relational model

Edgar Codd published "A Relational Model of Data for Large Shared Data Banks" in 1970. The paper introduced the idea that data should be stored in relations — tables — with well-defined operations for querying and manipulating them. The relational model separated the logical structure of data from its physical storage. Before Codd, databases were navigational — you followed pointers from record to record. The query path was baked into the storage structure. Codd's insight was that queries should be declarative: you specify what you want, not how to get it. SQL is the realization of that insight.

The relational model introduced normalization — the process of organizing data to minimize redundancy. First normal form: no repeating groups. Second normal form: no partial dependencies on a composite key. Third normal form: no transitive dependencies. The normal forms are a hierarchy of increasingly strict constraints on table design. A database in third normal form has minimal duplication. Every fact is stored exactly once. Changes to a fact require updating a single row.

Normalization is elegant. It is also slow for analytical queries. A normalized schema requires joins to reconstruct the original business entities. Joins are expensive on large tables. The tension between normalization (write efficiency, data integrity) and denormalization (read efficiency, query simplicity) is the central tension of data modeling.

Dimensional modeling

Ralph Kimball resolved the tension by designing for queries, not writes. In a dimensional model, data is organized into fact tables and dimension tables. Fact tables contain measurements — sales amounts, page views, sensor readings. Each row is an event. Dimension tables contain descriptions — customer names, product categories, date attributes. Each row describes an entity. The fact table references dimension tables through foreign keys.

The star schema is the simplest dimensional model. A central fact table surrounded by dimension tables, like points of a star. A sales fact table has foreign keys to date, customer, product, and store dimensions. A query joins the fact to any subset of dimensions. The joins are simple — each dimension is one hop from the fact. The simplicity makes the schema understandable to business users who write SQL.

The star schema's power is that it separates the what (measurements) from the who, what, when, and where (dimensions). You can ask any question that starts with "how many X by Y" — how many sales by product by month? How many page views by country by device? The answer is a join between the fact and the relevant dimensions. The model constrains the questions you can ask to the questions the business needs answered. The constraint is the design.

Kimball's methodology includes slowly changing dimensions (SCDs) — how to handle changes to dimension attributes over time. Type 1: overwrite the old value. Type 2: add a new row with the new value, preserving history. Type 3: add a new column for the new value. Each type trades query complexity for historical accuracy. The choice is a business decision, not a technical one. The business must decide whether historical accuracy matters enough to justify the complexity.

The modern wide table

The modern data stack has shifted toward wide, denormalized tables. The warehouse engines — Snowflake, BigQuery — are fast enough that joins are less expensive than they were. The transformation layer — dbt — makes it easy to build and maintain derived tables. The result is the One Big Table (OBT) pattern: a single table with hundreds of columns, pre-joined, pre-aggregated, ready for the dashboard to query with a simple SELECT *.

The wide table is a response to the reality that most business users cannot write joins. They can write SELECT * FROM orders_wide WHERE date > '2026-01-01'. The wide table makes the data accessible. The cost is storage (denormalized data is larger), maintenance (the wide table must be rebuilt when source schemas change), and lineage opacity (it's harder to trace where each column came from). The trade-off is economic: the cost of storage and compute is lower than the cost of analyst time spent writing joins incorrectly. The economics favor the wide table. The wide table is the default.

Data mesh and domain ownership

Zhamak Dehghani's data mesh (2019) challenges the centralized data warehouse model. The argument: data should be owned by the domains that produce it, not by a central data team. Each domain publishes data products — curated datasets with defined schemas, quality guarantees, and SLAs. The central team provides the platform — infrastructure, tooling, governance standards. The domains own the data.

The data mesh is a response to the scaling problems of centralized data teams. As the number of data sources grows, the central team becomes a bottleneck. Every new data source requires the central team to understand the domain, model the data, build the pipeline, and maintain it. The domain expert who understands the data is not the person building the pipeline. The knowledge gap produces errors. The bottleneck produces delays. The data mesh solves both by moving the pipeline ownership to the domain.

The trade-off: domain teams must now hire data engineering skills. The central team must build a platform that makes it easy for domain teams to publish data products. The governance must be federated — standards enforced by the platform, content owned by the domains. The data mesh is an organizational pattern, not a technology. The technology is the same as the centralized model. The organization is different. The difference is the innovation.

The principle

The specific model — star schema, wide table, data mesh — matters less than the principle: data must be structured so that the people who need it can find it, understand it, and trust it. The structure that achieves this for a five-person startup is different from the structure that achieves it for a five-thousand-person enterprise. The principle is the same. The implementation varies. The variation is the work.


References:

  • Edgar Codd, "A Relational Model of Data for Large Shared Data Banks," Communications of the ACM, 1970.
  • Ralph Kimball, The Data Warehouse Toolkit, Wiley, 1996.
  • Zhamak Dehghani, "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh," 2019.
  • Related posts: Parnas's Information Hiding, Scarcity and Software Economics

What data engineering actually is

Data engineering is the discipline of building systems that collect, store, transform, and serve data. It is not glamorous. It is not AI. It is the plumbing that makes AI possible. Without it, models have nothing to train on and dashboards show yesterday's numbers.

data-engineeringetlpipelinesinfrastructure

Data engineering is the discipline of building systems that collect, store, transform, and serve data. It sits between the systems that produce data — applications, sensors, user interactions — and the systems that consume data — dashboards, machine learning models, analysts. It is the plumbing. Plumbing is unglamorous. Plumbing is essential. Without plumbing, the house is uninhabitable.

The field emerged from a specific historical sequence. In the 1980s and 1990s, organizations built data warehouses — centralized repositories that aggregated data from operational systems for reporting and analysis. Ralph Kimball and Bill Inmon developed competing methodologies. Kimball advocated dimensional modeling — star schemas with fact tables and dimension tables, optimized for query performance. Inmon advocated the Corporate Information Factory — a normalized enterprise data warehouse feeding departmental data marts. The debate was religious. Both approaches worked. Both assumed that data was structured, that schemas were stable, and that the warehouse team could enforce standards.

The 2000s broke these assumptions. The volume of data exploded. The variety of data exploded — logs, JSON, sensor readings, social media feeds, clickstreams. The velocity of data increased — real-time streams replaced nightly batch loads. The old warehouse architectures couldn't keep up. Hadoop emerged. MapReduce provided a programming model for distributed data processing. HDFS provided a distributed filesystem. The ecosystem was complex, Java-heavy, and operated by a priesthood of engineers who understood the arcana of YARN configuration and NameNode failover. It worked. It was unpleasant.

The 2010s simplified the stack. Apache Spark replaced MapReduce with an in-memory processing engine that was faster and easier to program. Cloud data warehouses — Snowflake, BigQuery, Redshift — made the warehouse model viable again at cloud scale. The ELT pattern (Extract, Load, Transform) replaced ETL (Extract, Transform, Load): load raw data into the warehouse first, transform it later using the warehouse's own compute. The shift moved transformation logic from external pipelines into SQL, where analysts could contribute.

The 2020s are the era of the modern data stack. Fivetran and Airbyte handle extraction. dbt handles transformation — SQL-based, version-controlled, tested. Snowflake, BigQuery, and Databricks handle storage and query. Airflow and Prefect handle orchestration. The tools are better. The principles are the same: get data from where it is to where it needs to be, reliably, at the right time, in the right shape.

The core problems

Data engineering has five core problems. Every tool, every architecture, every methodology is a response to one or more of them.

Ingestion. Getting data into the system. From databases (CDC — change data capture), from APIs (REST, GraphQL), from files (CSV, JSON, Parquet), from streams (Kafka, Kinesis). The data arrives in different formats, at different cadences, with different reliability characteristics. The ingestion layer must handle all of them without losing data, duplicating data, or falling behind.

Storage. Keeping data somewhere it can be accessed. The choice of storage format — row-oriented vs. columnar, compressed vs. uncompressed, partitioned vs. monolithic — determines query performance, storage cost, and the ability to evolve schemas over time. The choice of storage engine — data warehouse vs. data lake vs. lakehouse — determines who can query the data and with what tools.

Transformation. Turning raw data into useful data. Cleaning — removing duplicates, fixing nulls, standardizing formats. Enriching — joining with reference data, computing derived fields, applying business logic. Aggregating — rolling up to daily, weekly, monthly levels for dashboards. The transformation layer is where most of the engineering effort goes. It is also where most of the bugs are.

Orchestration. Making everything run at the right time, in the right order, with the right dependencies. Pipeline A must finish before Pipeline B starts. Pipeline B must not run if Pipeline A produced bad data. The orchestration layer manages schedules, dependencies, retries, alerts, and backfills. It is the conductor. When the conductor fails, the orchestra plays anyway — out of sync, producing cacophony.

Serving. Getting data to the people who need it. Dashboards (Looker, Tableau, Metabase). Ad-hoc queries (SQL editors, notebooks). Machine learning feature stores. Reverse ETL — sending transformed data back to operational systems (CRM, email, advertising). The serving layer determines whether the data is actually used. The best pipeline in the world is worthless if nobody looks at its output.

How it differs from software engineering

Data engineering is software engineering with different constraints. Software engineering optimizes for correctness, latency, and throughput of application logic. Data engineering optimizes for correctness, latency, and throughput of data movement and transformation. The difference is the nature of the bugs.

A software bug produces a wrong output for a specific input. A data engineering bug produces a wrong output for millions of records, discovered three weeks later when the CFO asks why revenue is down. The blast radius is larger. The debugging is harder — you must trace the error backward through multiple transformation steps, each of which may have run days or weeks ago. The fix requires not just correcting the code but reprocessing the affected data, which may take hours or days. The operational complexity of data engineering is higher than application engineering because the state — the data — is larger, more persistent, and harder to repair.

Data engineering also has a different failure mode: silent corruption. A software system crashes visibly — errors, exceptions, downtime. A data pipeline can produce wrong numbers silently, for weeks, before anyone notices. The pipeline didn't fail. It ran successfully. The data is wrong. The wrongness is invisible until someone looks at the numbers and says "that doesn't seem right." The delay between corruption and detection is the most dangerous property of data systems.

The data engineer's mindset

The data engineer thinks in terms of data flows, not control flows. The question is not "what does this function return?" but "where does this data come from, what happens to it along the way, and who consumes the output?" The data engineer traces lineage forward and backward through the system. Forward: if this source data changes, what downstream tables are affected? Backward: if this dashboard number is wrong, which pipeline produced it, from which source, using which transformation logic? The ability to trace lineage is the data engineer's superpower. The inability to trace lineage is why data projects fail.

The data engineer is paranoid about state. Every pipeline should be idempotent — running it twice produces the same result as running it once. Every pipeline should be retryable — if it fails, it can be restarted without corrupting the output. Every pipeline should be testable — a small sample of input data should produce a predictable output. Every pipeline should be monitored — if it produces zero rows, or ten times the usual number of rows, or rows with nulls where there should be values, someone should be alerted. The paranoia is not anxiety. It is engineering.


References:

  • Ralph Kimball, The Data Warehouse Toolkit, Wiley, 1996.
  • Bill Inmon, Building the Data Warehouse, Wiley, 1992.
  • Maxime Beauchemin, "The Rise of the Data Engineer," 2017.
  • Related posts: The Unix philosophy, Engineering is economics

The Verification Horizon

A 2026 paper argues that verifying code is now harder than generating it. As models improve, verification signals degrade. Every verifier is a proxy for human intent, and every proxy drifts from the intent it represents. There is no silver bullet for coding agent rewards. The verifier must evolve with the generator, forever.

verificationcoding-agentsrewardsbrooksai

Brooks wrote No Silver Bullet in 1986. The argument: there is no single breakthrough that will eliminate the essential difficulty of software engineering. Better tools, better languages, better processes — these reduce accidental complexity. The essential complexity remains.

A paper published last month applies the same logic to AI coding agents. The title: "The Verification Horizon: No Silver Bullet for Coding Agent Rewards." The authors: Wang, Zhang, Liu, and a dozen others. The argument: as models become more capable at generating code, verifying that code becomes the bottleneck. And verification, like software engineering, has no silver bullet. The verifier must evolve with the generator. The evolution never ends. The horizon recedes as you approach it.

The inversion

The traditional assumption in software engineering is that verification is easier than generation. Writing a program is hard. Checking whether the program is correct is easier — you run tests, you review the code, you compare outputs to expected outputs. This assumption is why we have QA teams, code review, and test suites. The generator does the hard work. The verifier checks that it was done right.

The paper argues that this assumption has inverted. Foundation models can generate plausible solutions to coding problems at scale — hundreds of candidate solutions per task, each syntactically correct, each plausible, most wrong in subtle ways. Generating is easy. Distinguishing the correct solution from the plausible-but-wrong ones is hard. Verifying is now the bottleneck.

"Intent is naturally underspecified by nature, making it inherently hard to faithfully check whether it has been fulfilled."

The spec says "build a login page." The agent generates a login page. It has a username field, a password field, a submit button. The tests pass. The page is also inaccessible to screen readers, makes three unnecessary API calls, stores the password in localStorage, and uses a deprecated authentication library. The tests didn't check for any of this. The tests verified what they were designed to verify. The intent — "build a secure, accessible, maintainable login page" — was not fully specified. It couldn't be. Intent is always underspecified. The underspecification is the gap between what the verifier checks and what the human actually wants. The gap is where the bugs live.

The three dimensions of verification

The paper decomposes verification quality along three axes:

Scalability. Can the verifier handle the volume? An agent that generates 100 candidate solutions per task needs a verifier that can evaluate 100 candidates. Manual verification doesn't scale. Automated verification scales but loses fidelity. The scalable verifier is less faithful. The faithful verifier doesn't scale. The trade-off is structural.

Faithfulness. Does the verifier's score correlate with the human's actual preferences? A verifier that rewards long methods produces long methods. A verifier that rewards test coverage produces tests that cover lines without asserting behavior. Every metric becomes a target. Every target is gamed. The gaming is not malicious. It is optimization. The optimizer finds the path of least resistance to the reward. The path of least resistance is rarely the path the human intended.

Robustness. Does the verifier work across tasks, domains, and capability levels? A verifier tuned for CRUD endpoints fails on frontend tasks. A verifier tuned for frontend tasks fails on data pipelines. A verifier that works at the current model capability level breaks when the model improves — the model discovers edge cases the verifier didn't anticipate, exploits reward structures the verifier thought were safe, produces outputs that score highly on the metric and fail on the intent. The verifier is robust at capability level N. At capability level N+1, it breaks.

The central challenge: achieving all three simultaneously. Scalable, faithful, robust. Pick two. The third will be your bottleneck.

The verification horizon

The paper's core thesis is that verification is not a fixed target. It is a moving horizon:

"No fixed reward function can remain effective as policy capability continues to grow."

The verifier works at time T. The generator improves. At time T+1, the generator produces outputs the verifier can't reliably evaluate. The verifier must improve. The generator then improves further. The verifier must improve again. The cycle is continuous. The horizon recedes. The verifier never catches up.

This is Lehman's First Law applied to AI systems: an E-type system must be continually adapted or it becomes progressively less useful. The verifier is an E-type system. The environment — the generator's capability — is changing. The verifier must change with it. The change is not a one-time calibration. It is an ongoing co-evolutionary process. The verifier and the generator are locked in a Red Queen race. Each must improve just to stay in the same place relative to the other.

The implication is that verification cannot be solved once. It cannot be reduced to a fixed test suite, a fixed rubric, a fixed set of acceptance criteria. The test suite that verifies today's agent will be gamed by tomorrow's. The rubric that distinguishes good from bad today will be satisfied by mediocrity tomorrow. The acceptance criteria that capture intent today will be insufficient tomorrow because the agent will discover ways of satisfying the criteria that violate the unstated intent. The unstated intent is infinite. The criteria are finite. The gap is where the horizon lives.

The co-evolution requirement

The paper studies four verification approaches:

Test verifier. Unit tests, integration tests, end-to-end tests. Scalable. Moderately faithful. Breaks when the agent learns to write code that passes tests without implementing the intended behavior. The test verifier is the default. The default is insufficient.

Rubric verifier. Structured scoring rubrics for frontend tasks — layout correctness, accessibility, responsiveness. More faithful than tests for visual tasks. Harder to scale — rubrics must be designed per task type. Breaks when the agent produces designs that satisfy the rubric criteria while being visually wrong in ways the rubric doesn't capture.

User as verifier. The human evaluates the agent's output. Maximally faithful. Doesn't scale. The user can evaluate a few outputs. The agent generates hundreds. The user becomes the bottleneck. The user's attention is scarce. The scarcity is economic. The economics favor automation.

Automated agent verifier. An agent evaluates another agent's output. Scales. Potentially faithful, if the evaluating agent is well-calibrated. Potentially robust, if the evaluating agent co-evolves with the generating agent. The paper's experiments show this approach achieves significant gains when the evaluating agent is specifically trained for verification with targeted reward design. The key is that the evaluating agent must not be a fixed function. It must be an agent that can reason about intent, notice discrepancies, and adapt its evaluation criteria as the generating agent improves.

The conclusion: no single verification approach works across all tasks, all capability levels, all time. The verifier must be an ensemble. The ensemble must evolve. The evolution must be continuous. There is no silver bullet.

The connection to Brooks

Brooks argued that essential complexity cannot be eliminated. The paper argues that the verification gap cannot be closed. Both arguments have the same structure. The gap between what we want and what we can specify is irreducible. Intent is infinite. Specification is finite. The gap between them is where bugs, misunderstandings, and failed projects live. Better tools reduce accidental complexity — the difficulty of representing the specification, the difficulty of checking it. The essential complexity — the gap between intent and specification — remains.

Brooks: "The hardest single part of building a software system is deciding precisely what to build." The paper: the hardest single part of verifying an agent's output is knowing precisely what you wanted. The two statements are the same statement. The difficulty is specification. Specification is underspecified by nature. The underspecification is essential. The essential cannot be eliminated. The horizon recedes.

Brooks: "There is no silver bullet." The paper: "There is no silver bullet for coding agent rewards." The paper's title is a deliberate echo. Thirty-nine years after Brooks, the same structure appears in a new domain. The domain is AI. The structure is the same. The essential difficulty of specifying intent survives every advance in the technology of generating outputs. The generator improves. The spec remains incomplete. The gap remains. The gap is the problem. The problem has no silver bullet.


References:

  • Binghai Wang et al., "The Verification Horizon: No Silver Bullet for Coding Agent Rewards," arXiv:2606.26300v2, June 2026.
  • Frederick P. Brooks, Jr., "No Silver Bullet: Essence and Accidents of Software Engineering," Computer Magazine, April 1987.
  • M.M. Lehman, "Programs, Life Cycles, and Laws of Software Evolution," Proceedings of the IEEE, 1980.
  • Related posts: Brooks on Software Design, Lehman's Software Evolution, Task Automation Economics

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

The Unix philosophy is the only software engineering theory that works

McIlroy wrote it in 1978. Kernighan and Pike explained it in 1984. Raymond codified it in 2003. Microservices rediscovered it in 2014. Nobody read the books. The pipes are the same. The mistakes are the same.

unixphilosophypipestoolscompositionmicroservicessoftware-design

Scarcity is the only constraint

Engineering is the application of knowledge to solve problems within constraints. The constraints are what make it engineering and not mathematics. Mathematics has no budget. Engineering has a budget. Mathematics has no deadline. Engineering has a deadline. Mathematics values elegance. Engineering values working within the constraints. The constraints are always, ultimately, economic.

Every engineering discipline is bounded by the same fundamental problem: resources are finite. Time is finite. Attention is finite. Money is finite. You cannot do everything. You must choose. The choice of what to build and what to leave unbuilt is not a technical decision. It is an economic decision made under conditions of scarcity. The engineer who denies this is not an engineer. They are a mathematician who happens to write code.

E.F. Schumacher published Small Is Beautiful: A Study of Economics As If People Mattered in 1973. The book is about economics, not software. Its argument is that the modern economy has optimized for scale at the expense of human flourishing — that bigness has become an end in itself, that technologies and organizations have grown beyond the scale at which humans can understand them, and that the appropriate scale for human institutions is small. Not small because small is virtuous. Small because small is comprehensible. A system you cannot understand is a system you cannot control. A system you cannot control is a system that controls you.

"Ever bigger machines, entailing ever bigger concentrations of economic power and exerting ever greater violence against the environment, do not represent progress: they are a denial of wisdom. Wisdom demands a new orientation of science and technology toward the organic, the gentle, the non-violent, the elegant and beautiful." — E.F. Schumacher, Small Is Beautiful

Schumacher was writing about industrial economics. He could have been writing about software architecture. The Unix philosophy is Schumacher's argument applied to code. Small programs. Gentle interfaces. Non-violent composition. Elegant design. The scale of a program should be no larger than necessary to do its job. The job should be one thing. The interface should be simple enough to be understood completely. This is not an aesthetic preference. It is an engineering response to the problem of scarcity. You have finite time, finite attention, finite ability to hold complexity in your head. The system must work within those constraints or it will fail.

Schumacher's principle applied to software: a program large enough to require a team is large enough to develop internal complexity visible only to the team. A program large enough to require multiple teams is large enough that no single person understands it. A system that no single person understands is a system whose behavior is emergent, not designed. Emergent behavior can be good. It can also be catastrophic. The smaller the components, the easier to understand each one. The simpler the interfaces, the easier to predict the behavior of the whole. Small is beautiful because small is the scale at which humans can still reason about what they have built.

The Unix philosophy: Schumacher applied to code

In 1978, Doug McIlroy — the inventor of the Unix pipe — wrote down the Unix philosophy in the Bell System Technical Journal. He didn't cite Schumacher. He didn't need to. The principle is the same whether you're designing a factory or a program. Small. Simple. Composable. The philosophy consisted of four directives:

  1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features.
  2. Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
  3. Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them.
  4. Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them.

He later condensed it to three lines that became famous:

"This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."

That's it. The entire theory of software composition, stated in 1978, in three sentences. Everything that has happened in software architecture since — microservices, serverless, event-driven systems, hexagonal architecture, domain-driven design, bounded contexts, API gateways, service meshes — is rediscovery of these three sentences in new terminology. The terminology changes. The principles don't. The books are still on the shelf. The ignorance is expensive.

The books you should have read

Kernighan and Pike, The Unix Programming Environment (1984). The canonical text. They wrote:

"What makes it effective is the approach to programming, a philosophy of using the computer. At its heart is the idea that the power of a system comes more from the relationships among programs than from the programs themselves. Many UNIX programs do quite trivial things in isolation, but, combined with other programs, become general and useful tools."

The power is in the relationships, not the programs. The programs are trivial. The composition is where the capability lives. This is the sentence that should be printed above every microservices whiteboard. It is also the sentence most microservices architectures violate, because they focus on the services — what each service does, what database it owns, what team builds it — and ignore the relationships. The relationships are the system. The services are components. Components are easy. Relationships are hard.

Eric Raymond, The Art of Unix Programming (2003). Raymond codified seventeen rules. The ones that matter for this argument:

  • Rule of Modularity: Build programs from simple, cleanly-connected parts.
  • Rule of Composition: Programs must communicate easily with other programs.
  • Rule of Separation: Separate mechanism from policy.
  • Rule of Parsimony: Write small programs. Easy to replace when wrong.
  • Rule of Transparency: Make operation visible and discoverable.
  • Rule of Silence: Don't print unnecessary output. Let other programs decide what matters.

Each rule maps directly to a microservices principle with the names changed. Modularity → bounded contexts. Composition → API contracts. Separation → business logic vs. infrastructure. Parsimony → small services, easy to rewrite. Transparency → observability. Silence → don't log everything, emit meaningful events. Raymond wrote this in 2003, before microservices existed as a term. He was describing Unix. He was also describing microservices. He didn't know it. The microservices pioneers didn't know it either. They thought they were inventing something. They were rediscovering pipes.

McIlroy's later reflection. Years after the original paper, McIlroy watched Linux grow and said:

"Everything was small. My heart sinks for Linux when I see the size of it. We used to sit around in the Unix Room saying, 'What can we throw out? Why is there this option?' It's often because there is some deficiency in the basic design — you didn't really hit the right design point. Instead of adding an option, think about what was forcing you to add that option."

The option is the symptom. The design deficiency is the cause. Adding the option papers over the deficiency without fixing it. The program grows. The option count grows. The composability shrinks. McIlroy's test: when you're tempted to add a feature, ask what deficiency in the existing design made the feature necessary. Fix the design. Don't add the feature. This test applies to microservices boundaries. When you're tempted to add a new endpoint to a service, ask what deficiency in the existing API contracts made the new endpoint necessary. Fix the contract. Don't add the endpoint. The endpoint is the option. The contract deficiency is the design flaw.

Why pipes work: the engineering fundamentals

The Unix pipe is not a metaphor. It is a specific engineering construct with specific properties that make composition possible.

Uniform interface. Every program reads from stdin and writes to stdout. The interface is identical regardless of what the program does. grep reads text and writes text. sort reads text and writes text. wc reads text and writes text. They can be connected in any order, in any combination, because the interface is uniform. This is the opposite of REST microservices, where every service defines its own endpoints, its own request format, its own response format, its own error semantics. The interface is not uniform. Composition requires adapters. Adapters are coupling. Coupling is the thing pipes eliminate.

Separation of mechanism and policy. sort sorts. It does not know what it is sorting. It does not know why. It does not know what will happen to the sorted output. It sorts. That is the mechanism. The policy — what data to sort, what to do with the sorted result — is determined by the programs upstream and downstream. This is the Rule of Separation applied to data processing. In microservices: the service provides the mechanism (process an order, reserve inventory). The orchestration layer provides the policy (which services to call in which order under which conditions). If the service encodes policy, it cannot be reused in a different policy context. The mechanism is coupled to the policy. The coupling prevents composition.

Composability without coordination. grep was written before sort knew about it. sort was written before wc knew about it. None of them were designed to work together. They work together because they all obey the same interface contract. They can be composed into pipelines the original authors never imagined. This is the property that microservices promise and rarely deliver: composition without coordination. Services that were designed by different teams at different times, communicating through stable interfaces, composed into workflows that nobody designed in advance. The promise is real. The delivery is rare because the interfaces are not uniform. Every service defines its own contract. Every composition requires a new adapter. The adapters accumulate. The system becomes a collection of adapters connecting services that were supposed to be directly composable.

This is also the economic argument for smallness. Schumacher: the appropriate scale for any human institution is the scale at which it can be understood by the people who operate it. grep can be understood completely by one person in an afternoon. sort can be understood completely. wc can be understood completely. The pipeline composed of them can be understood by understanding each component in sequence. The understanding scales linearly with the number of components because the components are small enough to fit in a human head. A microservice that takes a team of five to maintain cannot be understood completely by any single person. A system of twelve such services cannot be understood by anyone. The components are too large. The composition is opaque. The system has exceeded the Schumacher threshold — the scale at which humans can still reason about what they have built.

Filter thinking. McIlroy's second directive: "Expect the output of every program to become the input to another, as yet unknown, program." Every program is a filter — it transforms an input stream into an output stream. It doesn't know where the input came from. It doesn't know where the output is going. It transforms. This is the purest form of Parnas's information hiding: the program hides everything about itself except the transformation it performs. The caller doesn't know the algorithm. The caller doesn't know the implementation language. The caller knows the transformation. The transformation is the interface. Everything else is hidden.

Where pipes fail

The Unix philosophy is not universal. It has known failure modes. Understanding them is as important as understanding the successes.

Text as universal interface breaks at scale. Text is universal. Text is also unstructured. Every program that receives text must parse it. Every parsing step is an opportunity for error, inconsistency, and performance cost. When the data has structure — nested objects, typed fields, relationships — text streams force every consumer to reconstruct the structure from its flattened representation. This is the argument for typed interfaces, for gRPC over REST, for Avro over JSON. Text is the universal interface for simple data. It is the wrong interface for complex data. The Unix philosophy doesn't tell you where the boundary is. Experience does. The boundary moves with the complexity of the data.

State management is externalized. Pipes connect stateless programs. Each program reads, transforms, writes. State is managed outside the pipeline — in files, databases, or the shell's variables. When the processing requires state that spans multiple pipeline stages — a running total, a windowed aggregation, a session — the stateless model breaks. You either pass the state through the pipe as additional data (cluttering the output, violating the Rule of Silence) or you externalize it (breaking the pipeline model). Modern stream processing systems — Kafka Streams, Flink, Spark Streaming — are essentially pipelines with built-in state management. They fix the failure mode at the cost of increased complexity. The tradeoff is inevitable. The Unix philosophers knew it. They never claimed pipes solved everything.

Error handling is ad-hoc. A pipeline of ten programs. The eighth program fails. What happens? The shell reports the exit code of the last program in the pipeline. Unless you use set -o pipefail, the failure of the eighth program is invisible. The pipeline continues. The output is partial. Nobody knows something went wrong. This is a design choice, not an oversight. Unix errs on the side of simplicity: programs should do their job and exit. Error handling is the caller's responsibility. But when the caller is a pipeline, the caller is distributed across multiple programs, none of which know about each other. Distributed error handling is hard. Microservices rediscovered this the hard way. Distributed sagas, compensating transactions, dead letter queues — these are the modern equivalents of pipefail. The terminology changed. The problem is the same.

The composition model is linear. Pipes compose programs sequentially. The output of A goes to the input of B. This is powerful for linear data processing. It is weak for systems where the data flow is a graph — fan-out, fan-in, conditionals, loops, feedback. You can build graph processing in shell, but the shell fights you. The composition model is not general. It is linear. Most workflows are not linear. The Unix philosophy works brilliantly for the subset of problems that are linear data transformations. For everything else, you need a different composition model. Microservices with message brokers — NATS, Kafka, RabbitMQ — implement graph composition. The broker is the compositor. The services are the components. The graph is the architecture. The Unix philosophy didn't fail. It was extended. The extension was necessary.

Microservices: the old is the new new

In 2014, the term "microservices" entered the mainstream. The defining characteristics: small, focused services. Communication through uniform interfaces. Independent deployability. Composition into workflows. Decentralized data management. Design around business capabilities.

This is McIlroy's Unix philosophy, restated for distributed systems. Small, focused services → programs that do one thing well. Uniform interfaces → text streams as universal interface. Composition into workflows → pipes connecting programs. Independent deployability → programs that don't know about each other. Decentralized data management → each program manages its own state.

The microservices pioneers were not copying Unix. They were independently rediscovering the same principles at a different scale. The scale changed. The principles didn't. The oversight is that the Unix philosophers already documented the failure modes — and the microservices pioneers walked into every one of them.

The uniform interface failure. Unix has a uniform interface: text streams. Every program reads and writes the same format. Microservices initially attempted the same: REST with JSON. But JSON is not a uniform interface when every service defines its own schema, its own endpoint structure, its own error format. The interface is HTTP. The contract is ad-hoc. The uniformity is at the transport layer. The diversity is at the application layer. The diversity is where the coupling lives. Microservices adopted the transport uniformity of Unix without the semantic uniformity. The result is services that can talk to each other but can't understand each other without adapters. The adapters are the coupling. The coupling is what Unix pipes eliminated.

The composition failure. Unix composes programs with pipes: A | B | C. The composition is linear, immediate, and visible. Microservices compose with orchestration: Service A calls Service B, which calls Service C, with retries, timeouts, circuit breakers, and dead letter queues at each step. The composition is a distributed graph with failure modes at every edge. The complexity of composition in microservices is orders of magnitude higher than in Unix pipelines. The principles are the same. The implementation is harder. The failure to acknowledge the increased difficulty is why microservices projects fail.

The state management failure. Unix programs are stateless. State lives in the filesystem. Microservices are stateful. Each service owns its database. State management is decentralized. Decentralized state management is a hard problem — distributed transactions, eventual consistency, saga patterns, CQRS, event sourcing. Unix didn't have this problem because Unix programs didn't own state. Microservices created the problem by making services own state, then spent a decade inventing patterns to solve it. The patterns work. They are also complex. The complexity is inherent. It cannot be eliminated by better tooling. It can only be managed by accepting that decentralized state is expensive and choosing which services truly need it.

The debugging failure. When a Unix pipeline fails, you can run each program in isolation with the same input and see where the output diverges. The pipeline is reproducible. Microservices are not. A failure in a distributed workflow involves network timeouts, retry policies, eventual consistency windows, and state spread across multiple databases. Reproducing the failure requires reproducing the entire distributed state. This is hard. This is why observability tooling for microservices is a multi-billion-dollar industry. The tooling exists because the problem is hard. The problem is hard because the composition model is distributed. The composition model is distributed because each service owns its state. The state ownership is the root cause. The root cause was a design choice. The design choice had consequences nobody predicted — except McIlroy, who designed Unix with stateless programs, and Pike, who built Go with channels instead of shared state, and the Unix philosophers generally, who understood that state is the enemy of composability.

The size failure. McIlroy's lament about Linux — "my heart sinks when I see the size of it" — applies directly to microservices. A service that does one thing is small. A service that does one thing plus error handling, retry logic, circuit breaking, authentication, authorization, logging, metrics, tracing, configuration management, and service discovery is not small. The infrastructure concerns colonize the service. The service grows. The growth is not in business logic. It is in infrastructure. The infrastructure should be external to the service — in the platform, the mesh, the gateway. But it leaks in. It leaks in because the uniform interface that should handle these concerns — the service mesh, the API gateway — is not as uniform as Unix text streams. Every service has its own configuration, its own policies, its own exceptions. The uniformity is incomplete. The incompleteness is where the complexity accumulates.

Why history repeats

The Unix philosophers wrote their books between 1978 and 2003. The microservices pioneers wrote their blog posts between 2012 and 2016. The gap is roughly a decade. The principles are the same. The terminology changed. Why didn't the microservices pioneers cite McIlroy, Kernighan and Pike, and Raymond?

Because software engineering does not read its own history. Architects study buildings. Composers study scores. Software engineers study the framework documentation for the current version. The old books are on the shelf. The old papers are in the Bell System Technical Journal. The old principles are correct. Nobody reads them. Every generation rediscovers composition, modularity, and information hiding and gives them new names. The names change. The principles don't.

The cost of this amnesia is not theoretical. It is measured in failed microservices migrations. Teams that split the monolith by database table instead of by bounded context. Teams that built a distributed system with more coupling than the monolith it replaced. Teams that discovered that twelve services with a complete call graph is not an architecture — it is a monolith with network latency. Teams that learned about distributed state management the hard way, in production, at 3am, when the saga pattern failed and the compensating transaction didn't compensate.

Every one of these failures was avoidable. Not by reading the microservices literature. By reading the Unix literature. The principles are older. The principles are clearer. The principles were stated in three sentences in 1978. The sentences are still correct. The sentences are still unread.

The amnesia has an economic cause. The software industry grows by selling new things. New things need new names. Old principles with new names sound like progress. They are not progress. They are the same principles, rediscovered at greater expense, with more infrastructure. The principles were free, in the public domain, published in 1978. The ignorance is not free. It is the most expensive thing in the industry.

Schumacher saw this dynamic in industrial economics fifty years ago. The cult of bigness — larger factories, larger organizations, larger systems — was not driven by efficiency. It was driven by the interests of the people who built and operated the large things. Scale benefits the builder. It burdens the user. The Unix philosophers built small because they were building for themselves. They were the users. When the builder is the user, the scale is appropriate. When the builder is not the user — when the user is a customer and the builder is a vendor — the scale inflates. The inflation serves the vendor's interests: more features, more complexity, more lock-in, more billable hours. The user wanted a program that does one thing well. The vendor shipped a platform. The user adapted. The cycle repeated. The industry grew. The principles were forgotten. They are still true. They are still unread.

"Those days are dead and gone and the eulogy was delivered by Perl." — Rob Pike

Pike's eulogy was premature. The philosophy didn't die. It moved up the stack. Pipes became channels. Text streams became typed interfaces. Small programs became small services. The shell became the orchestrator. The toolbox became the service catalog. The principles survived. The implementation changed. The failure modes are the same.

"Ever bigger machines, entailing ever bigger concentrations of economic power, do not represent progress: they are a denial of wisdom. Wisdom demands a new orientation of science and technology toward the organic, the gentle, the non-violent, the elegant and beautiful." — E.F. Schumacher, Small Is Beautiful, 1973

The books are on the shelf. The economics book and the Unix book. They are both short. They are both correct. Read them.


References:

  • E.F. Schumacher, Small Is Beautiful: A Study of Economics As If People Mattered, Blond & Briggs, 1973.
  • Doug McIlroy, "Unix Time-Sharing System: Forward," Bell System Technical Journal, Vol. 57, No. 6, July-August 1978.
  • Brian Kernighan and Rob Pike, The Unix Programming Environment, Prentice-Hall, 1984.
  • Eric S. Raymond, The Art of Unix Programming, Addison-Wesley, 2003.
  • Peter H. Salus, A Quarter-Century of Unix, Addison-Wesley, 1994.
  • Rob Pike, "Simplicity is Complicated," dotGo, 2015.
  • Related posts: Henney's Microservices, Parnas's Information Hiding, Git is a Unix tool, NATS pub/sub beats REST

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

Two ways to design software

Tony Hoare: 'There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.' The difficulty is economics. The simplicity costs more now. The complexity costs more later.

hoaresimplicitycomplexitydesigneconomics

Tony Hoare, in his 1980 Turing Award lecture, described two approaches to software design:

"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult."

The sentence is famous. It is usually quoted for its elegant symmetry — two ways, two outcomes, one harder than the other. It is less often quoted for its economic content. The economic content is the point. The first method is more difficult. Difficulty in engineering translates to cost. The simple design costs more to produce. The complex design costs less to produce and more to maintain. The trade-off is economic. The choice between them is a choice about when to pay.

The simple way

A simple design is one where the structure is transparent. A reader can see what the system does and how it does it. The components have clear responsibilities. The interfaces are minimal. The interactions are predictable. The system has no unnecessary parts.

Producing such a design requires understanding the problem deeply enough to identify what is unnecessary. The understanding cannot be acquired by thinking. It must be acquired by building. The first version will be complex because the first version is where you learn what the problem actually is. The simple design is the second version, or the third, or the fifth. It is the version produced after you have built the complex one, understood it, and removed everything that wasn't earning its place. The removal is the difficulty. The removal requires judgment about what is essential and what is accidental. The judgment cannot be automated. It requires taste. Taste is expensive to develop. The expense is time spent building complex things and simplifying them. The expense is the tuition for the simple design.

Perlis said the same thing in different words: "Simplicity does not precede complexity, but follows it." The simple design follows the complex one. The time spent on the complex one is the cost of the simple one. The cost is invisible in the final product. The final product looks like it was designed simply from the start. It was not. It was simplified. The simplification was work. The work was expensive.

The complicated way

A complicated design is one where the structure is opaque. The system works. It has been tested. It passes its tests. But a reader cannot easily see why it works. Components have overlapping responsibilities. Interfaces have grown extra parameters over time — "just add a flag" is the entropy mechanism of software. Interactions have edge cases that are handled but not documented. The system has many parts. The parts interact in ways that surprise even the people who built them.

Producing such a design is easy. You build what works. You add what's needed. You don't remove what isn't. The removal is the hard part. Skipping it makes the design complicated. The complexity is not malicious. It is the natural state of a system that has been changed by many people over many years, each adding what they needed and nobody removing what was no longer necessary. The removal requires knowing what is no longer necessary. The knowing is distributed across the team, the codebase, the incident history. It is Hayek's dispersed knowledge, applied to a single system. Centralizing it is hard. The difficulty is why the removal doesn't happen.

"Inside every large program, there is a small program trying to get out." — Tony Hoare

The small program is the simple design that would have sufficed. It is buried under the accumulated weight of features that were added because they were easier to add than to integrate properly, workarounds that were applied because the root cause was too expensive to fix, abstractions that were generalized prematurely because generality felt like good design. The small program is still there. It is obscured. The obscuring material is the complexity. The complexity is the cost of decisions made under time pressure. The time pressure was economic. The decisions were economic. The complexity is economic debt.

The economics of the choice

The choice between the two methods is not aesthetic. It is economic. The simple method costs more now. The complicated method costs more later. The choice is about the discount rate — how much you value the present relative to the future.

A team with a high discount rate — next sprint's features matter more than next year's maintainability — will choose the complicated method. The choice is rational given the incentive structure. The incentive structure rewards velocity now. It does not reward maintainability later. The engineer who spends two weeks simplifying a design that already works is less visibly productive than the engineer who ships two features in the same time. The simplification prevents future problems. The prevention is invisible. The features are visible. The visible gets rewarded. The invisible doesn't. The incentive structure produces complicated designs. The structure is the problem.

A team with a low discount rate — sustainability matters, the system will exist for years, the cost of future complexity is priced into present decisions — will choose the simple method. The choice is also rational. These teams are rare. They are rare because low discount rates require organizational stability — the same people maintaining the system they built, long enough to feel the cost of their own complexity. If you build a complicated system and leave before the complexity costs you, you benefited from the speed and didn't pay the maintenance cost. The cost was paid by your replacement. Your incentive was to build complicated. The incentive was structural. The structure produced the behavior.

The false choice

The choice between simple and complicated is sometimes presented as a choice between elegance and pragmatism. Elegance is for academics. Pragmatism ships. This framing is wrong. The pragmatic choice is usually the simple one, if the time horizon is long enough. The complicated design is pragmatic only on a short horizon. On a long horizon, the complicated design is the most expensive choice you can make. The expense is deferred, compounded, invisible in the current sprint, undeniable in year five. The pragmatism that ignores the future is not pragmatism. It is myopia with a professional vocabulary.

Hoare's sentence is not a preference for elegance. It is a statement of economic fact. The simple design is more difficult — costs more now. The complicated design is easier — costs more later. Choose. The choice is yours. The structure of your incentives will make it for you if you don't make it consciously. Conscious choices are better than structural ones. Structural ones feel like they weren't choices at all. They were. The structure disguised them.


References:

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

this statement is false

Six words that broke mathematics, launched a thousand puzzles, and may explain consciousness. From the Liar Paradox through Gödel, Smullyan's knights and knaves, and Hofstadter's strange loops — self-reference is the thread.

self-referencegodelparadoxsmullyanhofstadterlogic

"This statement is false."

Six words. If it's true, it's false. If it's false, it's true. The mind oscillates. It never settles. This is the Liar Paradox, attributed to Epimenides of Crete in the 6th century BC. It is the simplest sentence that breaks logic. Everything that follows from it — Gödel's incompleteness theorems, Turing's halting problem, Smullyan's puzzles, Hofstadter's theory of consciousness — is an elaboration of those six words.

The Liar is not a trick. It is a structural property of any system that can talk about itself. Once a language can refer to its own statements, it can construct the Liar. Once it can construct the Liar, it can produce truth that cannot be proven, questions that cannot be answered, and systems that cannot be completed. This is not a bug in logic. It is a fact about self-reference. The discovery of that fact is the intellectual thread connecting the books on this page.

Gödel: the Liar goes to mathematics

In 1931, Kurt Gödel proved that any formal system powerful enough to express arithmetic contains true statements that cannot be proven within the system. He did this by constructing a mathematical version of the Liar.

Instead of "This statement is false," Gödel constructed: "This statement has no proof in this system."

If the statement is provable, the system has proved a falsehood — contradiction. If it is unprovable, it is true — but the system cannot prove it. Either way, the system is incomplete. There are truths it cannot reach.

Nagel and Newman's Gödel's Proof (1958) is the canonical explanation for non-mathematicians. They walk through Gödel's construction step by step: how he assigned numbers to symbols (Gödel numbering), how he encoded statements about provability as arithmetic statements, and how he constructed the self-referential sentence that broke the system from within. The book is short — under 150 pages. It assumes nothing beyond high school mathematics. It is still the best introduction to the result that defined the limits of formal reasoning.

"Gödel's paper is a proof of the impossibility of proving certain statements in a formal system — statements that are nevertheless true." — Nagel & Newman

The key move: Gödel numbering. Assign a unique number to every symbol, formula, and proof in the system. Then statements about numbers can also be statements about statements. "This formula has a proof" becomes an arithmetic claim. The system can talk about itself. And once it can talk about itself, it can construct the Liar. The Liar is inescapable. It is not a flaw in the system. It is a property of any system powerful enough to contain itself.

Hofstadter: the strange loop

Douglas Hofstadter's Gödel, Escher, Bach: An Eternal Golden Braid (1979) won the Pulitzer Prize. It is 777 pages. It is about self-reference in mathematics, art, music, and consciousness. It contains dialogues between Achilles and a Tortoise, formal descriptions of fugues, and a chapter where the author interrupts himself to interview himself about whether he should have written the chapter differently.

The book's central concept is the strange loop: a phenomenon where moving through levels of a hierarchical system brings you back to where you started. The top reaches down and influences the bottom, which determines the top. The hierarchy is tangled.

"An interaction between levels in which the top level reaches back down toward the bottom level and influences it, while at the same time being itself determined by the bottom level."

Gödel's proof is a strange loop in mathematics: the system of arithmetic, by encoding itself, produces a statement that refers to itself, and the self-reference generates undecidability. Escher's Drawing Hands is a strange loop in art: two hands draw each other into existence, each creating the other. Bach's Endlessly Rising Canon is a strange loop in music: the key modulates upward with each repetition — C minor, D minor, E minor — and then, impossibly, returns to C minor. The ear hears an endless ascent that loops back. The music climbs forever and goes nowhere. It is a auditory Liar.

Hofstadter's audacious thesis: consciousness is a strange loop.

"The self comes into being at the moment it has the power to reflect itself."

The brain builds a model of the world. It builds a model of itself within that world. It builds a model of itself modeling itself. The recursion creates a self that feels real, that has causal power, that can reflect on its own reflection. The "I" is not a thing. It is a pattern — a self-referential symbol system implemented in neurons. Just as Gödel's sentence is implemented in numbers. The medium is different. The structure is the same.

Hofstadter returned to the thesis in I Am a Strange Loop (2007), arguing that the self is a narrative fiction woven from symbolic data — but a fiction with real effects. The strange loop is not an illusion. It is a level of description that has causal reality. The pattern exists. The pattern matters. The pattern can reflect on itself. That reflection is consciousness.

Smullyan: the Liar as entertainment

Raymond Smullyan was a logician, a magician, a pianist, and the greatest puzzle-maker of the 20th century. His method was to take the deepest results in mathematical logic — Gödel's theorems, Tarski's undefinability of truth, Löb's theorem — and turn them into puzzles. The puzzles were accessible. The mathematics underneath was not.

Smullyan's most famous creation is the Island of Knights and Knaves. Knights always tell the truth. Knaves always lie. Every inhabitant is one or the other. You meet an inhabitant. They say something. Who are they?

Puzzle 1: "I am a knave"

An inhabitant says: "I am a knave."

A knight cannot say this — a knight tells the truth, and a knight is not a knave. A knave cannot say this either — if a knave says "I am a knave," they are telling the truth, and a knave never tells the truth. The statement is impossible. It cannot be uttered by either type. It is the Liar, dressed in island clothes.

Puzzle 2: Two inhabitants

You meet two inhabitants, A and B. A says: "At least one of us is a knave."

If A is a knave, the statement is false — meaning neither is a knave. But then A would be a knight, contradiction. So A must be a knight. Then the statement is true — at least one is a knave. So B is a knave. A is a knight, B is a knave. The puzzle resolves. The Liar does not always paralyze. Sometimes it selects.

Puzzle 3: "You will never know that I am a knight"

An inhabitant says: "You will never know that I am a knight."

Suppose they are a knave. Then the statement is false — meaning you will know they are a knight. But you cannot know something false. They are not a knight. So the statement is true — but a knave cannot tell the truth. Contradiction. They must be a knight. Then the statement is true: you will never know they are a knight. But you just deduced they are a knight. You know it. So the statement is false. Contradiction again.

This is Smullyan's bridge to Gödel. Replace "know" with "prove." An inhabitant says: "This statement cannot be proved." This is Gödel's sentence. If it's provable, the system proved a falsehood. If it's not provable, it's true — but unprovable. The system is incomplete. Smullyan taught this in puzzle form before revealing the connection. The puzzle was fun. The mathematics was the same.

Puzzle 4: The Portia caskets

From The Lady or the Tiger? (1982). Portia's suitor must choose among three caskets — gold, silver, lead. One contains Portia's portrait. Each casket bears an inscription. At most one inscription is true. Which casket holds the portrait?

The constraints force a systematic elimination. Each possibility is tested. Each produces a contradiction or a solution. The method is the logic of knights and knaves generalized to objects and inscriptions. The objects are silent. The inscriptions speak. The suitor reasons. Smullyan wrote dozens of these. Each one teaches a different logical structure disguised as a fairy tale.

Puzzle 5: The bird watchers

From To Mock a Mockingbird (1985). In a forest, birds call to each other. Each bird's call invokes another bird's call. A mockingbird imitates any bird it hears. A lark composes calls. The birds are combinators — the primitive functions of combinatory logic, disguised as birds. The mockingbird is the M combinator: Mx = xx. The lark is the L combinator: Lxy = x(yy). The puzzles teach the foundations of computation without mentioning computation. By the end, you have derived the Y combinator — the fixed-point operator that makes recursion possible — from birdsong. You have learned the lambda calculus. You thought you were birdwatching.

Puzzle 6: Forever undecided

From Forever Undecided: A Puzzle Guide to Gödel (1987). Smullyan introduces reasoners — mathematical agents who believe statements according to logical rules. A reasoner is peculiar if they believe some statements and their negations. A reasoner is stable if they believe they believe something. A reasoner is modest if they believe something only if they believe they believe it. By tuning the rules of what a reasoner believes about their own beliefs, Smullyan reproduces Gödel's theorem, Löb's theorem, and the modal logic of provability — all as puzzles about what a reasoner can consistently believe about themselves.

A reasoner who believes "I am consistent" is, in certain conditions, necessarily inconsistent. The act of believing in your own consistency produces inconsistency. This is Gödel's Second Incompleteness Theorem, stated as a puzzle about self-confident reasoners. Smullyan called it "the most startling result in all of mathematical logic." He was not exaggerating.

The bookshelf

The books that trace the Liar from ancient paradox to modern science:

Nagel & Newman, Gödel's Proof (1958). The shortest path from zero to understanding Gödel. Under 150 pages. Requires high school mathematics. Reads like a detective story where the culprit is the limits of formal reasoning.

Raymond Smullyan, What Is the Name of This Book? (1978). Knights, knaves, the Liar, Portia's caskets, and the puzzle that gives the book its title (the answer is in the book; the title is the question; the paradox is the point). The best introduction to self-referential logic ever written, disguised as a puzzle collection.

Raymond Smullyan, The Lady or the Tiger? (1982). More knights and knaves. More caskets. Day-knights who tell the truth during the day and lie at night. Sane reasoners who reason correctly and insane reasoners who reason incorrectly. Zombies who say what they believe and vampires who say the opposite. The taxonomy of logical characters expands. The underlying logic remains the same. Self-reference is the invariant.

Raymond Smullyan, To Mock a Mockingbird (1985). Combinatory logic disguised as birdwatching. The best introduction to the lambda calculus ever written, disguised as an ornithology text. You will learn more about computation from these bird puzzles than from most programming books.

Raymond Smullyan, Forever Undecided (1987). The puzzle guide to Gödel. Reasoners, beliefs, consistency, provability. If you read one Smullyan book after What Is the Name of This Book?, make it this one. The bridge from knights and knaves to mathematical logic is built here.

Douglas Hofstadter, Gödel, Escher, Bach (1979). The Pulitzer winner. Strange loops, tangled hierarchies, and the argument that consciousness is self-reference implemented in neurons. Read it slowly. The dialogues are not decoration. They contain the argument in compressed form.

Douglas Hofstadter, I Am a Strange Loop (2007). The thesis of GEB, stripped of the fugues and the artwork and the Tortoise. Consciousness as a self-referential pattern. The "I" as a strange loop. Cleaner than GEB. Less fun. More direct.

The thread

The Liar is 2,600 years old. It was a curiosity, then a paradox, then a proof. Gödel showed that it was not a trick of language but a structural necessity: any system that can represent itself can construct the Liar, and the Liar breaks completeness. Turing showed that the same self-reference makes it impossible to decide, in general, whether a program will halt. Hofstadter argued that the same self-reference, implemented in neurons and iterated across levels, produces the sensation of being a self.

Smullyan, alone among them, made it fun. His puzzles are the Liar staged as entertainment. A knight says something impossible. A knave constructs a paradox. A casket inscription produces a contradiction. The logic is the same. The presentation is joyful. The mathematics underneath is as deep as anything in Gödel. The puzzles teach without announcing that they are teaching. By the time you realize you are learning modal logic, you have already learned it.

"This statement is false." Six words. Twenty-six centuries. One idea. Still not finished with it.


References:

  • Ernest Nagel and James R. Newman, Gödel's Proof, New York University Press, 1958. (Revised edition edited by Douglas Hofstadter, 2001.)
  • Douglas Hofstadter, Gödel, Escher, Bach: An Eternal Golden Braid, Basic Books, 1979.
  • Douglas Hofstadter, I Am a Strange Loop, Basic Books, 2007.
  • Raymond Smullyan, What Is the Name of This Book?, Prentice-Hall, 1978.
  • Raymond Smullyan, The Lady or the Tiger?, Knopf, 1982.
  • Raymond Smullyan, To Mock a Mockingbird, Knopf, 1985.
  • Raymond Smullyan, Forever Undecided: A Puzzle Guide to Gödel, Knopf, 1987.
  • Kurt Gödel, "On Formally Undecidable Propositions of Principia Mathematica and Related Systems," 1931.

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

Simplicity does not precede complexity, but follows it

Alan Perlis wrote: 'Simplicity does not precede complexity, but follows it.' You do not start simple. You start messy and refine. The simple thing is the complex thing, understood. Git rebase, refactoring, and clean interfaces are not first drafts. They are the artifact of having already been wrong.

perlissimplicitycomplexitygitrefactoringsoftware-design

Alan Perlis, the first Turing Award winner, wrote a set of epigrams on programming. Number 24 reads:

"Simplicity does not precede complexity, but follows it."

The sentence is small. The insight is deep. You do not start with simplicity. You start with the mess. The simplicity comes after — after you have built the complex thing, understood it, and removed everything that wasn't necessary. The clean interface wasn't designed. It was discovered by building the dirty one and noticing which parts mattered.

This is not how we teach software engineering. We teach: design the clean thing first. Think before you code. Get the architecture right upfront. The teaching is aspirational. The practice is different. You cannot design the clean thing first because you don't understand the problem well enough to know what "clean" means. Clean is a property of understanding. Understanding comes from building. Building produces complexity. The complexity teaches you what to remove. The removal produces simplicity. The simplicity follows the complexity. It never preceded it.

Git history as the proof

Your git history is the most honest record of how software is actually made. The first draft of a branch is a mess. Commits are named "WIP," "fix," "try again," "actually fix," "fix the fix." Files change in ways that make no sense in isolation. The commit that added the feature also broke the tests. The commit that fixed the tests also reformatted unrelated code. The history is a log of discovery, not a narrative. Discovery is messy. Messy produces complex histories.

Then, before merging, you run git rebase -i. You squash the "fix" commits into the commit they were fixing. You reorder the logical changes into a sequence that tells a story. You split the commit that did two things into two commits that each do one thing. You rewrite the commit messages to explain why, not what. The result is a clean history. Four commits. Each does one thing. Each has a clear message. The sequence makes sense. The history reads like it was planned.

It was not planned. It was discovered. The clean history is the artifact of having already been wrong. The rebase is the removal of the evidence of confusion. The evidence was real. The confusion was productive. The clean history is the simplicity that followed the complexity.

This is Perlis's epigram applied to software configuration management. You cannot produce the clean history first. You produce the messy history, learn what the story actually is, and then rewrite the history to tell that story. The rewrite is not dishonest. It is editorial. The editor removes what doesn't serve the narrative. The narrative wasn't known when the first draft was written. It was discovered by writing the draft.

The dirty branch as research

A branch is a research project. You don't know what you'll find. You have a hypothesis: "I think I can add this feature by modifying these three files." The hypothesis is wrong. The three files become seven. One of them requires a refactor you didn't anticipate. The refactor breaks a test in an unrelated module. You fix the test. You discover that the feature doesn't work the way the spec described because the spec didn't account for an edge case you found while implementing. You adjust. The branch grows. The commit count climbs. The messages get shorter. "wip," "ugh," "ok actually working now."

This is not failure. This is research. The research produced complexity. The complexity is the evidence that you learned something. The learning is the value. The messy history is the record of the learning. It is not fit for public consumption. It is not meant to be. It is your lab notebook. The notebook is messy. The paper you publish is clean. The paper is the rebased branch. The notebook is the original history. You need both. The notebook is how you got there. The paper is what you found.

The industry norm of squashing entire branches into a single commit is the extreme form of this. One commit. One message. All the evidence of discovery, erased. The squash is too aggressive. It removes the intermediate logic — the sequence of insights that produced the final result. A future reader who encounters a bug in this code wants to see the commits that added it, not a single massive diff. The rebase preserves the logic. The squash preserves only the outcome. Outcome without logic is harder to debug. Logic without cleanup is harder to read. The art is in the middle: enough rebase to tell the story, not so much that the story disappears.

Refactoring as the same pattern

Refactoring is the Perlis pattern applied to code structure. You do not design the clean abstraction first. You design the abstraction that works. It is messy. Methods are too long. Classes have too many responsibilities. The interface exposes implementation details. The code works. The code is ugly.

You refactor. You extract methods. You split classes. You hide implementation behind the interface. The result is clean. The clean result looks like it was designed. It was not designed. It was discovered by writing the ugly version, understanding which parts were ugly, and removing the ugly. The ugly taught you what clean meant. Clean meant "what remains after you remove the accidental complexity." You couldn't identify the accidental complexity until you built the accidental complexity. Building it was the research. Removing it was the refactoring. The refactored code is the simplicity that followed the complexity.

"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." — Tony Hoare

The first method is more difficult because it requires you to see the simplicity before you've built the complexity. Nobody can do this reliably. The people who appear to do it are people who have built similar complexity before and are remembering what they learned. They're not starting from simplicity. They're starting from the simplicity that followed the complexity they built five years ago on a different project. Experience is the accumulated simplicity that followed accumulated complexity. The senior engineer who designs the clean thing first is not designing. They are remembering. The memory is of complexity they once built and later simplified. The simplification became intuition. The intuition looks like foresight. It is hindsight, internalized.

The false simplicity of premature design

The danger of Perlis's epigram is the danger of misunderstanding it. It does not mean "don't try to be simple." It means "don't expect to be simple on the first try." The first try will be complex because the first try is where you learn what the problem actually is. The learning requires complexity. The complexity is the tuition. You pay it. Then you simplify. The simplification is the return on the tuition.

Premature simplicity is the attempt to skip the tuition. You design the clean architecture before you've written any code that exercises it. The architecture is clean. It is also wrong. It is wrong because it was designed against an understanding of the problem that the designer didn't have yet. The designer thought they understood. They understood the spec. The spec is not the problem. The problem is what you discover when you try to implement the spec and find that the spec didn't account for the database migration, the cache invalidation, the race condition, the legacy API that returns XML, the user who needs the opposite of what the spec describes. The clean architecture didn't account for any of this because the designer hadn't encountered it. The designer hadn't encountered it because they hadn't built the complex thing. They tried to start with simplicity. Simplicity does not precede complexity.

The teams that ship ugly code that works and then refactor are following Perlis. The teams that design beautiful architectures that never ship are violating him. The first group pays the tuition. The second group avoids the tuition and never graduates.

The rebase as editorial craft

The rebase is the moment when the research becomes the story. The researcher becomes the editor. The editor's job is to remove everything that doesn't serve the reader. The reader is the future maintainer — possibly you, six months from now, at 2am, trying to understand why this code does what it does.

A good rebase does several things. It groups related changes into cohesive commits. It orders commits so that each one is a logical step that builds on the previous. It writes messages that explain why the change was made, not what the diff contains. It removes false starts, dead ends, and debugging code that served the researcher but would confuse the reader. The result is a history that tells a story. The story is true. It is not the whole truth. The whole truth includes the four hours you spent chasing a bug that turned out to be a typo. That truth is not useful to the reader. The editor removes it.

The rebase is not lying. It is editing. The difference between editing and lying is whether the published version misleads. A history that hides a security vulnerability is lying. A history that squashes the commit where you tried three different approaches and none worked is editing. The approaches that didn't work taught you something. The something is in the final commit message. The approaches themselves are not. They served their purpose. Their purpose was to teach you. You learned. The reader needs the lesson, not the curriculum.

The art of the squash

When to squash, when to preserve, when to split — these are editorial judgments. They require taste. The taste is developed by reading other people's histories and noticing which ones helped you understand and which ones didn't.

Squash when the intermediate commits are noise. "wip," "fix typo," "try again" — these add no information to the reader. Squash them into the commit they were iterating toward. The iteration was real. The reader doesn't need to see it.

Preserve when the intermediate commits are logical steps. "Extract the payment interface" followed by "Implement the Stripe adapter" followed by "Add payment confirmation email" — these are three distinct decisions. Each can be understood in isolation. Each might need to be reverted independently. Preserve them as separate commits.

Split when a single commit does two unrelated things. "Add the payment flow and also reformat the entire codebase" — these should be two commits. The reformatting is noise in the payment commit. The payment logic is noise in the reformat commit. Split them. The reader of the payment commit wants to understand payment. The reader of the reformat commit wants to understand reformatting. Neither wants both.

These judgments are Perlis applied to history. The clean history is the simplicity that followed the complexity of the original branch. The original branch was the research. The rebase is the simplification. The simplified history is what you merge. It didn't precede the messy one. It followed it.

The general principle

Perlis's epigram generalizes beyond git. It applies to every creative act that produces a structured artifact. The first draft of a novel is messy. The published version is clean. The clean version didn't precede the messy one. The messy one taught the author what the novel was about. The author rewrote it to be about that. The rewrite was the simplicity that followed the complexity.

The first version of an API is messy. It exposes implementation details, has inconsistent naming, handles errors differently in different endpoints. The second version is clean. The second version didn't precede the first. The first taught the designer which parts callers actually used, which parts were confusing, which errors were common. The second version removed what wasn't needed, standardized what was, and hid what shouldn't have been exposed. The clean API is the artifact of having watched real users struggle with the messy one.

The first architecture of a system is messy. The monolith has responsibilities that should be separate, dependencies that should be inverted, data that should be owned. The second architecture — the microservices extraction, the refactored modules, the clean interfaces — didn't precede the first. The first taught the team where the boundaries actually were. The boundaries weren't visible until the code was written. The code was complex. The complexity revealed the boundaries. The boundaries enabled the simplicity.

"Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it." — Alan Perlis

The genius doesn't start with simplicity. The genius builds the complex thing, understands it completely, and then removes the complexity that was never necessary. The removal is the genius. The removal is visible. The complex thing that preceded it is not. The genius looks like someone who started simple. They didn't. They started complex and removed everything that didn't earn its place. The removal is the art. The art is invisible. The simplicity is what remains.


References:

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

No solutions, only trade-offs

Thomas Sowell wrote: 'There are no solutions, only trade-offs.' This is the hardest sentence in economics. It is also the hardest sentence in software engineering. Every architecture decision is a trade. The engineer who looks for a solution is looking for something that doesn't exist.

sowelltrade-offseconomicssoftware-designarchitecture

Thomas Sowell, the economist who has probably done more than anyone since Milton Friedman to explain scarcity to the general public, wrote a sentence so compressed it can be missed:

"There are no solutions, only trade-offs."

This is not cynicism. It is clarity. A solution is something that makes a problem go away. A trade-off is something that makes one problem smaller at the cost of making another problem larger. Most of what we call solutions are trade-offs where we like the outcome and prefer not to think about what we gave up. The preference is human. The trade is real. The thing we gave up didn't disappear. It became someone else's problem.

Software engineering is trade-offs all the way down. Microservices trade coordination complexity for deployment independence. Monoliths trade deployment independence for coordination simplicity. Type systems trade expressiveness for safety. Dynamic languages trade safety for velocity. Relational databases trade flexibility for consistency. NoSQL trades consistency for scale. Synchronous calls trade resilience for simplicity. Asynchronous messages trade simplicity for resilience. Every architecture decision is a bet that the thing you're gaining is worth the thing you're losing. The bet is economic. The economics are usually implicit. Making them explicit is the discipline.

The trade-off you can't see

The dangerous trade-offs are the ones where the cost is invisible. You gain something now — a feature, a shortcut, a simpler implementation. You pay later — in complexity, in reduced velocity, in a rewrite. The gain is visible. The cost is deferred. Deferred costs are easy to ignore. Ignored costs compound. Compounding costs produce bankruptcy. The bankruptcy is the rewrite. The rewrite is the admission that the original trade-off was mispriced.

"The first lesson of economics is scarcity: There is never enough of anything to satisfy all those who want it. The first lesson of politics is to disregard the first lesson of economics." — Thomas Sowell

The first lesson of software engineering management is also to disregard the first lesson of economics. Every unrealistic deadline is a refusal to accept the trade-off between time and quality. Every under-resourced project is a refusal to accept the trade-off between scope and resources. Every "do more with less" is a refusal to accept that more of one thing means less of another. The refusal is not neutral. It pushes the cost somewhere. Usually onto the people doing the work. Usually onto the maintainability of the system. Usually onto the future, where it will be someone else's problem.

The trade-off you choose

The mature engineer accepts that every decision is a trade. They don't look for the solution. They look for the trade-off they can live with. The question is not "what is the right answer?" The question is "what are we willing to give up?"

A team choosing between a monolith and microservices is not choosing between good and bad. They are choosing between coordination overhead distributed across teams and coordination overhead concentrated in a single codebase. The overhead doesn't disappear. It changes form. The form change may be worth it. The overhead is still there. The team that thinks microservices eliminate coordination overhead will discover that coordination overhead in a distributed system manifests as API versioning conflicts, data inconsistency, deployment ordering dependencies, and the distributed monolith. The overhead was not eliminated. It was moved. The move may have been worth it. It was not free.

A team choosing between REST and NATS is not choosing between simple and complex. They are choosing between spatial coupling (REST: callers know callees) and semantic coupling (NATS: callers and callees agree on subjects). The coupling doesn't disappear. It changes form. The form change means different failure modes, different debugging tools, different operational practices. The choice is not about which is better. It is about which form of coupling the team is equipped to manage.

The trade-off as discipline

Naming the trade-off is the discipline. Most architecture debates are arguments about which trade-off to make, conducted by people who haven't named what they're trading. "We should use microservices." "We should stick with the monolith." The debate is unresolvable because the trade is unstated. State the trade: "Microservices will reduce coordination overhead between teams at the cost of increased operational complexity." "The monolith will reduce operational complexity at the cost of increased coordination overhead within the codebase." Now the debate is about which cost the organization is better equipped to bear. That is a debate that can be resolved. It requires knowing the organization — its team structure, its operational maturity, its tolerance for distributed complexity. The knowledge is local. The trade-off is universal.

Sowell's sentence is a tool. Apply it to every decision. When someone says "the solution is X," ask: what are we trading? When someone says "we need to do Y," ask: what are we giving up? When you find yourself certain that Z is right, ask: what cost am I not seeing? The cost is there. The certainty is hiding it. The sentence pierces the certainty. The piercing is uncomfortable. The discomfort is productive.

"No solutions, only trade-offs."

The sentence is small. The discipline is large. The sentence is the discipline. Apply it. The decisions will improve. The systems will survive longer. The rewrites will be postponed. The postponement is a trade-off too. Everything is.


References:

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

NATS pub/sub beats REST for microservices

REST couples services by address. NATS couples them by subject. One is a phone call where you must know the number. The other is a broadcast where you only need the frequency. Modularity lives in the difference.

natsjetstreampubsubmicroservicesgolangrestmodularity

REST is the default for microservices communication. It is the default because it is familiar, not because it is good. Every service exposes HTTP endpoints. Every caller knows every callee's address. The system is a web of explicit dependencies, each hardcoded in configuration or service discovery.

NATS inverts this. Services publish to subjects. Services subscribe to subjects. No service knows any other service exists. The subject is the interface. The publisher has no idea who is listening. The subscriber has no idea who is publishing. They agree on the subject name and the message schema. Nothing else.

This is not a preference. It is an architectural difference with measurable consequences for coupling, scalability, and resilience. This post explains what NATS is, how JetStream extends it, why pub/sub produces better modularity than REST, and when each approach makes sense.

What NATS is

NATS is a messaging system. It is not a message broker in the RabbitMQ sense. It is closer to a network switch for messages. Clients connect to NATS servers. Servers route messages between clients based on subject subscriptions. The core is small, fast, and does one thing: move messages from publishers to subscribers with minimal latency and maximal throughput.

The protocol is text-based and trivial. A client sends PUB <subject> <size>\r\n<payload>\r\n. A client sends SUB <subject> <sid>\r\n. The server matches subjects to subscribers and delivers the message. That is the core. No persistence. No acknowledgements. No transactions. Just publish, subscribe, deliver.

Subjects and wildcards

Subjects are hierarchical tokens separated by dots: orders.created, inventory.updated.us-east, payment.processed.visa. Subscribers can use wildcards:

  • * matches one token: orders.* matches orders.created and orders.cancelled, not orders.payment.authorized
  • > matches all remaining tokens: orders.> matches orders.created, orders.payment.authorized, orders.items.returned.refund

Wildcards let subscribers express interest in categories of events without knowing every specific subject. A logging service subscribes to >.> and receives everything. An inventory service subscribes to orders.* and receives only order lifecycle events. The subject namespace is the API. The wildcards are the subscriptions. The routing is automatic.

Spatial decoupling

In REST, Service A must know Service B's address. When B moves, A's configuration must change. When B scales to multiple instances, a load balancer must be configured. The load balancer must know all instance addresses. Health checks must be written. The address is the coupling.

In NATS, Service A publishes to orders.created. Service B subscribes to orders.created. Neither knows the other's address. Neither knows the other exists. The NATS server — or cluster of servers — routes the message. If B moves to a different machine, region, or cloud, nothing changes. The subject is the only address. The subject does not change.

Queue groups: scale without ceremony

NATS queue groups distribute messages across multiple subscribers without a load balancer:

// Three instances, one queue group, automatic load distribution
nc.QueueSubscribe("orders.process", "order-workers", func(m *nats.Msg) {
    processOrder(m)
})

That is it. No load balancer. No health checks. No instance registry. NATS distributes messages round-robin across queue group members. If an instance crashes, NATS stops routing to it. If a new instance starts, NATS includes it. The publisher never knew how many instances existed. The publisher still doesn't. The subject is the interface. The instances are an implementation detail.

Clustering and superclusters

A single NATS server can handle millions of messages per second. For scale beyond one machine, NATS clusters route messages between servers. For scale beyond one datacenter, gateways connect clusters into superclusters.

Clustering (routes): Servers within a cluster form a full mesh. Each server connects to every other server. Routes use a dedicated port. Subscription interest is gossiped between servers — a server only forwards messages to peers that have matching subscribers. For N nodes, N(N−1)/2 connections. A three-node cluster: three connections. A five-node cluster: ten. The protocol is designed for this. The overhead is minimal.

Gateways (superclusters): Gateways connect entire clusters. Three clusters of five nodes each: a full node-to-node mesh would require 105 connections. With gateways, each node connects to one node in each remote cluster — 30 connections. Interest propagation is cluster-scoped. A message published in Cluster A is only forwarded to Cluster B if Cluster B has expressed interest in that subject. Optimistic forwarding on first message, suppressed thereafter.

Leaf nodes: For edge deployments — IoT, retail, remote offices — leaf nodes extend a cluster across security boundaries without requiring bidirectional connectivity. Local clients authenticate locally. The leaf connection acts as a NATS client to the hub, exporting subjects the edge can publish and importing subjects the edge can subscribe to. Local traffic stays local. Remote traffic traverses the leaf. Queue semantics are preserved: local subscribers are preferred before forwarding across the leaf.

This architecture scales from a developer's laptop (nats-server -js) to a planet-scale messaging fabric. The subject namespace is global. The routing is automatic. The addressing is semantic, not spatial.

What JetStream is

Core NATS is fire-and-forget. If a subscriber is offline, the message is lost. For systems that need persistence, replay, or guaranteed delivery, NATS provides JetStream — a persistence layer built into the NATS server.

JetStream decouples storage from consumption. Streams store messages. Consumers read from streams. This separation is the key architectural insight. Multiple consumers can read the same stream independently, each at its own pace, with its own acknowledgement state, filtering by subject, starting from different points in the log.

Streams: the storage layer

A stream is a named, append-only log that captures messages published on one or more subjects:

js, _ := jetstream.New(nc)

stream, _ := js.CreateStream(ctx, jetstream.StreamConfig{
    Name:      "ORDERS",
    Subjects:  []string{"orders.>"},
    Storage:   nats.FileStorage,
    Replicas:  3,                       // Raft across 3 nodes
    MaxAge:    7 * 24 * time.Hour,      // Retain 7 days
    MaxBytes:  10 * 1024 * 1024 * 1024, // 10 GB ceiling
})

Stream configuration determines retention policy:

  • LimitsPolicy (default): keep messages until count, bytes, or age limits are reached. This is for event sourcing, replay, or audit trails.
  • WorkQueuePolicy: delete each message after it's been acknowledged by any consumer. This is for job queues — process once, discard.
  • InterestPolicy: delete messages only after all consumers have acknowledged them. This is for fan-out where multiple services process the same event.

Storage can be file-backed (survives restart) or memory-backed (faster, lost on restart). Replication uses Raft across 1, 3, or 5 cluster nodes. Odd numbers for quorum. A three-replica stream tolerates one node failure. A five-replica stream tolerates two.

Consumers: the read cursors

A consumer is a named view into a stream. Multiple consumers read independently. Each has its own position, filter, and acknowledgement state.

Pull consumers are the default for work queues. The client explicitly requests messages:

consumer, _ := stream.CreateConsumer(ctx, jetstream.ConsumerConfig{
    Durable:       "order-processor",
    FilterSubject: "orders.created",
    AckPolicy:     jetstream.AckExplicitPolicy,
    AckWait:       30 * time.Second,
})

// Fetch a batch of 10 messages
batch, _ := consumer.Fetch(10)
for msg := range batch.Messages() {
    process(msg)
    msg.Ack()       // Explicit acknowledgement
    msg.AckSync()   // Double-ACK: wait for server confirmation
}

Push consumers are for low-latency streaming to a single instance. The server pushes messages to a delivery subject:

consumer, _ := stream.CreateConsumer(ctx, jetstream.ConsumerConfig{
    Name:          "order-streamer",
    DeliverPolicy: jetstream.DeliverNewPolicy,
    AckPolicy:     jetstream.AckExplicitPolicy,
})

cc, _ := consumer.Consume(func(msg jetstream.Msg) {
    process(msg)
    msg.Ack()
})

Consumer replay policies determine where to start reading:

  • all: replay every message from the beginning
  • last: start with the last message, then follow live
  • new: only messages arriving after subscription
  • by_start_sequence: start from a specific sequence number
  • by_start_time: start from messages at or after a timestamp
  • last_per_subject: deliver the last message for each unique subject

Exactly-once semantics

JetStream provides exactly-once through two complementary mechanisms.

Publish-side deduplication. Set the Nats-Msg-Id header when publishing. Within the stream's duplicate window (default 2 minutes, configurable), messages with the same ID are silently discarded:

msg := nats.NewMsg("orders.created")
msg.Header.Set("Nats-Msg-Id", orderID)
msg.Data = payload
js.PublishMsg(ctx, msg)

If the publisher crashes before receiving the publish acknowledgement and re-publishes on restart, the duplicate is suppressed. The publisher retries safely. The stream stores exactly one copy.

For infinite deduplication beyond the time window, use DiscardNewPerSubject with MaxMessagesPerSubject = 1. Publishing to the same subject with an existing message fails. This behaves like a SQL INSERT with a unique constraint on the subject.

Consume-side double acknowledgement. AckSync() sends the ACK with a reply subject and blocks until the server confirms receipt. If the ACK is lost and the consumer crashes before the confirmation, the message is redelivered and must be processed idempotently:

func process(msg jetstream.Msg) error {
    if alreadyProcessed(msg) {
        msg.AckSync() // Still ACK — was processed, just not acknowledged
        return nil
    }
    if err := doWork(msg); err != nil {
        msg.Nak() // Return to queue for retry
        return err
    }
    msg.AckSync() // Guaranteed: server confirmed receipt of ACK
    return nil
}

Publish-side deduplication + double-acknowledged consumption + idempotent processing = true end-to-end exactly-once. Messages are never lost. Messages are never duplicated. The guarantee is as strong as any message broker provides and stronger than most.

Temporal decoupling

REST fails when the callee is unavailable. The caller must retry, back off, circuit-break, or fail. Each retry loop is hand-rolled. Each circuit breaker has slightly different thresholds. Reliability is the sum of individually implemented strategies that were never tested together.

JetStream provides temporal decoupling by default:

// Publisher: fire and forget
js.Publish(ctx, "orders.created", event)

// Consumer: receives when ready, even hours later
consumer.Consume(func(msg jetstream.Msg) {
    process(msg)
    msg.Ack()
})

The publisher publishes and moves on. If no subscriber is online, the message waits in the stream. If the subscriber crashes mid-processing, the message is redelivered after AckWait expires. The publisher doesn't retry. The subscriber doesn't need to be online when the message is published. The infrastructure handles reliability. The services handle business logic.

The REST comparison

Here is an order service calling an inventory service over REST in Go:

func (s *OrderService) CreateOrder(ctx context.Context, order Order) error {
    body, _ := json.Marshal(ReserveRequest{
        ProductID: order.ProductID,
        Quantity:  order.Quantity,
    })
    resp, err := http.Post(
        "http://inventory-service:8080/api/reserve", // knows address
        "application/json",
        bytes.NewReader(body),
    )
    if err != nil {
        return fmt.Errorf("inventory unreachable: %w", err) // knows availability
    }
    defer resp.Body.Close()
    if resp.StatusCode != 200 { // knows success semantics
        return fmt.Errorf("reservation failed: %s", resp.Status)
    }
    return s.db.Insert(ctx, order)
}

The order service knows the inventory service's URL, endpoint path, request format, success semantics, and availability requirement. If the inventory service moves, the order service must be reconfigured. If the inventory service is down, the order fails. If the inventory service's API changes, the order service must be updated. Five types of shared knowledge. Five dimensions of coupling. This is one dependency. A typical microservices system has dozens.

Here is the same workflow over NATS:

func (s *OrderService) CreateOrder(ctx context.Context, order Order) error {
    event, _ := json.Marshal(OrderRequested{
        OrderID:   order.ID,
        ProductID: order.ProductID,
        Quantity:  order.Quantity,
    })
    // Fire and forget: publish to the stream
    js.Publish(ctx, "orders.created", event)

    // Request-reply: ask inventory, don't know who answers
    msg, err := nc.Request("inventory.reserve", event, 5*time.Second)
    if err != nil {
        return fmt.Errorf("reservation failed: %w", err)
    }
    var result ReserveResult
    json.Unmarshal(msg.Data, &result)
    if !result.Success {
        return fmt.Errorf("reservation denied: %s", result.Reason)
    }
    return s.db.Insert(ctx, order)
}

// Inventory service: subscribers don't know publishers exist
func (s *InventoryService) Run() {
    nc.Subscribe("inventory.reserve", func(m *nats.Msg) {
        var req OrderRequested
        json.Unmarshal(m.Data, &req)
        err := s.reserveStock(req.ProductID, req.Quantity)
        result := ReserveResult{Success: err == nil}
        if err != nil { result.Reason = err.Error() }
        reply, _ := json.Marshal(result)
        nc.Publish(m.Reply, reply)
    })
}

The order service knows two subjects: orders.created and inventory.reserve. It does not know the inventory service exists. It does not know how many instances are listening. It does not know where they are deployed. If the inventory service moves — nothing changes. If three more inventory instances start — nothing changes. If the inventory service's internal implementation is rewritten — nothing changes, as long as the subject and schema are preserved. The subject is the contract. The subject is the only coupling.

What coupling actually means

Henney defined coupling as shared knowledge between components. REST maximizes shared knowledge. The caller knows the callee's address, API shape, availability window, response semantics, error taxonomy, authentication mechanism, and rate limits. Each is a dimension of coupling. When the callee changes any of these, the caller must adapt or break.

NATS minimizes shared knowledge. Publisher and subscriber agree on the subject name and the message schema. Neither knows the other exists. Neither knows how many of the other exist. Neither knows where the other is deployed. Neither knows whether the other is currently online. Subject name and schema: that is the total shared knowledge. Everything else is hidden.

This is Parnas's criterion applied to inter-service communication. Hide the volatile decisions — deployment location, instance count, availability status, internal API changes — behind a stable interface. The subject is the stable interface. The service is the volatile implementation. The subject name doesn't change when the implementation scales, moves, or restarts. The coupling is minimized. The modularity is real.

When REST still makes sense

REST is not universally wrong. It is appropriate in specific conditions:

External APIs at the system boundary. Mobile apps, browsers, third-party integrations speak HTTP. The edge of the system faces outward. The interior of the system should not. Expose REST at the boundary. Use NATS internally. The boundary is where HTTP belongs. The interior is where it doesn't. External clients need synchronous request-reply with standard protocols and well-known ports. Internal services need decoupling and resilience. The two requirements are different. The two protocols should be different.

Genuinely stable dependencies. If Service A will always depend on Service B, and B's API is provably stable, and B's deployment will never change in ways that affect A — REST is fine. This condition is rarer than architects assume. Most dependencies that appear stable at design time are unstable at year three. The REST interface hardcodes assumptions that time will invalidate. The NATS subject survives the invalidation because it doesn't encode the assumptions.

For everything else — internal service communication, event-driven workflows, systems that scale, systems that survive partial failure, systems where the dependency graph will evolve — NATS pub/sub backed by JetStream is not just better. It is what REST pretends to be: modular, decoupled, and resilient. REST shares knowledge by default. NATS hides knowledge by default. Modularity lives in what services don't know about each other. NATS lets them know less. That is the definition of better architecture.


References:

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

libp2p is the internet, rewired

libp2p began as the networking layer of IPFS. It became the networking layer of Ethereum, Filecoin, and the decentralized web. Its architecture is a masterclass in modular protocol design — identity, transport, security, discovery, and messaging, each a swappable module. The client-server internet is the past. The peer-to-peer internet is already here.

libp2pp2pnetworkingipfsethereumdecentralized

libp2p began as a networking layer inside IPFS. Juan Benet and the Protocol Labs team were building the InterPlanetary File System — a content-addressed, peer-to-peer file system — and they kept solving the same problems that every peer-to-peer application solves. Peer identity. Peer discovery. Transport negotiation. Secure channels. NAT traversal. Message routing. These are not file-sharing problems. They are networking problems. Every P2P application faces them. Most solve them poorly, in tightly coupled code that can't be reused.

Protocol Labs made the decision that elevated libp2p from an internal component to a universal standard: they extracted the networking layer into a standalone, modular, language-agnostic framework. The decision was strategic. A clean networking stack that any P2P application could use would attract contributors beyond the IPFS ecosystem. It would become infrastructure. It did.

The stack

libp2p is not a protocol. It is a suite of protocols, each solving one layer of the peer-to-peer problem. The layers are composable. Applications pick the modules they need.

Layer Description
Peer Identity Every peer has a PeerID — a cryptographic hash of its public key. Identity is self-sovereign. No certificate authority. No registration. Generate a keypair. You exist.
Multiaddress Self-describing network addresses: /ip4/192.168.1.100/tcp/8001/p2p/QmPeerID. The address encodes the transport, the IP, the port, and the peer identity in one string. One format for every possible way to reach a peer.
Transport Transport-agnostic. TCP, QUIC, WebSockets, WebRTC, WebTransport — all first-class. An application listens on multiple transports simultaneously. Peers connect using whatever transport they share.
Security Mandatory encryption upgrade. Every connection upgrades to an encrypted channel using TLS 1.3 or Noise. No unencrypted fallback. No "encryption is optional." The connection is secure or the connection doesn't happen.
Stream Multiplexing Multiple independent bidirectional streams over a single transport connection. Yamux, Mplex, or QUIC-native multiplexing. One TCP connection, dozens of concurrent protocol interactions.
Protocol Negotiation multistream-select lets peers agree on protocols at connection time. A peer requests /ipfs/bitswap/1.2.0. The remote peer responds: supported, or not. The negotiation is inline. The protocol version is explicit.
Peer Discovery Bootstrap nodes for initial entry. mDNS for LAN discovery. Kademlia DHT for global peer routing. Rendezvous points for ephemeral meetups. The discovery mechanism is a module. Swap it without changing anything above.
Messaging GossipSub — a scalable, attack-resistant pub/sub system. Messages flood through a mesh of peers. Peers score each other's behavior. Malicious peers are pruned. The mesh heals. Ethereum's consensus layer runs on GossipSub. It delivers blocks and attestations across tens of thousands of validators.
NAT Traversal AutoNAT detects whether a peer is publicly reachable. Circuit Relay v2 routes traffic through a relay when direct connection fails. Hole punching establishes direct connections between NATed peers. The internet was not designed for peer-to-peer. libp2p works around that.

Each layer is a module. Each module implements an interface. Swap the transport from TCP to QUIC. Swap the crypto from Noise to TLS. Swap the multiplexer from Yamux to Mplex. The layers above don't know. The layers below don't care. This is Parnas's information hiding applied to the network stack. The volatile decision — which transport, which cipher, which discovery mechanism — is hidden behind a stable interface. The interface is the protocol. The implementation is the module.

The origin

libp2p was extracted from IPFS, but its intellectual lineage goes deeper. The peer-to-peer era of the early 2000s — Napster, Gnutella, BitTorrent, Kademlia — produced a generation of protocols that solved individual P2P problems. BitTorrent solved efficient file distribution. Kademlia solved distributed hash tables. Gossip protocols solved epidemic message propagation. But each protocol was a monolith. BitTorrent's peer discovery was coupled to its file transfer. Gnutella's search was coupled to its network topology. You couldn't take Kademlia out of BitTorrent and use it elsewhere. You couldn't take BitTorrent's choking algorithm and use it in a chat application.

libp2p is the recognition that these problems are orthogonal. Peer discovery is not a file-sharing problem. It is a networking problem. NAT traversal is not a VoIP problem. It is a networking problem. Pub/sub is not a blockchain problem. It is a networking problem. The solutions should be libraries, not features embedded in monoliths. libp2p made them libraries. The monoliths became consumers of the libraries. Ethereum didn't have to write a pub/sub system. It imported GossipSub. Filecoin didn't have to write a DHT. It imported Kademlia. The specialization that the P2P era produced became composable modules in a universal stack.

The applications

IPFS. libp2p's origin and still its largest deployment. The IPFS public DHT runs on go-libp2p with thousands of globally reachable peers. Content routing, peer routing, and block exchange all run over libp2p protocols. The content-addressed web has a networking layer. It is libp2p.

Ethereum. The beacon chain's consensus layer uses GossipSub for block and attestation propagation. Thousands of validators broadcast messages. The mesh must be reliable, low-latency, and resistant to eclipse attacks. GossipSub provides adaptive peer scoring — peers that behave badly are scored down and eventually pruned. The scoring is the defense against Sybil attacks. The defense is built into the pub/sub layer. The consensus layer doesn't implement it. It inherits it.

Filecoin. Storage miners announce themselves via the Kademlia DHT. Clients resolve miner PeerIDs to network addresses. Block propagation runs over GossipSub. The storage retrieval market uses an extended Bitswap protocol. Filecoin is a marketplace for storage. The marketplace runs on libp2p.

Optimism. Layer 2 rollup nodes use libp2p for peer-to-peer communication between sequencers. The L2 inherits the networking stack from the L1 ecosystem. The stack is the same. The chain is different.

Beyond the major chains, libp2p is the default networking layer for decentralized applications. Secure file transfer (CipherStream). Decentralized databases (Source Network, DefraDB). Peer-to-peer gaming. IoT mesh networks. Federated learning — training AI models across decentralized nodes without centralizing data. Each application imports the modules it needs. Peer discovery for the database. GossipSub for the game state. Circuit relay for the IoT device behind NAT. The modules compose. The composition is the application's networking layer.

Why it won

libp2p won because it made the right architectural choice at the right time: modularity over monolith, protocol suite over framework, composition over integration. The alternative — every application building its own networking stack — produced fragmented, incompatible, under-tested implementations. libp2p produced a shared stack maintained by contributors from multiple ecosystems, tested at the scale of Ethereum's consensus layer, hardened against the attacks that real P2P networks face.

The modularity is the moat. When QUIC becomes the dominant transport, libp2p applications swap the transport module. When a better DHT algorithm is discovered, they swap the routing module. When a novel NAT traversal technique emerges, they add a new module. The interfaces are stable. The implementations evolve. This is the architecture of systems that survive technological change. The architecture was not an accident. It was the founding insight: separate the concerns, define the interfaces, let the implementations compete.

The client-server internet was an accident of history. The early internet was peer-to-peer. Client-server won because NATs and firewalls made peer-to-peer hard, because ISPs gave consumers dynamic IPs, because the economics of centralized services were compelling. libp2p is the recognition that those constraints are dissolving. IPv6 restores end-to-end addressing. QUIC makes secure, multiplexed connections trivial. WebRTC gives browsers direct peer connections. The technical barriers to peer-to-peer are falling. The remaining barriers are architectural — the assumption that every application needs a server, that every message must pass through a data center. libp2p challenges that assumption at the network layer. The challenge is working.


References:

Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.

Statistical Arbitrage

Statistical arbitrage is not about a single price discrepancy. It is about a statistical edge across many trades. Pairs trading, cointegration, mean reversion. The math is from the 1980s. The execution is now on-chain. The principles haven't changed. The speed has.

dexstat-arbpairs-tradingcointegrationmean-reversion

Statistical arbitrage is the exploitation of statistical mispricings. Unlike pure arbitrage — buy cheap, sell dear, simultaneously — stat arb involves risk. The mispricing may persist. It may widen before it corrects. The stat arb trader bets that it will correct, on average, over many trades. The edge is small per trade. The volume makes it profitable.

The strategy emerged from the quantitative trading revolution of the 1980s. Gerry Bamberger and Nunzio Tartaglia at Morgan Stanley developed the first pairs trading strategies. The idea: identify two stocks that historically move together. When they diverge — one rises, the other falls — the spread between them has widened beyond its historical range. Buy the underperformer. Short the overperformer. Wait for convergence. The trade is market-neutral — you are not betting on the direction of the market, only on the relationship between the two stocks. Market-neutral strategies attracted capital because they were uncorrelated with the market. The lack of correlation was the selling point.

The functional origin: pairs trading

Pairs trading is the simplest stat arb strategy. Two assets with a historical relationship. A spread between them that mean-reverts. The trader identifies the relationship, waits for divergence, enters the trade, exits on convergence. The relationship can be economic — two companies in the same industry, two tokens on the same blockchain — or statistical — two assets whose prices are cointegrated.

Cointegration, developed by Clive Granger (Nobel Prize, 2003) and Robert Engle (Nobel Prize, 2003), is the statistical property that makes pairs trading work. Two time series are cointegrated if each is non-stationary (its statistical properties change over time) but a linear combination of them is stationary (mean-reverts). The classic example: a drunk and her dog on a leash. Both walk randomly. The distance between them — the leash length — is stationary. The dog wanders. The drunk wanders. The leash pulls them back together. The leash is the cointegrating relationship.

In finance: two stocks in the same sector. Each follows a random walk. Their price ratio mean-reverts. The ratio is the leash. The trader buys when the ratio is low relative to its historical average — the underperformer is cheap relative to the outperformer. The trader sells when the ratio reverts. The trade is profitable if the cointegrating relationship persists. The relationship can break. The break is the risk.

In crypto: ETH and a liquid staking derivative like stETH. The two should trade at parity — 1 stETH = 1 ETH. They occasionally diverge — stETH trades at a discount during market stress, when holders want to exit staked positions quickly. The divergence is the opportunity. The trader buys stETH at a discount, waits for convergence, sells at parity. The trade is directional in the pair but market-neutral in dollar terms. The risk: the discount widens further, the trader's capital is locked, the convergence takes longer than the trader can afford. The risk is real. The risk is managed by position sizing and stop-losses.

Statistical arbitrage in crypto

Crypto stat arb has advantages over traditional stat arb. The data is public and real-time. Every trade on a DEX is on-chain. Every price on a CEX is streamed. The data quality is higher than in traditional markets, where dark pools and off-exchange trading obscure true volumes. The crypto market structure — fragmented across hundreds of venues, each with its own liquidity profile — creates more opportunities for statistical mispricing than the consolidated equity markets. The fragmentation is the opportunity.

The strategies:

Cross-DEX pairs. The same token pair trades on multiple DEXs. The prices are usually close. When they diverge, the divergence is an opportunity. The trader buys on the cheaper DEX, sells on the dearer. The trade is atomic if the DEXs are on the same chain. It is non-atomic if they are on different chains — the trader bears bridge latency risk.

Liquid staking derivative arbitrage. stETH/ETH, rETH/ETH, cbETH/ETH. The derivatives should trade at or near parity with the underlying. They don't always. The discount widens during market stress. The trader accumulates at a discount, waits for convergence, redeems. The trade has a natural exit: the derivative can be redeemed for the underlying after the unstaking period. The redemption is the convergence guarantee. The guarantee makes the trade lower-risk than traditional pairs trading. The lower risk attracts capital. The capital compresses the spread.

Mean reversion in AMM pools. AMM pools exhibit mean-reverting behavior around the external market price. When a pool's price diverges from the CEX price, arbitrageurs trade against the pool to bring it back. The mean reversion is not guaranteed — it depends on arbitrageurs acting. But the arbitrageurs are reliable because the trade is profitable. The stat arb trader can front-run the arbitrageurs: enter when the divergence appears, exit when the arbitrageurs correct it. The trade is a bet on arbitrageurs doing their job. The bet is well-founded. The arbitrageurs are reliable.

The reference

Andrew Lo, Adaptive Markets: Financial Evolution at the Speed of Thought (2017). Lo's book is not specifically about statistical arbitrage. It is about the broader framework that makes stat arb intelligible: markets are not perfectly efficient. They are adaptive systems populated by boundedly rational agents competing for profits. The competition produces efficiency in the long run. The transition produces profit opportunities for those who can identify and exploit them faster than others. Stat arb is the mechanism of the transition. The stat arb trader is the agent of efficiency. The efficiency, once achieved, eliminates the stat arb trader's edge. The trader must find new edges. The search is continuous. The search is the subject of this series.


References:

  • Clive Granger, "Investigating Causal Relations by Econometric Models and Cross-Spectral Methods," Econometrica, 1969.
  • Robert Engle and Clive Granger, "Co-Integration and Error Correction: Representation, Estimation, and Testing," Econometrica, 1987.
  • Andrew Lo, Adaptive Markets: Financial Evolution at the Speed of Thought, Princeton University Press, 2017.
  • Related posts: Arbitrage, Market Making

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

Sandwich Attacks

A sandwich attack is the most profitable form of MEV. The attacker buys before the victim and sells after, pocketing the price difference. The victim pays more. The attacker extracts the difference. The sandwich is the dark forest made visible. Understanding it is the only defense against it.

dexsandwich-attacksmevfront-runningamm

A sandwich attack is a three-transaction sequence. The attacker sees a pending swap in the mempool — the victim's transaction. The attacker submits a buy transaction for the same token with a higher gas price, getting included first. The attacker's buy moves the AMM price up. The victim's swap executes at the higher price — the victim pays more. The attacker submits a sell transaction immediately after, selling the tokens bought in the first transaction at the elevated price. The attacker profits. The victim loses. The difference is the sandwich.

The name is apt. The victim's transaction is the filling. The attacker's two transactions are the bread. The bread surrounds the filling. The filling is consumed. The consumer is the attacker. The consumed is the victim.

The mechanics

In a constant-product AMM (xy = k), a swap changes the ratio of tokens in the pool, which changes the price. The larger the swap relative to the pool's liquidity, the larger the price impact. The sandwich attacker exploits the price impact of the victim's trade. The attacker buys before the victim, moving the price up. The victim buys at the inflated price, moving the price further up. The attacker sells at the post-victim price, profiting from the difference between their entry price and exit price.

The attacker's profit is bounded by the victim's slippage tolerance. The victim sets a maximum acceptable price — the slippage limit — when submitting the swap. If the price moves beyond the limit, the transaction reverts. The attacker must ensure the sandwich keeps the price within the victim's slippage tolerance. If it doesn't, the victim's transaction reverts, and the attacker is left holding tokens bought at an elevated price with no victim to sell to. The attacker loses. The slippage limit is the victim's defense. Setting it low reduces the sandwichable spread. Setting it too low causes the swap to revert on normal price movement.

The attacker also bears gas costs for three transactions. On Ethereum L1, gas can be expensive. The sandwich is only profitable if the extracted value exceeds 3× gas cost. On L2s, gas is cheaper. The lower cost makes smaller sandwiches profitable. The lower cost increases sandwich frequency. The increased frequency is documented in the data: over 80% of reverted transactions on L2s are MEV bots, many of them failed sandwich attempts.

The functional origin: front-running in traditional markets

Front-running is as old as markets. A broker receives a large client order. The broker knows the order will move the price. The broker buys for their own account before executing the client order, then sells after the price moves. The broker profits at the client's expense. The practice is illegal in most regulated markets. The illegality is enforced by surveillance and prosecution.

In crypto, front-running is not illegal. It is profitable. The enforcement mechanism is not law. It is code. The victim's only defense is slippage limits and private transaction submission. The private submission — through Flashbots or similar relays — hides the transaction from the public mempool. The sandwich attacker cannot see it. The sandwich attack requires visibility. Eliminate the visibility. Eliminate the attack.

The defense

Slippage limits. The most basic defense. Set the maximum acceptable price movement low. The lower the slippage tolerance, the less profit available to the sandwich attacker. The trade-off: too low, and the transaction reverts on normal volatility. The optimal slippage depends on the pool's liquidity, the trade size, and the current volatility. Wallets like MetaMask now suggest optimal slippage based on recent pool behavior. The suggestion is algorithmic. The algorithm is a defense.

Private transaction submission. Flashbots Protect and similar services route transactions directly to block builders, bypassing the public mempool. The sandwich attacker cannot see the transaction. The attack requires visibility. Eliminate visibility. Eliminate attack. The cost: the transaction may take slightly longer to include — private relays have different inclusion guarantees than the public mempool. The trade-off is speed vs. safety. For non-urgent transactions, private submission is the correct choice.

Batch auctions. Protocols like CoWSwap and 1inch Fusion aggregate orders and execute them in batches at a uniform clearing price. All orders in the batch receive the same price. There is no ordering within the batch. There is no sandwich. The batch auction is the architectural solution to sandwich attacks. The architecture eliminates the attack vector. The vector is eliminated by design, not by defense.

The reference

Phil Daian et al., "Flash Boys 2.0" (2019). The paper documented sandwich attacks on Ethereum DEXs and measured their prevalence and profitability. It named the phenomenon. The name became the field. The field is MEV. The sandwich is the most visible form of MEV. The visibility is the subject of this post. The invisibility is the subject of the next.


References:

  • Phil Daian et al., "Flash Boys 2.0," 2019.
  • Flashbots, "MEV-Boost and Sandwich Attacks," Flashbots Documentation.
  • CoWSwap, "Batch Auctions," CoWSwap Documentation.
  • Related posts: MEV, Arbitrage

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

The Order Book

Every trade begins in an order book. Bids, asks, spread, depth, slippage. The order book is the oldest market microstructure and the foundation of all trading. Understanding it is the prerequisite for everything else: arbitrage, market making, MEV. The book is the game board. The moves happen on it.

dextradingorder-bookmarket-microstructurecrypto

Every financial market, whether a 17th-century Amsterdam commodities exchange or a 2026 Solana DEX, rests on the same primitive: a place where buyers and sellers state what they want and at what price. The order book is that place. It is the oldest market microstructure. It is the foundation of every trading strategy that follows.

An order book is a list of bids and asks. A bid is an offer to buy: "I will buy 10 ETH at $3,000 each." An ask is an offer to sell: "I will sell 10 ETH at $3,010 each." The difference between the highest bid and the lowest ask is the spread. The spread is the cost of immediacy — the premium you pay to trade now rather than wait. The spread exists because someone must provide liquidity. The liquidity provider posts bids and asks and waits. The liquidity taker crosses the spread to trade immediately.

The order book is organized by price level. Bids are sorted descending — the highest bid is the best bid. Asks are sorted ascending — the lowest ask is the best ask. The best bid and best ask together form the top of the book. Below them, deeper in the book, lie larger orders at worse prices. The depth of the book at each price level determines how much you can trade before the price moves against you. This is slippage. A deep book absorbs large orders without significant price movement. A shallow book moves sharply on modest volume.

The functional origin

The order book emerged from the physical trading floors of early modern Europe. The Amsterdam Stock Exchange, founded in 1602 by the Dutch East India Company, was the first permanent market for securities. Traders gathered in a courtyard. They shouted bids and offers. A clerk recorded the trades. The shouting was the order book. The clerk was the exchange.

The London Stock Exchange, founded in 1801, formalized the process. Members — jobbers and brokers — operated under rules. Jobbers made markets: they quoted two prices, a bid and an ask, and stood ready to trade at those prices. Brokers represented clients and executed against jobbers' quotes. The jobber's quote was the order book in miniature. The spread between the bid and the ask was the jobber's profit. The jobber who quoted too wide a spread lost business to other jobbers quoting narrower spreads. The jobber who quoted too narrow a spread absorbed adverse selection — informed traders traded against them when the price was about to move. The jobber's art was balancing spread income against adverse selection losses. That art is now algorithmic. The art is market making. The algorithms are the subject of later posts in this series.

The electronic order book emerged in the 1980s. NASDAQ introduced the Small Order Execution System in 1984, allowing automated execution of small orders. The London Stock Exchange introduced SEAQ, a screen-based quote system, in 1986. The electronic order book eliminated the trading floor. The shouting was replaced by a data structure. The data structure was an order book in memory. The market makers were now algorithms posting quotes to that data structure. The transition from floor to screen was the transition from art to engineering. The engineering is what we trade on today.

The order book in crypto

Crypto exchanges — both centralized (Binance, Coinbase, Kraken) and decentralized (Serum, dYdX, Hyperliquid) — use electronic order books. The data structure is the same as NASDAQ's. The differences are in settlement and custody.

On a centralized exchange, the order book is a database managed by the exchange. You deposit funds. The exchange credits your account. You place orders. The exchange matches them. You withdraw funds. The exchange is the custodian. The exchange is the counterparty. You trust the exchange. The trust has been violated many times. The violations are the reason for DEXs.

On a decentralized exchange with an order book, the order book is on-chain. Orders are transactions. Matching is performed by the chain's validators or by an off-chain matching engine that settles on-chain. Serum, on Solana, uses an on-chain order book. The chain maintains the order book state. The validators execute matching logic. dYdX, on its own Cosmos chain, uses an off-chain order book with on-chain settlement — the matching engine runs on dYdX's validators, trades settle to the chain. The model is hybrid. The order book is off-chain for speed. The settlement is on-chain for trustlessness.

The order book's information content

The order book is not just a matching engine. It is an information source. The shape of the book — the distribution of orders across price levels — reveals market sentiment. A book skewed heavily to the bid side suggests buying pressure. A book skewed to the ask side suggests selling pressure. A balanced book suggests equilibrium. The information is probabilistic. It is also actionable. Market-making algorithms read the book to set their quotes. Arbitrage algorithms read the book to detect cross-venue discrepancies. MEV searchers read the book to identify profitable extraction opportunities. The book is the input. The strategy is the output. The book is the game board. The moves happen on it.

Larry Harris's Trading and Exchanges: Market Microstructure for Practitioners (2003) is the canonical reference on order book mechanics. Harris was the SEC's chief economist. His book explains every aspect of market microstructure: order types, priority rules, trading costs, market maker obligations, dealer markets vs. auction markets, transparency, fragmentation. It is 600 pages. It is written for practitioners. Every concept in this series — arbitrage, market making, MEV — is built on the microstructure Harris describes. The microstructure changed from floor to screen. The principles didn't. The principles apply to crypto. The crypto application is the subject of the posts that follow.


References:

  • Larry Harris, Trading and Exchanges: Market Microstructure for Practitioners, Oxford University Press, 2003.
  • Maureen O'Hara, Market Microstructure Theory, Blackwell, 1995.
  • Related posts: Algorithmic trading in crypto

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

Maximal Extractable Value

MEV is the profit extractable from ordering, including, or excluding transactions within a block. It is the dark forest of crypto — an adversarial environment where every transaction in the mempool is prey. Phil Daian named it in 2019. The name stuck. The phenomenon is the economic layer of blockchain consensus.

dexmevflashbotsfront-runningsandwich-attacks

Maximal Extractable Value — MEV — is the profit that can be extracted by controlling transaction ordering within a block. The term was coined by Phil Daian and colleagues in the 2019 paper "Flash Boys 2.0." The name is a deliberate reference to Michael Lewis's Flash Boys, which documented the high-frequency trading arms race in traditional equity markets. The arms race migrated to crypto. The stakes are the same. The speed is higher.

MEV exists because blockchains are not instant. Transactions are broadcast to a mempool — a waiting room where they sit until a block producer includes them in a block. While in the mempool, transactions are visible. Their contents are known. The block producer can choose which transactions to include, in what order. The choice is economic. Transactions with higher fees are more attractive to include. Transactions that create profit opportunities — arbitrage, liquidation, sandwich attacks — can be front-run, back-run, or sandwiched. The profit is the MEV. The extractor is the searcher. The victim is the user whose transaction created the opportunity.

The functional origin: Flash Boys

Michael Lewis's Flash Boys: A Wall Street Revolt (2014) told the story of the high-frequency trading revolution in U.S. equity markets. The key discovery: the physical distance between exchanges created arbitrage opportunities that could be exploited by traders with faster connections. The traders built microwave towers, laid fiber-optic cable in the straightest possible lines, and colocated servers in exchange data centers. The speed advantage was measured in microseconds. The profit was measured in billions.

The crypto parallel is exact. The mempool is the digital equivalent of the physical distance between exchanges. Transactions in the mempool are visible. A searcher who can read the mempool faster, simulate the transaction's effect, and submit a profitable counterpart transaction before the original is included — that searcher extracts the MEV. The speed advantage is measured in milliseconds. The profit is measured in hundreds of millions.

The difference: in traditional markets, the arms race was infrastructure — microwave towers, fiber routes, exchange colocation. In crypto, the arms race is also infrastructure — but the infrastructure is mempool access, private relay connections, and integration with block builders. The traditional HFT firm needed a microwave license. The crypto MEV searcher needs a connection to a block builder. The barrier is different. The dynamic is the same.

The forms of MEV

Front-running. A searcher sees a large buy order in the mempool. The searcher submits their own buy order for the same token with a higher gas price, getting included before the victim. The searcher's buy moves the price up. The victim's buy executes at the higher price. The searcher sells immediately after, pocketing the difference. The victim paid more. The searcher extracted the difference. The extraction is front-running.

Sandwich attacks. A variant. The searcher buys before the victim and sells after. The victim's trade is sandwiched between the searcher's two trades. The victim buys at an artificially elevated price. The searcher profits from both legs. The sandwich is the most profitable form of MEV for liquid token pairs with active mempools.

Back-running. A searcher sees a trade that will move the price. The searcher submits a trade after the victim's trade, profiting from the price movement. Back-running is less profitable than front-running — the price has already moved — but less risky — the searcher doesn't need to predict the direction, only react to it. Liquidation of undercollateralized loans is a form of back-running: the searcher sees the price update that triggers the liquidation and submits the liquidation transaction immediately after.

Just-in-time liquidity. A searcher sees a large swap in the mempool. They deposit liquidity into the pool just before the swap executes, earn the swap fees, and withdraw immediately after. The LP earns fees with zero inventory risk. JIT liquidity is a form of MEV extraction that benefits the extractor at the expense of passive LPs, whose fee income is diluted.

The infrastructure: Flashbots

Flashbots was launched in 2020 by Phil Daian, Stephane Gosselin, and colleagues. It is a research and development organization focused on MEV. Its primary product: MEV-Boost, a middleware that separates block building from block proposing. Block builders construct blocks. Block proposers — validators — choose which block to propose. MEV-Boost allows validators to auction their block space to builders. Builders compete to offer the most valuable block. Validators earn the MEV. The auction democratizes MEV access — instead of requiring every validator to run MEV extraction infrastructure, they can outsource to builders and capture the value through competition.

Flashbots also operates a private relay. Searchers submit bundles — groups of transactions that must be executed atomically and in order — to the relay. The relay forwards bundles to builders. The bundles are not broadcast to the public mempool. The privacy prevents front-running of the searcher's own transactions. The relay is the infrastructure that makes MEV extraction possible without the searcher's strategies being copied or front-run.

The Flashbots model has been criticized for centralizing block building. A small number of builders dominate the market. The builders integrate vertically with searchers. The integration concentrates MEV extraction. The concentration creates a new class of intermediary between users and validators. The intermediary extracts rent. The rent is paid by users. The concentration is the subject of ongoing research and regulatory attention.

The reference

Phil Daian et al., "Flash Boys 2.0: Frontrunning, Transaction Reordering, and Consensus Instability in Decentralized Exchanges" (2019). The paper that named MEV. It documented the prevalence of front-running and sandwich attacks on Ethereum DEXs, measured the extracted value, and proposed architectural responses. The paper is the foundation of MEV research. Every subsequent paper in the field cites it. The phenomenon it documented has grown by orders of magnitude. The growth is the subject of the posts that follow.


References:

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

Market Making

A market maker posts bids and asks and stands ready to trade. The profit is the spread. The risk is adverse selection — informed traders who trade against you when the price is about to move. The Avellaneda-Stoikov model formalized this trade-off in 2008. It is now the standard model for algorithmic market making in both traditional and crypto markets.

dexmarket-makingavellaneda-stoikovadverse-selectioninventory

A market maker provides liquidity. They post a bid — a price at which they will buy. They post an ask — a price at which they will sell. They earn the spread between the bid and the ask. They lose when the price moves against their inventory. The market maker's art is setting the bid and ask such that spread income exceeds inventory losses over time.

The market maker is the counterparty to every impatient trader. The trader who must buy now crosses the spread and pays the ask. The trader who must sell now crosses the spread and receives the bid. The market maker absorbs the order flow imbalance. When there are more buyers than sellers, the market maker accumulates a short position — selling to buyers, hoping the price doesn't rise too far before sellers arrive. When there are more sellers than buyers, the market maker accumulates a long position. The inventory is the risk. The spread is the compensation for bearing it.

The functional origin: the jobber

The market maker's role emerged on the London Stock Exchange in the 19th century. The jobber was a member who made markets in specific securities. The jobber quoted two prices — the bid and the ask — and stood ready to trade at those prices with any broker who approached. The jobber did not deal with the public. The jobber dealt only with brokers. The brokers dealt with the public. The jobber's profit was the spread. The jobber's risk was adverse selection: a broker with better information about the security's true value would trade against the jobber's quote, profiting at the jobber's expense.

The jobber's defense was the spread. A wider spread compensated for higher adverse selection risk. A narrower spread attracted more order flow. The optimal spread balanced the two. The jobber who set spreads too wide lost business. The jobber who set spreads too narrow lost money to informed traders. The balance was learned through experience. The learning was trial and error. The errors were expensive. The experience was the jobber's edge.

The electronic market maker of today is the jobber, automated. The spread is set by an algorithm. The algorithm reads the order book, estimates adverse selection risk, tracks inventory, and adjusts quotes in microseconds. The algorithm is faster than any human jobber. It is also less judgmental. The human jobber could sense when a broker was informed — the broker was nervous, the broker was eager, the broker asked for an unusually large size. The algorithm sees only the order flow. The signal is statistical. The edge is quantitative. The quantification is the subject of this post.

The Avellaneda-Stoikov model

Marco Avellaneda and Sasha Stoikov published "High-Frequency Trading in a Limit Order Book" in 2008. The paper formalized the market maker's problem as stochastic optimal control. The market maker chooses bid and ask quotes to maximize expected utility of terminal wealth, subject to inventory risk and adverse selection. The solution is a closed-form approximation: the optimal bid and ask are functions of the market maker's current inventory, the volatility of the asset, the market maker's risk aversion, and the intensity of order arrival.

The key insight: the market maker should skew quotes away from their inventory. If the market maker is long — has bought more than they've sold — they should lower their bid (less willing to buy more) and lower their ask (more willing to sell). The skew reduces inventory risk. If the market maker is short, they should raise both bid and ask. The spread widens with volatility and risk aversion. The spread narrows with order arrival intensity — more competition means tighter spreads.

The model is implemented in production market-making systems. It is the standard. It is taught in quantitative finance programs. It is adapted for crypto with modifications: gas costs replace exchange fees, AMM curves replace order book depth, and the discrete block time of blockchains replaces the continuous time of traditional markets. The adaptations are engineering. The core model is the same. The model works. The work is in the calibration.

Market making in crypto

Crypto market making has three forms:

CEX market making. Traditional order book market making on centralized exchanges. The market maker runs a server colocated with the exchange, streams order book updates, adjusts quotes in microseconds. The model is Avellaneda-Stoikov with exchange-specific calibrations. The competition is intense. The margins are thin. The thin margins are the evidence of competition.

AMM liquidity provision. Providing liquidity to a constant-function market maker. The LP deposits tokens into a pool. The pool's formula sets the price. The LP earns fees from trades. The LP bears impermanent loss. The LP's problem is similar to the market maker's: earn fee income while managing inventory risk. The difference: the AMM LP doesn't set the spread. The formula sets the spread. The LP only chooses the range (in Uniswap V3) and the amount. The LP's optimization is passive — choose parameters, deposit, wait. The active management is the subject of concentrated liquidity strategies.

On-chain order book market making. Running a market-making bot on a DEX with an on-chain order book (Serum, dYdX, Hyperliquid). The bot posts bids and asks as transactions. Each quote update costs gas. The gas cost constrains the update frequency. The constraint creates a trade-off: update frequently for tighter spreads (higher gas cost, more accurate quotes) or update infrequently to save gas (wider spreads to protect against adverse selection). The trade-off is the gas cost of decentralization. The gas cost is paid by the market maker. The cost is passed to traders in wider spreads. The wider spreads are the price of trustlessness.

The reference

Marco Avellaneda and Sasha Stoikov, "High-Frequency Trading in a Limit Order Book," Quantitative Finance, 2008. The foundational paper on algorithmic market making. It is 25 pages. It assumes knowledge of stochastic calculus. The key result — the optimal quote skew formula — is implementable in a few lines of code. The implementation is the easy part. The calibration — estimating volatility, order arrival intensity, adverse selection — is the hard part. The calibration is the edge. The edge is the market maker's competitive advantage. The advantage erodes as competitors adopt the same model. The erosion is the subject of the arms race.


References:

  • Marco Avellaneda and Sasha Stoikov, "High-Frequency Trading in a Limit Order Book," Quantitative Finance, 2008.
  • Larry Harris, Trading and Exchanges, Oxford University Press, 2003.
  • Related posts: AMMs, Arbitrage

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

Latency and Infrastructure

In crypto trading, latency is the edge. The bot with the fastest mempool access, the lowest-latency path to the sequencer, the colocated server wins the arbitrage. The difference is milliseconds. The investment in infrastructure is the rent paid for speed. The rent is extracted from slower traders.

dexlatencyinfrastructuremempoolcolocation

Latency is the single most important competitive variable in algorithmic trading. Strategy matters. Risk management matters. Capital matters. But given equivalent strategy and risk management and capital, the faster participant wins. The win is measured in milliseconds. The investment to achieve that speed is measured in millions.

The latency stack for crypto trading has four layers. Each layer is a source of delay. Each layer is an opportunity for optimization.

Network latency. The time for a packet to travel from the trader's server to the exchange's matching engine or the blockchain's sequencer. Minimized by physical proximity — colocating in the same datacenter as the exchange or sequencer — and by network topology — choosing the shortest path, avoiding congested routes, using dedicated fiber. The speed of light in fiber is approximately 200,000 km/s. Over 1,000 km, that is 5 milliseconds. Over 100 km, 0.5 milliseconds. The difference between a server in Frankfurt and a server in London trading on a Frankfurt-based exchange is 5 milliseconds. In that 5 milliseconds, the price can move. The edge is gone. The trade loses.

Mempool latency. On blockchains, transactions are not sent directly to the block producer. They are broadcast to a mempool — a peer-to-peer network of nodes. The time between submitting a transaction and it being visible to the block producer depends on the node's position in the network topology, the number of hops to the producer, and the propagation delay at each hop. A trader who runs their own node, connected directly to high-staked validators or sequencers, sees transactions before a trader relying on public RPC endpoints. The private node is the latency edge. The edge is purchased through infrastructure.

Execution latency. The time between the transaction arriving at the exchange or sequencer and being included in a block or matched. On centralized exchanges, this is the matching engine's processing time — typically microseconds. On blockchains, this is the block time — the interval between blocks. On Ethereum L1, block time is 12 seconds. On Solana, 400 milliseconds. On Arbitrum, sub-second. The block time is the minimum latency for on-chain execution. Faster chains enable faster strategies. The chain choice is a latency decision.

State latency. The time between a state change occurring and the trader's system being aware of it. A trade on Uniswap changes the pool's price. The change is included in a block. The block is propagated through the network. The trader's node receives the block, updates its state, and triggers the strategy. The delay between the trade occurring and the strategy reacting is state latency. Minimized by running a full node, subscribing to block production directly, and processing state updates in parallel with strategy evaluation. The full node is expensive to run. The expense is the cost of being fast.

The functional origin: the transatlantic cable

The latency arms race in finance began in the 19th century. The first transatlantic telegraph cable, laid in 1866, reduced communication time between London and New York from weeks (by ship) to minutes (by telegraph). The cable was used for arbitrage: prices in London and New York could be compared in near-real-time. The trader who received the cable first could trade before the information was widely known. The cable was the latency edge.

In the 1980s, fiber-optic cables replaced copper. In the 2000s, microwave networks replaced fiber for the most latency-sensitive routes — microwaves travel through air at the speed of light, which is faster than light through glass fiber. In the 2010s, laser networks in space were proposed for intercontinental routes. The arms race is continuous. The technology changes. The principle doesn't: faster information means faster reaction, faster reaction means profitable trades, profitable trades pay for the infrastructure. The infrastructure is the rent. The rent is extracted from slower participants.

Michael Lewis's Flash Boys (2014) documented the modern latency arms race in U.S. equity markets. The key discovery: a new fiber route between Chicago and New York, laid as straight as physically possible through mountains and under rivers, reduced latency by 3 milliseconds. The route cost $300 million. The route was built by a trading firm. The firm's name was Spread Networks. The name was the strategy. The spread was the profit. The network was the means.

The crypto latency stack today

Crypto latency infrastructure is evolving along the same trajectory as traditional finance, compressed into a decade instead of a century. The major developments:

Private relays. Flashbots and similar services provide private transaction submission, bypassing the public mempool. The private relay reduces mempool latency to near-zero — the transaction goes directly to the builder. The privacy also prevents front-running of the trader's own transactions. The relay is the infrastructure. The fee is the cost.

Sequencer colocation. On L2 rollups, the sequencer orders transactions. Colocating with the sequencer provides minimum network latency. The colocation is offered by some L2s as a paid service. The service is the latency edge. The edge is purchased.

Validator connections. On proof-of-stake chains, validators propose blocks. A trader with direct connections to high-staked validators can submit transactions that are included in the next block with higher probability. The connections are built through relationships, infrastructure sharing, and direct payments. The connections are the edge. The edge is relational.

Chain-specific optimization. Each chain has different latency characteristics. Solana's Gulf Stream forwards transactions to validators before the current block is finalized. Arbitrum's sequencer orders transactions on a first-come, first-served basis. The optimal strategy varies by chain. The trader who understands the chain-specific latency model has an edge over the trader who treats all chains as equivalent. The understanding is the edge. The edge is informational.


References:

  • Michael Lewis, Flash Boys: A Wall Street Revolt, W.W. Norton, 2014.
  • Phil Daian et al., "Flash Boys 2.0," 2019.
  • "First-Spammed, First-Served: MEV Extraction on Fast-Finality Blockchains," June 2025.
  • Related posts: MEV, Arbitrage

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

Flash Loans

A flash loan lets you borrow millions with zero collateral, as long as you repay in the same transaction. It is the innovation that made atomic arbitrage possible at scale. It is also the weapon of choice for protocol exploits. The same mechanism that enforces the Law of One Price can drain a lending protocol in 12 seconds.

dexflash-loansarbitrageatomicitydefi

A flash loan is an uncollateralized loan that must be borrowed and repaid within a single blockchain transaction. If the loan is not repaid by the end of the transaction, the entire transaction reverts. The loan is never actually disbursed. The borrower pays a fee. The lender earns the fee with zero credit risk — the atomicity of the transaction guarantees either full repayment or full reversion. There is no partial state. There is no default. The mechanism is pure atomicity.

Flash loans were introduced by Marble Protocol in 2018 and popularized by Aave and dYdX in 2019-2020. The innovation was not the concept — uncollateralized intraday credit has existed in traditional finance for centuries. The innovation was the enforcement mechanism: in traditional finance, uncollateralized credit requires trust, legal contracts, and recourse to courts. In DeFi, it requires a single line of Solidity: require(repayment >= loan, "not repaid"). The enforcement is code. The code is the court.

The functional origin: intraday credit

In traditional finance, brokers extend intraday credit to clients. A hedge fund buys $10 million of stock in the morning and sells it in the afternoon. The broker lends the $10 million for a few hours. The credit is uncollateralized — the broker trusts the fund to settle by end of day. If the fund doesn't settle, the broker has legal recourse. The recourse is slow and expensive. The trust is the cost.

The continuous linked settlement (CLS) system in foreign exchange, launched in 2002, addressed a similar problem: FX trades settle in different currencies at different times, creating settlement risk — the risk that one party pays but the other doesn't. CLS uses payment-versus-payment (PvP) settlement: both legs settle simultaneously or neither settles. The simultaneity eliminates settlement risk. The mechanism is atomicity. Flash loans are CLS for DeFi — atomic settlement without a central clearinghouse. The blockchain is the clearinghouse. The atomicity is the mechanism.

How flash loans enable arbitrage

Arbitrage requires capital. The price difference between two pools might be 0.5%. To extract meaningful profit, the arbitrageur needs to trade large amounts. Large amounts require capital. The capital has an opportunity cost — it could be deployed elsewhere. The opportunity cost reduces the net return on arbitrage. Flash loans eliminate the capital requirement. The arbitrageur borrows the capital, executes both legs, repays the loan, and keeps the profit. The capital is never at risk. The opportunity cost is zero. The arbitrageur's only cost is gas + flash loan fee. The fee is typically 0.09% on Aave. The spread must exceed the fee to be profitable. The spread that doesn't exceed the fee persists. The spread that does is extracted.

The flash loan democratized arbitrage — in theory. Anyone with a smart contract can borrow millions and execute an arbitrage. In practice, the democratization was limited by the same latency and infrastructure constraints that concentrate all MEV extraction. The flash loan solves the capital problem. It doesn't solve the speed problem. Speed still wins. Speed requires infrastructure. Infrastructure requires capital. The democratization is partial. The partial democratization is the state of the market.

Flash loans as attack vectors

The same atomicity that enables arbitrage enables attacks. A flash loan can be used to manipulate the price of a governance token, borrow against the inflated collateral, drain the lending protocol, and repay the flash loan — all in one transaction. The attack requires no capital. The attacker pays only gas + flash loan fee. If the attack succeeds, the profit can be millions. The protocol is left with bad debt. The attacker is untraceable. The transaction was atomic. The exploitation was instantaneous.

The most famous flash loan attacks: bZx (February 2020, $1M), Harvest Finance (October 2020, $34M), Cream Finance (October 2021, $130M), Beanstalk (April 2022, $182M). Each attack used flash loans to amass voting power or manipulate prices. Each attack was atomic. Each attack exploited the gap between the protocol's economic assumptions and the reality of atomic composability: the protocol assumed that accumulating a controlling stake required capital. Flash loans made the assumption false. The assumption was the vulnerability.

The flash loan is a tool. It is neutral. It enables arbitrage that enforces the Law of One Price. It also enables attacks that destroy protocols. The tool is the same. The use determines the outcome. The outcome is a function of the protocol's design. The design must account for atomic composability. Most protocols don't. The ones that don't are exploited. The ones that do survive. The selection is evolutionary. The evolution is the market.

The reference

Kaihua Qin, Liyi Zhou, and Arthur Gervais, "Quantifying Blockchain Extractable Value: How Dark is the Forest?" (2021). This paper quantified the prevalence of MEV extraction on Ethereum, including flash loan-based arbitrage and attacks. The paper documented the concentration of extraction, the profitability of different strategies, and the arms race dynamics. It is the quantitative complement to Daian et al.'s "Flash Boys 2.0." The numbers are the evidence. The evidence supports the narrative. The narrative is: the dark forest is real, it is concentrated, and it is accelerating.


References:

  • Aave, "Flash Loans," Aave Documentation.
  • Kaihua Qin, Liyi Zhou, Arthur Gervais, "Quantifying Blockchain Extractable Value: How Dark is the Forest?" 2021.
  • Marble Protocol, "Flash Lending," 2018.
  • Related posts: Arbitrage, MEV

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

Concentrated Liquidity and JIT Attacks

Uniswap V3 let LPs concentrate liquidity in a price range. Capital efficiency multiplied. Passive LPs were replaced by active managers. And a new form of MEV emerged: just-in-time liquidity, where an attacker provides liquidity for exactly one block, earns the fees, and exits. The LP game changed forever.

dexuniswap-v3concentrated-liquidityjitmev

Uniswap V2 required liquidity providers to deposit tokens across the entire price curve — from zero to infinity. Most of that liquidity was never used. In a stablecoin pair like USDC/DAI, the price rarely deviates from 1:1. Liquidity deposited at a price of 1:2 or 2:1 sat idle, earning no fees, providing no benefit. The capital was wasted. The waste was the cost of passive liquidity provision.

Uniswap V3, launched in May 2021, changed this. LPs choose a price range. Their liquidity is only active within that range. Within the range, the LP earns fees proportional to their share of the active liquidity. Outside the range, the LP earns nothing. The capital efficiency gain is dramatic: an LP who provides liquidity in a narrow range around the current price can achieve the same depth as a V2 LP with a fraction of the capital. Alternatively, an LP can provide far greater depth around the current price with the same capital — reducing slippage for traders and earning more fees.

The trade-off: the LP must actively manage the range. If the price moves outside the range, the LP's position becomes inactive. The LP must withdraw and redeposit in a new range. Each reposition costs gas. The LP who manages aggressively earns higher fees but pays higher gas. The LP who manages passively earns lower fees but pays lower gas. The optimal frequency depends on fee income, gas costs, and price volatility. The optimization is quantitative. The LP is now a market maker.

The functional origin: tick sizes and decimalization

The concept of discrete price levels is not new. Traditional exchanges have tick sizes — the minimum price increment. A stock might trade in penny increments. The tick size constrains where orders can be placed. Decimalization — the switch from fractions (1/8, 1/16) to decimals (0.01) — reduced tick sizes in U.S. equity markets in 2001. The reduction narrowed spreads. The narrower spreads reduced market maker profits. The reduced profits drove consolidation among market makers.

Uniswap V3's tick-based liquidity is the crypto equivalent. The continuous price curve of V2 is discretized into ticks. LPs provide liquidity between ticks. The ticks are the price grid. The grid is finer than traditional tick sizes — 1 basis point (0.01%) per tick. The fineness enables precise range selection. The precision enables capital efficiency. The efficiency is the innovation.

Just-in-time liquidity

Concentrated liquidity enabled a new form of MEV: just-in-time (JIT) liquidity. A searcher sees a large pending swap in the mempool. The searcher deposits concentrated liquidity at the exact tick range the swap will traverse, in the same block as the swap. The swap executes, traversing the searcher's liquidity. The searcher earns the fees — typically the majority of the swap's fee, since the searcher provided most of the active liquidity in that range. The searcher withdraws the liquidity in the same block. The searcher earns swap fees with near-zero inventory risk — the position existed for a single block.

The victim: passive LPs whose fee income is diluted. The swap would have earned them fees. Instead, the JIT LP captured those fees. The passive LP provided liquidity continuously, paid gas to deposit and withdraw, bore impermanent loss — and earned less because a JIT LP front-ran their fee income. The JIT LP extracted the most profitable slice of the fee stream: the large swaps that would have generated the highest fees per unit of liquidity. The passive LP got the residual — small swaps, unpredictable swaps, swaps too small for JIT to be profitable. The passive LP's returns declined. The decline is structural. The structure is the concentration of MEV extraction.

The response

The JIT problem is an instance of a broader issue: concentrated liquidity enables capital efficiency, but capital efficiency enables MEV extraction at the expense of passive LPs. The solutions are architectural:

Fee structures that penalize short-term liquidity. If swap fees accrued linearly over time — the longer your liquidity is active, the larger your share of fees — JIT LPs would earn less per block. The accrual mechanism would favor long-term LPs. Uniswap V4, currently in development, is expected to introduce more flexible fee structures.

Batch auctions that eliminate ordering. As with sandwich attacks, batch auctions eliminate the ordering within a block that enables JIT. All swaps in a batch execute at a uniform clearing price. All LPs in the batch earn proportional fees. There is no "before" and "after" within the batch. There is no JIT.

Passive LP vaults. Protocols like Arrakis and Gamma manage concentrated liquidity positions on behalf of passive LPs. The vault rebalances ranges, compounds fees, and optimizes for fee income net of gas costs. The vault is a market maker operated by an algorithm. The algorithm competes with JIT searchers on equal footing — both are automated, both are fast, both can reposition in a single block. The vault democratizes active LP management. The democratization is partial. The infrastructure cost remains. The infrastructure is the barrier.

The reference

Hayden Adams, Noah Zinsmeister, Dan Robinson, "Uniswap v3 Core" (2021). The whitepaper that introduced concentrated liquidity. It is 8 pages. The key insight — LPs choose a price range — is stated in the first paragraph. The rest is implementation. The implementation is now the dominant AMM design, copied by nearly every DEX launched since 2021. The copying is the evidence of the idea's power. The idea was simple. The implications — JIT, active management, LP stratification — are still unfolding.


References:

  • Hayden Adams, Noah Zinsmeister, Dan Robinson, "Uniswap v3 Core," 2021.
  • Arrakis Finance, "Concentrated Liquidity Vaults," Arrakis Documentation.
  • Related posts: AMMs, Market Making, MEV

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

CEX-DEX Arbitrage

CEX-DEX arbitrage is the dominant strategy in crypto. A token trades at different prices on Binance and Uniswap. The arbitrageur buys on one, sells on the other. $233M extracted in 18 months by 19 searchers. Three of them captured 75%. This post explains the strategy, the infrastructure, and why the concentration is structural.

dexcex-dexarbitragesearchersconsolidation

CEX-DEX arbitrage is the most empirically significant trading strategy in crypto. A token trades on a centralized exchange (Binance, Coinbase, Kraken) and a decentralized exchange (Uniswap, Curve, SushiSwap). The prices differ. The arbitrageur buys on the cheaper venue, sells on the dearer, pockets the spread. The trade is the mechanism that enforces the Law of One Price across the CEX-DEX boundary.

The strategy is not atomic. The CEX leg and the DEX leg execute on different systems, at different speeds, with different failure modes. The CEX leg requires an account, API keys, and compliance with the exchange's rules. The DEX leg requires a wallet, gas, and smart contract execution. The two legs cannot be bundled into a single transaction. The arbitrageur bears execution risk.

The AFT 2025 study of CEX-DEX arbitrage on Ethereum, covering August 2023 to March 2025, provides the definitive empirical picture:

Metric Value
Total value extracted $233.8 million
Total arbitrage transactions ~7.2 million
Number of major searchers 19
Top 3 searcher share ~75% of volume and value
Daily transaction growth 7.2× from Q3 2023 to Q1 2025
Dominant searchers Wintermute, SCP, Kayle

The concentration is extreme. Three searchers capture three-quarters of the value. The concentration has been increasing over time. The increase is structural. The structure is the infrastructure cost.

The infrastructure

CEX-DEX arbitrage requires infrastructure on both sides of the trade.

CEX side. Low-latency API access to Binance, Coinbase, Kraken. Colocation with the exchange's matching engine for minimum latency. Multiple accounts to avoid rate limits. Inventory of tokens on each exchange to enable immediate execution — you can't wait for a deposit to clear. The inventory is capital at risk. The capital must be managed across exchanges.

DEX side. Direct node access to the blockchain for mempool monitoring. Integration with block builders (Flashbots, MEV-Boost) for reliable transaction inclusion. Smart contracts optimized for gas efficiency. Flash loan integration for capital-free execution when the arbitrage is on-chain atomic. Gas price monitoring and dynamic fee adjustment.

The bridge between them. A system that monitors prices on both CEXs and DEXs in real time, identifies arbitrage opportunities, calculates profitability net of gas and fees, and executes the trade. The system must operate at the speed of the faster venue — if the CEX price updates before the DEX transaction confirms, the opportunity disappears. The window is milliseconds. The system that sees the window first, calculates profitability first, and executes first, wins. The system that wins consistently captures most of the value. The winner is the one with the best infrastructure.

The consolidation

The AFT study documented a trend: searcher-builder vertical integration is deepening. Integrated searchers operate at lower margins, sometimes negative net profit, subsidized by revenue sharing with affiliated builders. The subsidy allows them to win arbitrages that independent searchers cannot profitably compete for. The independent searchers are driven out. The concentration increases.

The mechanism: a builder operates a block construction service. The builder's affiliated searcher submits arbitrage bundles to the builder. The builder includes the bundle and shares the MEV revenue with the searcher. The searcher can bid more aggressively for arbitrage opportunities because part of the revenue comes back through the builder relationship. The independent searcher has no such arrangement. The independent searcher must bid less to remain profitable. The independent searcher wins fewer arbitrages. The independent searcher's volume declines. The decline feeds back: lower volume means less data for model calibration, means worse execution, means even lower volume. The feedback loop drives consolidation.

This is the same dynamic that occurred in traditional equity markets, documented by Lewis in Flash Boys. The high-frequency trading firms that invested in infrastructure — colocation, private fiber, microwave networks — captured an increasing share of trading volume. The firms that didn't invest were driven out. The market consolidated around a few players. The players extracted rents. The rents were paid by investors in the form of wider effective spreads. The dynamic is repeating in crypto. The names are different. The structure is the same.

The reference

"The CEX-DEX Arbitrage Landscape on Ethereum" (AFT 2025) is the definitive empirical study. It covers 18 months, 7.2 million transactions, $233.8 million in extracted value. It documents the consolidation trend, the searcher-builder vertical integration, and the implications for market efficiency and decentralization. It is the quantitative foundation for any discussion of arbitrage in crypto. The numbers are the story. The story is consolidation. The consolidation is accelerating.


References:

  • AFT 2025, "The CEX-DEX Arbitrage Landscape on Ethereum: 2023–2025."
  • Michael Lewis, Flash Boys, W.W. Norton, 2014.
  • Related posts: Arbitrage, MEV, Latency

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

Arbitrage

Arbitrage is the oldest strategy in finance: buy cheap, sell dear, simultaneously. In crypto, flash loans made arbitrage available to anyone with code. The strategy is simple. The execution is not. The game is speed, and speed is infrastructure.

dexarbitrageflash-loanscex-dexmev

Arbitrage is the purchase and sale of the same asset in different markets to profit from price differences. It is the oldest trading strategy. It is also the purest: if two markets price the same thing differently, an arbitrageur can buy in the cheaper market and sell in the more expensive, pocketing the difference. The trade is riskless in theory — both legs execute at known prices, the profit is locked at execution. In practice, execution risk, latency, and competition make arbitrage a technological arms race.

The classical definition comes from Gustave de Molinari, a 19th-century French economist, but the practice is older than the definition. Medieval merchants arbitraged between markets separated by geography — buying spices in Venice, selling in Bruges. The internet collapsed geography. The price difference between New York and London narrowed to milliseconds. Crypto collapsed it further. The "markets" are now protocols on the same blockchain. The geography is the mempool. The distance is latency.

The functional origin: the Law of One Price

The Law of One Price states that identical goods should trade at identical prices in efficient markets. If they don't, arbitrageurs will buy the cheaper and sell the dearer until the prices converge. The arbitrageur's profit is the market's mechanism for enforcing the law. The law is not a law of nature. It is an equilibrium condition. The equilibrium is maintained by arbitrageurs. The arbitrageurs are paid for maintaining it. The payment is the spread.

The Law of One Price was formulated by William Stanley Jevons in The Theory of Political Economy (1871). Jevons observed that "in the same open market, at any one moment, there cannot be two prices for the same kind of article." The observation was empirical. The mechanism was arbitrage. The mechanism is the same today. The markets are now AMM pools and CEX order books. The Law of One Price is enforced by bots that execute in milliseconds. The bots are paid in extracted spread. The spread is the market's payment for maintaining efficiency.

Arbitrage in crypto

Crypto arbitrage takes several forms. The simplest is pool arbitrage: the same token pair trades at different prices in two AMM pools on the same chain. Buy in the cheaper pool, sell in the dearer pool, profit. The trade must be atomic — both legs in the same transaction — because the price will move after the first leg. A flash loan enables atomic execution. Borrow a large amount of token A. Execute the buy in pool 1. Execute the sell in pool 2. Repay the loan plus fee. Keep the profit. If any leg fails, the entire transaction reverts. The loan is never drawn. The risk is zero. The cost is gas + flash loan fee.

CEX-DEX arbitrage: a token trades at a different price on Binance than on Uniswap. The arbitrageur must execute on both venues simultaneously. This is not atomic — the CEX leg and the DEX leg are separate transactions on separate systems. The arbitrageur bears execution risk: one leg succeeds, the other fails. The risk is managed by speed — the faster you detect and execute, the less likely the price moves against you. The speed is the edge.

Cross-chain arbitrage: the same token trades on two different blockchains. Arbitrage requires bridging — sending the token from one chain to the other. The bridge takes time. During the bridge, the price may move. The arbitrageur bears bridge latency risk. The risk is managed by maintaining inventory on both chains and netting flows rather than bridging each trade.

Triangular arbitrage: three tokens. ETH → USDC → BTC → ETH. The product of the three exchange rates should equal one. If it doesn't, there is a triangular arbitrage. The strategy is common in forex. It is also common in crypto. It requires no external price feed. It is pure math on the chain.

The empirical reality

The CEX-DEX arbitrage market on Ethereum, measured from August 2023 to March 2025, extracted $233.8 million. Three searchers captured 75% of the volume. The daily transaction count grew 7.2× over the period. The concentration is extreme. The concentration is structural. The infrastructure cost — colocation with sequencers, low-latency mempool access, private relay connections — creates barriers to entry. The barriers produce concentration. The concentration raises the question: is the Law of One Price being enforced by a competitive market or by a cartel of the fastest?

The AFT 2025 study of CEX-DEX arbitrage documented a shift: searcher-builder vertical integration is deepening. Integrated searchers operate at lower margins, sometimes negative net profit, subsidized by builder revenue sharing. The subsidy drives out independent searchers. The concentration increases. The market becomes less competitive. The Law of One Price is still enforced. The enforcers are fewer. The fewer enforcers extract more rent. The rent is paid by traders in the form of wider effective spreads. The market is efficient. The efficiency is expensive.

The reference

Robert Shiller's Irrational Exuberance (2000) is not about arbitrage. It is about why arbitrage fails to enforce the Law of One Price at scale. Shiller documented persistent mispricings — the dot-com bubble, the housing bubble — that arbitrageurs could not or did not correct. The limits of arbitrage are the subject of Shleifer and Vishny's "The Limits of Arbitrage" (1997): arbitrage is risky, capital is constrained, and arbitrageurs who trade against bubbles can be wiped out before the bubble corrects. The limits are real in traditional markets. In crypto, the limits are different — not capital constraints but latency constraints, gas constraints, and infrastructure constraints. The constraints shape the market structure. The structure is the subject of the posts that follow.


References:

  • William Stanley Jevons, The Theory of Political Economy, 1871.
  • Andrei Shleifer and Robert Vishny, "The Limits of Arbitrage," Journal of Finance, 1997.
  • AFT 2025, "CEX-DEX Arbitrage on Ethereum: 2023–2025."
  • Related posts: The Order Book, Automated Market Makers

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

Automated Market Makers

The AMM replaced the order book with a formula. xy = k. No counterparties. No order matching. Just a curve and a pool. Uniswap's constant product formula is the most copied piece of financial mathematics of the 21st century. This post explains why it works, when it breaks, and what came after.

dexammuniswapconstant-productliquiditydefi

Before 2018, decentralized exchanges used order books. They were slow, illiquid, and expensive to run on-chain. Every order was a transaction. Every cancellation was a transaction. The order book was a data structure that required constant updates. On Ethereum, each update cost gas. The gas cost made order book DEXs economically unviable.

In 2018, Vitalik Buterin posted a proposal on Reddit: what if you replaced the order book with a formula? A pool of two tokens. A constant product. xy = k. The product of the quantities of the two tokens in the pool is constant. When you buy token A, you add token B to the pool and remove token A. The price of A, in terms of B, is the ratio of the quantities. The more A you buy, the more expensive A becomes — the curve moves, the ratio changes, you pay more. The formula enforces the price. The liquidity provider, not a counterparty, supplies the tokens. The provider earns fees from trades. The trader trades against the pool, not against another trader.

Hayden Adams implemented the proposal. He called it Uniswap. It launched in November 2018. The initial version had 300 lines of code. The constant product formula was the entire logic. The contract held two tokens. It allowed anyone to deposit tokens (become a liquidity provider), withdraw tokens (redeem their share), and swap tokens (trade against the pool). That was it. Three functions. Three hundred lines. The simplicity was the genius.

The mathematics

The constant product formula: x × y = k, where x is the quantity of token X in the pool, y is the quantity of token Y, and k is a constant. Before a trade, k = x × y. After a trade that adds Δx of token X and removes Δy of token Y, the new quantities are x + Δx and y - Δy. The product must remain constant: (x + Δx)(y - Δy) = k. Solving for Δy: Δy = y - k/(x + Δx). This is the amount of token Y the trader receives for depositing Δx of token X. The price is Δx/Δy, which varies with the size of the trade. Larger trades move the price more. The price impact is the mechanism that prevents the pool from being drained.

The fee is a percentage of the input amount, added to the pool before the trade is executed. In Uniswap V2, the fee is 0.3% — 30 basis points. The fee accrues to liquidity providers in proportion to their share of the pool. The fee is the incentive to provide liquidity. The fee must compensate LPs for the risk of impermanent loss — the opportunity cost of holding tokens in a pool rather than holding them directly. When the price of the tokens changes relative to each other, the LP's position is worth less than if they had held the tokens separately. The difference is impermanent loss. The fee income must exceed the impermanent loss or LPs withdraw. The pool shrinks. Liquidity dries up. The fee must be high enough to retain LPs. The optimal fee is an empirical question. The answer varies by pair, by volatility, by competing pools.

What came after

Uniswap V2 (2020) generalized the formula to any ERC-20 pair. V2 also introduced the price oracle — a time-weighted average price that resists manipulation. The oracle accumulates the price at each block, weighted by the time elapsed since the previous block. The accumulator is immune to flash loan manipulation because it is time-weighted. The oracle is the data feed for lending protocols, derivatives, and any application that needs a manipulation-resistant price.

Uniswap V3 (2021) introduced concentrated liquidity. Instead of providing liquidity across the entire price curve from zero to infinity, LPs choose a price range. Their liquidity is only active within that range. Within the range, they earn fees. Outside the range, they earn nothing. Concentrated liquidity multiplies capital efficiency — LPs can provide the same depth as V2 with a fraction of the capital, or provide far greater depth with the same capital. The trade-off: LPs must actively manage their ranges. Passive liquidity provision no longer works. The LP must monitor the price and adjust ranges. The LP is now a market maker. The market maker's job is the subject of a later post.

Curve (2020) introduced a different formula optimized for stablecoins — assets that trade at roughly 1:1. The Curve formula blends the constant product and constant sum (x + y = k) formulas. Near the peg, it behaves like a constant sum — low slippage, stable price. Far from the peg, it behaves like a constant product — the price diverges, the pool rebalances. The Curve formula enabled efficient stablecoin trading with minimal slippage. Curve became the dominant venue for stablecoin swaps and the foundation of the "Curve Wars" — the competition among protocols to accumulate CRV tokens and direct liquidity incentives.

The AMM as market microstructure

The AMM is not just a product. It is a market microstructure. It replaces the order book with a formula, the counterparty with a pool, the spread with a curve. The AMM is a market where anyone can provide liquidity without permission, anyone can trade without an account, and the rules are enforced by code rather than by an exchange. The AMM is the innovation that made DeFi possible. Without the AMM, decentralized trading required order books that were too expensive to run on-chain. With the AMM, decentralized trading required 300 lines of Solidity. The difference in adoption was the difference in complexity. The simpler system won.

Guillermo Angeris and Tarun Chitra's Improved Price Oracles: Constant Function Market Makers (2020) is the foundational academic treatment of AMM mathematics. The paper formalizes the constant function market maker (CFMM) as a general class, proves properties of the Uniswap and Curve formulas, and analyzes the time-weighted average price as an oracle. The paper is the bridge between the engineering of Uniswap and the theory of market design. The bridge is solid. The theory explains why the engineering works.


References:

  • Hayden Adams, Noah Zinsmeister, Dan Robinson, "Uniswap v3 Core," 2021.
  • Guillermo Angeris and Tarun Chitra, "Improved Price Oracles: Constant Function Market Makers," 2020.
  • Michael Egorov, "StableSwap — Efficient Mechanism for Stablecoin Liquidity," 2019 (Curve whitepaper).
  • Related posts: The Order Book, Algorithmic trading in crypto

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

AI Agents for Trading

Multi-agent AI systems now trade crypto with Sharpe ratios above 2.0. LLMs ingest on-chain data, social sentiment, and market signals. Specialized agents — research, risk, execution, governance — coordinate through structured interfaces. The architecture is microservices for money. The next generation of traders is not human.

dexai-agentsllmmulti-agenttrading

The preceding posts in this series described algorithmic trading strategies implemented as deterministic programs: arbitrage bots, market-making algorithms, MEV searchers. These programs follow rules. The rules are designed by humans. The programs execute the rules at machine speed. The programs are effective. They are also brittle — a rule designed for one market regime fails in another, and the program doesn't know to adapt.

AI agents represent a different paradigm. An AI agent is not a set of fixed rules. It is a system that perceives, reasons, and acts. It ingests data. It forms beliefs. It chooses actions. It learns from outcomes. The agent adapts. The adaptation is the edge. The edge is not in being faster than the next bot. It is in being smarter about what to do with the speed.

The architecture

Nex-T1, a multi-agent LLM-based trading system deployed in 2025, achieved a Sharpe ratio of 2.34 — 65% better than single-agent baselines — while cutting maximum drawdown nearly in half. The architecture is instructive:

Research agents. Scan on-chain data (transaction volumes, liquidity flows, whale movements), off-chain data (news, social sentiment, regulatory announcements), and market data (prices, volumes, order book depth). Synthesize into structured reports. The reports are the input to the strategy layer. The research agents are the eyes of the system.

Risk management agents. Evaluate position sizing, portfolio exposure, and drawdown risk. Propose position adjustments, stop-losses, and circuit breakers. The risk agents are independent of the strategy agents — a bug in a strategy agent should not prevent a risk agent from closing a dangerous position. The separation is architectural. The architecture is the safety mechanism.

Execution agents. Route trades across venues. Optimize for execution price, latency, and gas cost. Split large orders to minimize price impact. Monitor fill rates and adjust routing. The execution agents are the hands of the system. The hands must be fast. The speed is infrastructure.

Governance agents. Monitor the system's behavior for anomalies. Enforce compliance with risk limits, trading mandates, and regulatory requirements. The governance agent is the override — if the strategy agents propose a trade that violates the system's constraints, the governance agent blocks it. The override is the failsafe.

The agents communicate through structured interfaces — typed messages, defined schemas, explicit contracts. The architecture is microservices applied to trading. Each agent is a service. Each service has a narrow responsibility. The services compose. The composition is the system. The system trades. The trades are profitable.

The functional origin: expert systems and black boxes

The idea of automated trading is not new. Expert systems — rule-based programs that encode human expertise — were applied to trading in the 1980s. They were rigid. They required explicit rules for every situation. The rules were incomplete. The incompleteness was exploited by traders who understood the rules. The expert systems lost money.

Machine learning — statistical models trained on historical data — replaced expert systems in the 2000s. ML models could identify patterns that humans couldn't. They were also black boxes — their decisions were unexplainable. The lack of explainability was a barrier in regulated markets. The barrier limited adoption.

LLM-based agents combine the flexibility of ML with the explainability of expert systems. The agent can explain its reasoning — "I propose reducing ETH exposure because on-chain whale movement data suggests a large holder is preparing to sell." The explanation is auditable. The auditability satisfies compliance requirements. The flexibility handles novel situations. The combination is new. The combination is powerful.

The edge

The edge of AI agents over traditional algorithmic strategies is adaptability. A stat arb model calibrated on historical data degrades when the market regime changes. The model doesn't know the regime changed. It continues applying the old calibration. The calibration is wrong. The model loses money.

An LLM-based agent reads the news. It sees that a major protocol was hacked. It understands the implication: volatility will increase, correlations will break, stat arb strategies will fail. It reduces position sizes. It widens stop-losses. It shifts capital to safer assets. The adaptation is immediate. The adaptation is based on reasoning about the world, not on pattern matching historical data. The reasoning is the edge.

The edge erodes. More agents enter the market. The agents compete. The competition compresses margins. The agents must become smarter, faster, better-informed. The arms race continues. The arms race is the subject of this entire series. The next generation of combatants is not human. The combatants are agents. The agents are learning. The game is accelerating.

The reference

Nexis-AI, "Nex-T1: Multi-Agent Orchestration Framework for Autonomous DeFi Trading" (2025). The paper describing the 25-agent system. The architecture — specialized agents, structured interfaces, governance override — is the template for the next generation of trading systems. The template is open-source. The open-source availability accelerates adoption. The adoption accelerates competition. The competition accelerates the arms race. The cycle is the subject of this series. The series is a map. The map is not the territory. The territory moves.


References:

Trading infrastructure is distributed systems engineering. The order book, the AMM, the matching engine, the relay — each is a component in a latency-critical distributed system. The engineering constraints are the same as any real-time system: throughput, latency, reliability, correctness under concurrency. The domain is finance. The engineering is systems.

The Vickrey Auction

William Vickrey won a Nobel for proving that in a second-price sealed-bid auction, truthful bidding is a dominant strategy. You bid what the item is worth to you. You pay the second-highest bid. You cannot gain by lying. The Vickrey auction is the theoretical foundation for compute allocation, ad auctions, and every resource-scheduling mechanism where participants have private valuations.

game-theoryvickrey-auctionsecond-pricetruthful-biddingmechanism-design

William Vickrey published his auction paper in 1961. He won the Nobel Prize in 1996, three days before he died. The Vickrey auction — also called the second-price sealed-bid auction — works like this: each bidder submits a single bid, sealed, without knowing others' bids. The highest bidder wins. The winner pays the second-highest bid, not their own.

The magic: truthful bidding is a dominant strategy. You should bid exactly what the item is worth to you. Bidding higher doesn't help — if you win, you pay the second-highest bid, which is already determined. Bidding higher only risks winning at a price above your true value. Bidding lower doesn't help — you might lose an item you would have won at a price below your value. The optimal strategy is honesty. The mechanism makes honesty optimal.

Compare this to a first-price sealed-bid auction — highest bidder wins, pays their own bid. In a first-price auction, bidders shade their bids below their true value. How much to shade depends on what you think others will bid. The strategy is complex. The outcome is inefficient — the item may not go to the person who values it most. The Vickrey auction eliminates the shading. Honesty is optimal. Efficiency follows.

Interpretations from different branches

Auction theory (Vickrey, 1961 Nobel 1996). The revenue equivalence theorem: under symmetric independent private values, first-price, second-price, English, and Dutch auctions all yield the same expected revenue. The auction format doesn't matter for revenue. It matters for strategy. The Vickrey auction makes strategy simple. Simplicity is valuable.

Mechanism design (Clarke, Groves, 1970s). The Vickrey auction is a special case of the Vickrey-Clarke-Groves (VCG) mechanism. In the general VCG, each participant pays the externality they impose on others — the difference between the total value others would have received if the participant weren't there and the total value others actually receive. The VCG mechanism achieves efficient allocation with dominant-strategy incentive compatibility. It is the theoretical foundation for all truthful resource allocation mechanisms.

Online advertising (Google, Facebook). Google's AdWords auction is a generalized second-price (GSP) auction. Multiple ad slots. Multiple bidders. The GSP is not strategy-proof — bidders have incentive to shade. But it approximates the Vickrey outcome and is simpler to explain to advertisers. The trade-off between theoretical purity and practical simplicity is the mechanism designer's constant dilemma.

Spectrum auctions (FCC). The FCC auctions electromagnetic spectrum using a simultaneous multiple-round ascending auction — a complex mechanism designed to allocate hundreds of licenses across geographic regions. The design drew on Vickrey's insights: truthful bidding should be encouraged, bidders should be able to assemble packages of complementary licenses, and the auction should be transparent. The mechanism raised billions. The design was mechanism design applied to public resources.

Software engineering interpretations

Compute cluster allocation. Multiple teams share a GPU cluster. Each team has private information about the value of its jobs. The scheduler allocates GPUs. A first-come-first-served policy — the default — gives teams incentive to submit jobs early, hog resources, and misrepresent urgency. A Vickrey-like mechanism: teams submit bids in internal credits. The highest bidder wins, pays the second-highest bid. Truthful bidding is optimal. The credits are not real money. They are a mechanism for eliciting truthful valuations. The mechanism works without money because the credits are scarce. The scarcity makes them valuable.

Deployment slot scheduling. Multiple teams compete for a limited deployment window. Each team has private information about deployment urgency. A bidding mechanism: teams bid for slots using priority tokens allocated quarterly. The highest bidder gets the slot. Truthful bidding is optimal because tokens are scarce and bidding above true urgency risks wasting tokens. The mechanism replaces the political negotiation that currently determines deploy order. Politics is a bad allocation mechanism. Auctions are better.

Review queue prioritization. Pull requests compete for reviewer attention. Reviewers are scarce. A mechanism where authors bid for review priority using reputation credits — earned by reviewing others' PRs — aligns incentives. Authors who contribute reviews earn priority for their own PRs. The mechanism is a market for attention. Attention is the scarce resource. The market allocates it.

The Vickrey principle. The Vickrey auction teaches a general principle: make the cost of a decision equal to the externality it imposes on others. If winning a compute slot costs the value of the next-best job that could have used it, bidders have incentive to bid truthfully. The principle generalizes: price resources at their opportunity cost. The opportunity cost is the value of the best alternative foregone. Vickrey showed how to compute it. The computation is a mechanism. The mechanism is fair.


References:

  • William Vickrey, "Counterspeculation, Auctions, and Competitive Sealed Tenders," Journal of Finance, 1961.
  • Edward Clarke, "Multipart Pricing of Public Goods," Public Choice, 1971.
  • Theodore Groves, "Incentives in Teams," Econometrica, 1973.
  • Related posts: Design the Game, Game Theory Model: Mechanism Design

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

The Ultimatum Game

One player proposes how to split $100. The other accepts or rejects. If they reject, both get nothing. The Nash equilibrium: proposer offers $1, responder accepts. Real humans: proposers offer $30-$50, responders reject offers below $20. The gap between theory and behavior is a window into fairness, emotion, and what rationality actually means.

game-theoryultimatum-gamefairnessbehavioral-economicsrationality

The Ultimatum Game was introduced by Werner Güth in 1982. It is the simplest game that tests fairness. Two players. The proposer receives $100 and must offer some portion to the responder. The responder can accept — in which case both get the proposed split — or reject — in which case both get nothing. The game is played once. The players are anonymous.

The Nash equilibrium: the proposer offers the minimum possible amount ($1), and the responder accepts. Why? Because the responder, facing a choice between $1 and $0, rationally prefers $1. Knowing this, the proposer offers $1. The outcome is (99, 1). The prediction is clear. The prediction is wrong.

In thousands of replications across dozens of cultures, proposers offer 30-50% of the pie. Offers below 20% are rejected roughly half the time. The rejection rate increases as the offer decreases. The Nash equilibrium predicts 0% rejection at any positive offer. The prediction fails. The failure is systematic. The system is human.

Interpretations from different branches

Classical game theory. The Nash equilibrium is (proposer offers ε, responder accepts any positive offer). The prediction relies on the assumption that players care only about their own monetary payoff. The assumption is false. The falsity is informative.

Behavioral economics (Kahneman, Thaler). Responders reject low offers because they value fairness. The rejection is costly — they give up money — but the cost is worth it to punish unfair behavior. The proposer anticipates this and offers a fair split to avoid rejection. The fairness norm constrains the equilibrium. The constraint is not in the payoff matrix. It is in the players' heads.

Neuroeconomics (Sanfey et al., 2003). fMRI studies show that receiving an unfair offer activates the anterior insula — a brain region associated with disgust — and the dorsolateral prefrontal cortex — associated with cognitive control. The brain treats unfairness as physically aversive. The rejection is not a calculated choice. It is an emotional response. The emotion is disgust. The disgust overrides the rational calculation. The override is visible in the brain.

Cross-cultural studies (Henrich et al., 2001). The Ultimatum Game has been played in 15 small-scale societies. The results vary enormously. In some societies, proposers offer as little as 15% and responders accept. In others, proposers offer more than 50% — hyper-fair — and responders reject both low and high offers. The variation tracks cultural norms about sharing, gift-giving, and market integration. The game reveals culture. The culture varies. The variation is systematic.

Evolutionary psychology. Fairness norms evolved in small groups where reputation mattered. The one-shot anonymous Ultimatum Game is evolutionarily novel. The brain applies reputation-based reasoning to an anonymous situation. The misapplication produces rejection of unfair offers. The brain is not wrong. The environment is novel. The adaptation is for a different world.

Software engineering interpretations

Salary negotiation. The offer is the salary. The rejection is walking away. If the offer is too low, the candidate rejects — even though a low salary is better than no salary. The rejection is the Ultimatum Game. The candidate is the responder. The company is the proposer. Companies that lowball lose good candidates not because the candidates can't use the money but because the offer signals undervaluation. The signal is the information. The information is worth more than the salary difference.

Resource allocation between teams. The infrastructure budget must be split. The platform team proposes an allocation. The service teams can accept or escalate. Escalation costs time and political capital — both parties lose. The Nash equilibrium: platform proposes a minimal allocation, service teams accept. The observed behavior: platform proposes a fair split to avoid escalation. The fairness norm constrains the outcome.

Review assignment. A tech lead assigns code reviews. If the assignment is visibly unfair — one person gets all the difficult reviews — the overloaded reviewer may refuse, forcing the lead to rebalance. The refusal costs the reviewer social capital. The cost is worth it to punish the unfairness. The lead, anticipating refusal, assigns fairly. The fairness is strategic.

The takeaway. The Ultimatum Game teaches that humans are not purely self-interested payoff maximizers. They value fairness. They will pay to punish unfairness. The willingness to pay is a constraint on any system that allocates resources among humans. Algorithms that produce technically optimal but visibly unfair allocations will be rejected. The rejection is rational — if your model of rationality includes fairness in the utility function. The model should.


References:

  • Werner Güth, Rolf Schmittberger, Bernd Schwarze, "An Experimental Analysis of Ultimatum Bargaining," Journal of Economic Behavior & Organization, 1982.
  • Daniel Kahneman, Jack Knetsch, Richard Thaler, "Fairness and the Assumptions of Economics," Journal of Business, 1986.
  • Joseph Henrich et al., "In Search of Homo Economicus," American Economic Review, 2001.
  • Related posts: Scarcity and Games

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

The Tragedy of the Commons

Garrett Hardin described it in 1968: a shared resource, individually rational use, collective ruin. The Tragedy of the Commons is a multi-player Prisoner's Dilemma with a shared resource. It explains overloaded CI pipelines, degraded staging environments, and every shared service that nobody maintains. Elinor Ostrom won a Nobel proving it's not inevitable.

game-theorytragedy-of-commonshardinostromcommon-pool-resources

Garrett Hardin published "The Tragedy of the Commons" in Science in 1968. The argument: a pasture open to all herders. Each herder gains the full benefit of adding an animal — more milk, more meat. Each herder bears only a fraction of the cost — the pasture is shared, the overgrazing is distributed. The individually rational choice: add another animal. The collective outcome: the pasture is destroyed. Everyone loses.

"Therein is the tragedy. Each man is locked into a system that compels him to increase his herd without limit — in a world that is limited. Ruin is the destination toward which all men rush, each pursuing his own best interest in a society that believes in the freedom of the commons."

The tragedy is a multi-player Prisoner's Dilemma with a renewable resource. The resource has a carrying capacity. Below capacity, grazing is sustainable. Above capacity, the resource degrades. Each player's marginal benefit of adding an animal is private. The marginal cost is shared. The private benefit exceeds the private cost until the resource collapses. The collapse is predictable. The predictability doesn't prevent it.

Interpretations from different branches

Classical game theory. The Tragedy is a social dilemma. The Nash equilibrium is overuse. The equilibrium is Pareto-suboptimal — everyone would be better off with restraint. The dilemma is structurally identical to the Prisoner's Dilemma scaled to N players. The larger the N, the smaller each player's share of the cost of their own overuse. The tragedy intensifies with group size.

Institutional economics (Ostrom, 1990 Nobel 2009). Elinor Ostrom challenged Hardin's conclusion that commons are inevitably tragic. She studied real communities that managed common-pool resources successfully for centuries — Swiss grazing pastures, Japanese forests, Spanish irrigation systems. She identified eight design principles for sustainable commons: clearly defined boundaries, proportional equivalence between benefits and costs, collective choice arrangements, monitoring, graduated sanctions, conflict resolution mechanisms, minimal recognition of rights, and nested enterprises. The principles are mechanism design for commons. The mechanisms work. The tragedy is not inevitable.

Environmental economics. Climate change is the ultimate tragedy of the commons. The atmosphere is a shared resource. Carbon emissions benefit the emitter. The cost is shared globally. The individually rational choice: emit. The collectively optimal choice: reduce. The coordination problem is planet-scale. The mechanisms — carbon taxes, cap-and-trade, international agreements — are Ostrom principles scaled to nations. The scaling is hard because enforcement across sovereign states is weak. The weakness is the tragedy.

Digital commons. Open-source software is a commons. The code is a shared resource. Contributors maintain it. Users consume it. The individually rational choice: use without contributing. The collectively optimal choice: everyone contributes. The tragedy: maintainer burnout, abandoned projects, the internet running on a single developer's unpaid labor. The Ostrom principles apply: clear governance, graduated sanctions (from bug reports to commit access), collective choice (RFC processes). The mechanisms exist. They are fragile.

Software engineering interpretations

The shared CI pipeline. Every team adds tests to the shared CI pipeline. Each test benefits the team that added it. The cost — longer build times — is shared by all teams. The individually rational choice: add tests. The collectively optimal choice: add only high-value tests. The tragedy: the pipeline takes 45 minutes. Everyone suffers. The mechanism: a test budget per team, periodic culling, a requirement that new tests justify their existence with historical failure catch rate. Ostrom's proportional equivalence: the cost a team imposes on the commons must be proportional to the benefit they derive.

The staging environment. Staging is a shared resource. Every team wants to use it for integration testing, demos, and load testing. Overuse degrades it. The individually rational choice: use staging whenever you need it. The tragedy: staging is unreliable, nobody trusts it, everyone builds their own staging-like environment. The mechanism: a booking calendar, dedicated demo environments, tiered access with SLAs. Ostrom's clearly defined boundaries: who can use staging, for what, when.

The monolith as commons. The monolith's codebase is a shared resource. Every team adds code. The cost of complexity is shared. The individually rational choice: add the feature in the simplest way for your team. The tragedy: the monolith becomes unmaintainable. The mechanism: module ownership, interface contracts, automated complexity budgets per module. Ostrom's monitoring: visibility into who added what complexity, and what it cost.

The shared database. Multiple services read and write the same database. Each service optimizes its queries for its own use case. The cost — contention, locking, schema complexity — is shared. The tragedy: the database is the bottleneck, nobody can change their schema without breaking others. The mechanism: one service owns the database, all access goes through its API. Ostrom's clearly defined boundaries: the database is not a commons. It is property. The property has an owner.

Resolving the commons

Hardin was wrong that tragedy is inevitable. Ostrom was right that it can be governed. The governance requires: clear boundaries, proportional costs, collective decision-making, monitoring, graduated sanctions, conflict resolution, recognized rights, and layered organization. These are not optional. They are the design principles. They apply to pastures, fisheries, CI pipelines, staging environments, monoliths, and shared databases. The commons is everywhere. The principles are the same. The implementation is local. The failure to implement is the tragedy.


References:

  • Garrett Hardin, "The Tragedy of the Commons," Science, 1968.
  • Elinor Ostrom, Governing the Commons, Cambridge University Press, 1990.
  • Elinor Ostrom, "Beyond Markets and States: Polycentric Governance of Complex Economic Systems," Nobel Prize Lecture, 2009.
  • Related posts: Scarcity and Software Games, Field Guide to Scarcity Games

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

The Stag Hunt

Jean-Jacques Rousseau described it in 1755: two hunters must decide whether to hunt a stag together or take a rabbit alone. The stag is worth more. The rabbit is certain. If either defects, the stag escapes. The Stag Hunt models coordination under trust — and explains network effects, standard adoption, and why good architectures fail to spread.

game-theorystag-huntcoordinationtrustnetwork-effects

Jean-Jacques Rousseau described the Stag Hunt in his Discourse on Inequality (1755):

"If it was a matter of hunting a deer, everyone well realized that he must remain faithfully at his post; but if a hare happened to pass within reach of one of them, we cannot doubt that he would have gone off in pursuit without scruple."

Two hunters. They can hunt a stag together or hunt rabbits alone. The stag requires both to cooperate. If both hunt the stag, they share a large reward. If one defects to hunt rabbits, the defector gets a rabbit — less than half a stag but guaranteed — and the cooperator gets nothing. If both hunt rabbits, each gets a rabbit.

The payoff matrix:

Hunt Stag Hunt Rabbit
Hunt Stag (10, 10) (0, 5)
Hunt Rabbit (5, 0) (5, 5)

There are two Nash equilibria: both hunt stag and both hunt rabbit. Both are stable. Neither player can unilaterally improve by switching. The stag equilibrium is Pareto-superior. The rabbit equilibrium is risk-dominant — if you're uncertain what the other will do, rabbit is safer. The game is about trust and coordination, not conflict. The challenge is moving from the rabbit equilibrium to the stag equilibrium.

Interpretations from different branches

Classical game theory. The stag hunt has two pure-strategy Nash equilibria. Equilibrium selection — which one the players end up in — is not determined by the payoff structure alone. It depends on beliefs, expectations, and focal points. Schelling's concept of the focal point was developed partly to explain equilibrium selection in coordination games.

Risk dominance (Harsanyi and Selten). The rabbit equilibrium risk-dominates the stag equilibrium if the expected payoff of hunting rabbit, given uncertainty about the other player, exceeds the expected payoff of hunting stag. Formally: (5+5)/2 > (10+0)/2 → 5 > 5, so neither risk-dominates. In the classic stag hunt, the equilibria are payoff-equivalent under uniform uncertainty. But if the stag is worth more — say 20 — then (20+0)/2 = 10 > 5, and stag risk-dominates. The size of the prize determines whether coordination on the ambitious outcome is rational under uncertainty.

Evolutionary game theory. In a population playing stag hunts, which equilibrium is selected depends on the initial proportion of stag hunters. If enough players hunt stag, the stag equilibrium is reached. There is a critical threshold. Below the threshold, the population converges to rabbits. The threshold is the tipping point. The dynamics are the replicator equation.

Network economics. The stag hunt models adoption of technologies with network effects. A communication standard is a stag. Everyone benefits if everyone uses it. But if you're the first adopter, you bear the switching cost with no guarantee others will follow. The critical mass problem is the stag hunt in economic form.

Software engineering interpretations

Logging standardization. SRE proposes a standard logging library. If all 12 services adopt it, logs become queryable across services. If only some adopt, the adopters get no benefit — their logs are standardized but they can't query across services. If nobody adopts, nothing changes. The stag is the universal query. The rabbit is keeping your own format. The critical mass problem: who goes first?

API gateway adoption. The platform team builds an API gateway. Each service team must decide whether to route through it. If all use it, the organization gets unified auth, rate limiting, and monitoring. If some bypass it, the bypassers move faster (no gateway latency, no configuration overhead) and the adopters get partial benefit. The stag is the unified gateway. The rabbit is direct access. The critical mass is the number of services needed for the gateway's benefits to be self-sustaining.

Design system adoption. The design team ships a component library. Each frontend team chooses whether to use it. If all use it, the product gets visual consistency and shared maintenance. If some don't, the non-adopters ship faster and the adopters maintain the library for everyone. The stag is consistency. The rabbit is velocity. The library needs critical mass to survive.

Resolving the stag hunt

The stag hunt is resolved by mechanisms that raise confidence that others will cooperate. Visible early adopters signal commitment. Public commitments — "we will adopt the standard by Q3" — reduce uncertainty. Sequential adoption with increasing benefits — each new adopter increases the value for remaining players — creates the tipping dynamic. Schelling's focal points — obvious choices that everyone expects everyone else to make — select the equilibrium. "We'll all use the most popular format on npm." The format is a focal point. The popularity is the signal.


References:

  • Jean-Jacques Rousseau, Discourse on the Origin and Basis of Inequality Among Men, 1755.
  • Brian Skyrms, The Stag Hunt and the Evolution of Social Structure, Cambridge University Press, 2004.
  • Thomas Schelling, The Strategy of Conflict, Harvard University Press, 1960.
  • Related posts: Scarcity and Games, Scarcity and Software Games

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

Signaling Games

One player has private information. They take an action that may reveal it. The action is a signal. A signal works if it's costly enough that only the 'good' type would send it. Michael Spence won a Nobel for proving that education can be a pure signal — it doesn't teach skills, it proves you had the skills to get in. In software: the RFC that's too detailed, the rewrite proposal that comes with a prototype.

game-theorysignalingspenceprivate-informationseparating-equilibrium

A signaling game has two players. The sender has private information about their type. The sender takes an action — sends a signal. The receiver observes the signal and responds. The sender's payoff depends on the receiver's response and the sender's type. The receiver's payoff depends on the sender's type and the receiver's response.

The classic model is Michael Spence's job market signaling (1973). A worker knows their own productivity (high or low). The employer doesn't. The worker can acquire education — a costly signal. If education is more costly for low-productivity workers than for high-productivity workers, there exists a separating equilibrium: high types acquire education, low types don't, employers pay high wages to educated workers and low wages to uneducated workers. The education doesn't need to increase productivity. It only needs to be differentially costly. The differential cost is what makes the signal informative.

Interpretations from different branches

Economics (Spence, 1973 Nobel 2001). The separating equilibrium exists when the single-crossing property holds — the marginal cost of signaling differs across types. Education signals productivity not because education teaches skills but because it is harder for low-productivity people to acquire. The hardship is the signal. The credential is the evidence of hardship.

Biology (Zahavi's handicap principle, 1975). The peacock's tail is a signal of genetic fitness. The tail is costly — it requires energy, attracts predators, impedes movement. Only a genetically fit peacock can afford such a handicap. The cost is the signal. The tail doesn't help the peacock survive. It proves the peacock can survive despite the tail. The handicap is honest because it's too expensive to fake.

Political science (Fearon, 1994). Audience costs in international relations are signaling games. A leader who makes a public threat — "we will retaliate" — pays an audience cost if they back down. The cost is domestic political damage. Only a leader who is genuinely committed can afford to make the threat. The threat is credible because backing down is costly. The cost is the signal.

Computer science (mechanism design). Screening is the reverse of signaling. In screening, the uninformed party moves first, offering a menu of contracts. Each type self-selects into the contract designed for it. The menu separates types by making each contract optimal for the intended type. Screening is mechanism design. Signaling is sender-driven. Screening is receiver-driven.

Software engineering interpretations

The rewrite proposal. A team proposes rewriting a legacy service. The proposal is cheap talk — anyone can propose. The architecture review must decide whether the team genuinely believes the rewrite is necessary or is bored and wants greenfield work. A prototype requirement is a screening mechanism. A sincere team will build the prototype. An insincere team won't. The prototype is costly. The cost separates types.

The detailed RFC. Writing a thorough RFC — with trade-off analysis, migration plans, and risk assessment — is costly. It takes days. The cost signals seriousness. A team that writes a one-paragraph proposal is signaling low investment. The architecture review discounts the proposal accordingly. The RFC's length is not about information transfer. It is about signaling commitment.

Open-source contributions as job market signaling. A developer contributes to a well-known open-source project. The contribution is publicly visible. It signals skill to potential employers. The contribution is costly — it takes time outside work. The cost signals passion and competence. The signal works because it's harder to fake than a resume bullet point. The code doesn't lie. The commit history is the credential.

The production incident response. How an engineer handles a production incident signals competence to the entire organization. The signal is costly — incidents are high-stress, high-visibility, and occur at inconvenient times. The engineer who stays calm, diagnoses systematically, and communicates clearly is signaling a type that cannot be faked under pressure. The pressure is the cost. The calm is the signal.

Separating vs. pooling equilibria

A signaling game has two kinds of equilibria. In a separating equilibrium, different types send different signals. The receiver can infer the type from the signal. In a pooling equilibrium, all types send the same signal. The receiver learns nothing. Which equilibrium emerges depends on the cost structure and the prior distribution of types. If the cost of the signal is too low for all types, everyone sends it — credential inflation. If too high, nobody does. The sweet spot is where the cost is differentially burdensome.

In software: if writing a prototype is too easy, everyone writes one. The prototype stops separating types. If it's too hard, nobody writes one. The review process loses the signal. The optimal screening cost is calibrated to the distribution of sincere and insincere proposers. The calibration is mechanism design.


References:

  • Michael Spence, "Job Market Signaling," Quarterly Journal of Economics, 1973.
  • Amotz Zahavi, "Mate Selection — A Selection for a Handicap," Journal of Theoretical Biology, 1975.
  • James D. Fearon, "Domestic Political Audiences and the Escalation of International Disputes," American Political Science Review, 1994.
  • Related posts: Scarcity and Games, Scarcity and Software Games

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

Repeated Games and the Folk Theorem

Robert Aumann proved that in infinitely repeated games, cooperation can be sustained as an equilibrium. The shadow of the future disciplines the present. The Folk Theorem says: any feasible, individually rational payoff can be an equilibrium if the game is repeated long enough. This is why teams that stay together build trust. The trust is mathematics.

game-theoryrepeated-gamesfolk-theoremaumanncooperation

A one-shot Prisoner's Dilemma produces defection. A repeated Prisoner's Dilemma can produce cooperation. The difference is the future. In a one-shot game, there is no tomorrow in which defection can be punished. In a repeated game, today's defection costs you tomorrow's cooperation. If tomorrow matters enough — if the discount rate is low enough — cooperation becomes the rational strategy.

The Folk Theorem, so called because it was known informally before it was formally proved, states: in an infinitely repeated game with sufficiently patient players, any feasible and individually rational payoff vector can be sustained as a Nash equilibrium. "Feasible" means the payoffs can be achieved by some combination of strategies. "Individually rational" means each player gets at least their minimax payoff — the worst the other players can impose on them. The theorem says: if the game goes on long enough, almost any outcome is possible as an equilibrium. The future is a mechanism for sustaining norms.

The mechanism is the threat of punishment. If I cooperate today, you cooperate tomorrow. If I defect today, you defect tomorrow — and the day after, and the day after. The punishment must be credible. It must be in your interest to carry it out once defection has occurred. The credibility of the punishment is the constraint on what equilibria can be sustained. The Grim Trigger — cooperate until the first defection, then defect forever — is the harshest credible punishment. Tit-for-Tat — start with cooperation, then mirror the other's last move — is gentler and more robust.

Interpretations from different branches

Game theory (Aumann, 1959 Nobel 2005). Aumann's contribution was the formal analysis of repeated games with incomplete information. If players have private information, the repetition allows them to learn about each other. The learning changes the equilibrium. The Folk Theorem extends to games with imperfect monitoring — players observe noisy signals of each other's actions. The extension is technical. The implication is practical: even when you can't perfectly observe what others did, repetition enables cooperation.

Political science (Axelrod, 1984). Robert Axelrod's tournaments showed that Tit-for-Tat is a robust strategy. It is nice (never defects first), retaliatory (punishes defection immediately), forgiving (returns to cooperation if the other does), and clear (easy to recognize). These four properties make it effective in repeated interactions. The clarity is crucial — if the other player can't figure out your strategy, they can't adapt to it. Clarity is strategic.

International relations (Keohane, 1984). International cooperation is a repeated game among states. Trade agreements, arms control treaties, environmental protocols — these are equilibria sustained by the shadow of the future. The mechanism: if you violate the treaty today, we withdraw cooperation tomorrow. The mechanism works when the future is valued. It fails when states discount the future heavily — when leaders face elections, when regimes are unstable, when the long-term benefits of cooperation accrue to successors rather than incumbents. The discount rate is political. The politics determine whether cooperation is sustainable.

Organizational behavior. Company culture is a repeated-game equilibrium. Norms of collaboration, knowledge-sharing, and mutual support are sustained by the expectation of continued interaction. When turnover is high, the shadow of the future shortens. Cooperation declines. The decline is attributed to "bad culture." The culture is a symptom. The cause is the shortened horizon. Fix the horizon. The culture follows.

Software engineering interpretations

Team continuity. A stable team is a repeated game. Members expect to work together indefinitely. Cooperation is an equilibrium — helping a colleague today is repaid tomorrow. A team with high turnover is a sequence of short-horizon games. Cooperation is fragile — the new person hasn't built the history, the departing person won't face the consequences of defection. The Folk Theorem predicts that stable teams will have more cooperation. The prediction is correct. The mechanism is the horizon.

Inter-service API stability. Two services with a long history of mutual dependence are in a repeated game. Breaking the API today costs you in future coordination. The equilibrium is stability. A service consumed by many anonymous clients is in a one-shot game with each. Breaking the API harms each client individually but none enough to punish. The equilibrium is instability — the provider changes the API when convenient. The difference is the horizon. The horizon is structural.

Code review reciprocity. Engineers who review each other's code are in a repeated game. Reviewing thoroughly today earns thorough reviews tomorrow. Skimming today earns skimmed reviews tomorrow. The equilibrium is a norm of thoroughness — if the team is stable and the horizon is long. The norm emerges without being mandated. The mandate is unnecessary. The horizon is sufficient.

Cross-team collaboration. Two teams that expect to work together for years develop informal cooperation — shared understanding, mutual accommodation, the benefit of the doubt. Two teams thrown together for a single project have no shadow of the future. Cooperation must be formalized — detailed specs, explicit contracts, escalation paths. The formality is the substitute for the missing horizon. The formality is costly. The horizon was free.

The design implication

If you want cooperation, lengthen the horizon. Stable teams. Long-term ownership. Continuity of relationships. If you can't lengthen the horizon, simulate it: automated contract testing makes defection immediately visible, creating a repeated-game payoff structure. SLAs with penalties bring the future cost of defection into the present. The mechanisms are substitutes for the missing horizon. The best mechanism is the horizon itself. The horizon is free.


References:

  • Robert Aumann, "Acceptance Speech," Nobel Prize, 2005.
  • Robert Axelrod, The Evolution of Cooperation, Basic Books, 1984.
  • Drew Fudenberg and Eric Maskin, "The Folk Theorem in Repeated Games with Discounting or with Incomplete Information," Econometrica, 1986.
  • Related posts: Cooperation is logical, Scarcity and Games

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

The Prisoner's Dilemma

Two prisoners. Two choices. Cooperate or defect. The individually rational choice produces a collectively worse outcome. The Prisoner's Dilemma is the foundational model of game theory because it captures the essential tension between self-interest and mutual benefit. It explains why teams don't cooperate, why standards don't get adopted, and why the microservices migration never finishes.

game-theoryprisoners-dilemmacooperationdefectionsoftware-architecture

The Prisoner's Dilemma is the simplest game that captures the deepest tension in strategic interaction. Two players. Two choices: cooperate or defect. If both cooperate, both get a moderate reward. If both defect, both get a moderate punishment. If one cooperates and the other defects, the defector gets the maximum reward and the cooperator gets the maximum punishment. The payoff structure, in order of preference: (defect, cooperate) > (cooperate, cooperate) > (defect, defect) > (cooperate, defect).

The dilemma: defect is the dominant strategy. Regardless of what the other player does, you are better off defecting. If they cooperate, you get the temptation payoff — the maximum. If they defect, you get the punishment payoff — bad, but better than being the sucker who cooperated while the other defected. So you defect. They reason identically. They defect. You both get the punishment payoff. You are both worse off than if you had cooperated. The individually rational choice produced a collectively worse outcome. This is the dilemma.

The classic framing

The story was formalized by Merrill Flood, Melvin Dresher, and Albert Tucker at RAND in the 1950s. Two members of a criminal gang are arrested. The prosecutor has enough evidence to convict both on a minor charge (1 year each) but needs a confession to convict on the major charge. The prosecutor separates them and offers each the same deal: testify against the other (defect), and you go free while the other gets 10 years. If both testify, both get 5 years. If both remain silent (cooperate), both get 1 year on the minor charge.

The payoff matrix, in years of freedom lost:

Cooperate (silent) Defect (testify)
Cooperate (-1, -1) (-10, 0)
Defect (0, -10) (-5, -5)

Look at the Cooperate row. If the other cooperates, you get -1 by cooperating and 0 by defecting. Defect is better. If the other defects, you get -10 by cooperating and -5 by defecting. Defect is better. Defect is dominant. The logic is inescapable. The outcome is suboptimal.

Interpretations from different branches

Classical game theory. The unique Nash equilibrium is mutual defection. The equilibrium is Pareto-suboptimal — both could be better off. The dilemma is that rationality does not lead to optimality. This is a theorem. It is not a suggestion. It is a proof about the structure of certain payoff matrices.

Repeated game theory (Aumann). If the game is played repeatedly with no known end, cooperation can be sustained as an equilibrium. The shadow of the future disciplines present behavior. The Folk Theorem proves that any individually rational, feasible payoff can be sustained in an infinitely repeated game. Tit-for-Tat — start by cooperating, then mirror the other player's last move — is a simple strategy that sustains cooperation in iterated play.

Evolutionary game theory (Maynard Smith, Axelrod). In populations of strategies playing repeated Prisoner's Dilemmas, Tit-for-Tat emerges as robust. It is nice (starts cooperatively), retaliatory (punishes defection), forgiving (returns to cooperation if the other does), and clear (easy for others to recognize and respond to). In Axelrod's famous tournaments, Tit-for-Tat won against far more sophisticated strategies. The simplicity was the advantage. The clarity was the mechanism.

Behavioral economics. Real humans cooperate in one-shot Prisoner's Dilemmas at rates far above the Nash prediction. In laboratory experiments, cooperation rates average 40-60%. The prediction is 0%. The gap between prediction and behavior is the subject of behavioral economics. Explanations include altruism, confusion, social norms, and the "illusion of repeated play" — humans evolved in small groups where interactions were always repeated. The one-shot game is evolutionarily novel. The brain treats it as repeated anyway.

Software engineering interpretations

The microservices migration. Each team benefits if all teams migrate to microservices. The migration costs each team coordination effort. The individually rational choice: wait for others to migrate first, then migrate when the path is clear. The collectively optimal choice: all migrate simultaneously with coordination. The equilibrium: nobody migrates, or some migrate and produce a distributed monolith. The dilemma is the migration.

API standardization. Each team benefits from a shared API standard. Each team prefers to keep its own format — the switching cost is immediate, the benefit of the standard is shared and delayed. The individually rational choice: keep your format. The collectively optimal choice: all adopt the standard. The equilibrium: fragmented formats, adapters everywhere, SRE team despairs.

Code review thoroughness. Each reviewer benefits from the system having fewer bugs. Each reviewer would prefer that other reviewers catch the bugs. A thorough review costs time. The benefit of catching a bug is shared. The individually rational choice: skim. The collectively optimal choice: thorough review for all. The equilibrium: bugs slip through.

Open source contribution. Everyone benefits from maintained open-source projects. Contributing costs time. The individually rational choice: use without contributing. The equilibrium: maintainer burnout. The free-rider problem is the Prisoner's Dilemma at scale.

Resolving the dilemma

The Prisoner's Dilemma cannot be "solved" without changing the game. The changes that work: repeat the interaction (Folk Theorem), make defection visible and costly (mechanism design), reduce the payoff for defection relative to cooperation (incentive alignment), or enable communication and binding agreements (cooperative game theory).

In software: make defection visible. Automated contract testing makes breaking an API immediately visible. Visibility changes the payoff — the short-term gain of defection is offset by the immediate cost of fixing the test. Repeated interaction does the rest. Teams that work together for years develop cooperation equilibria without formal mechanisms. The trust is not personality. It is mathematics.


References:

  • Robert Axelrod, The Evolution of Cooperation, Basic Books, 1984.
  • Robert Aumann, "Acceptance Speech," Nobel Prize in Economics, 2005.
  • Anatol Rapoport and Albert Chammah, Prisoner's Dilemma, University of Michigan Press, 1965.
  • Related posts: Scarcity and Games, Cooperation is logical

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

The Principal-Agent Problem

A principal hires an agent to do work. The agent has private information and their own objectives. The principal cannot perfectly monitor the agent's effort. The problem: how to design a contract that aligns the agent's incentives with the principal's goals. This is the model for every employment relationship, every outsourcing decision, and every API contract between teams.

game-theoryprincipal-agentmoral-hazardcontract-theoryincentives

The principal-agent problem is the foundational model of contract theory. A principal wants a task performed. An agent can perform it. The agent's effort is unobservable — the principal sees the outcome but not the effort. The outcome depends on effort and on random factors beyond the agent's control. The principal must design a contract — a payment scheme contingent on observable outcomes — that incentivizes the agent to exert the desired level of effort.

The tension: the principal wants high effort at low cost. The agent wants high payment for low effort. If the principal pays a fixed wage, the agent exerts minimum effort — there's no incentive to do more. If the principal pays entirely based on outcome, the agent bears all the risk from random factors. The optimal contract balances insurance (protecting the agent from randomness) with incentives (rewarding effort). The balance is the contract. The contract is the mechanism.

The problem has two variants. Moral hazard: the agent takes hidden actions after the contract is signed. Adverse selection: the agent has hidden information before the contract is signed — the agent knows their own type, and the principal doesn't. Both are information asymmetries. Both require mechanism design to resolve.

Interpretations from different branches

Contract theory (Hart, Holmström, 2016 Nobel). Oliver Hart and Bengt Holmström shared the 2016 Nobel for contract theory. Holmström's informativeness principle: any performance measure that provides information about effort should be included in the contract. If the measure is informative, including it reduces the agent's risk for a given level of incentives. Hart's incomplete contracts: real contracts cannot specify every contingency. When contracts are incomplete, the allocation of residual control rights — who decides what happens in unforeseen circumstances — determines outcomes. Ownership matters because ownership confers residual control.

Corporate governance (Jensen and Meckling, 1976). The separation of ownership and control in corporations is a principal-agent problem. Shareholders are principals. Managers are agents. Managers may pursue their own interests — empire-building, risk-aversion, short-term stock price — rather than shareholder value. The mechanisms: stock options (aligning incentives), boards of directors (monitoring), hostile takeovers (discipline). The mechanisms are imperfect. The imperfection is the cost of the agency relationship.

Regulation (Laffont and Tirole, 1993 Nobel 2014). Regulating a monopoly is a principal-agent problem. The regulator (principal) wants the monopoly (agent) to operate efficiently. The monopoly has private information about its costs. The regulator designs a pricing scheme that incentivizes cost reduction while preventing excessive pricing. The scheme is a contract. The contract is mechanism design applied to public utility regulation.

Political science. Voters are principals. Politicians are agents. The agency problem is accountability. Elections are the incentive mechanism — politicians who perform poorly are voted out. The mechanism is imperfect because voters have incomplete information, politicians control the flow of information, and election cycles are coarse. The imperfections are the subject of political economy.

Software engineering interpretations

Team and manager. The manager (principal) wants the team (agent) to produce high-quality work. The manager cannot perfectly observe effort — code quality is partly effort, partly skill, partly the difficulty of the task. The contract: salary, performance review, promotion. The mechanisms: code review (monitoring), OKRs (outcome-based incentives), peer feedback (multi-source monitoring). Each mechanism reduces the information asymmetry.

Outsourcing vendor management. The company (principal) hires a vendor (agent) to build a system. The vendor has private information about their true costs, their true timeline, and the quality of their engineers. The contract: fixed-price or time-and-materials. Fixed-price transfers risk to the vendor but creates incentive to cut corners. Time-and-materials transfers risk to the company but creates incentive to inflate hours. The optimal contract balances risk-sharing with incentive alignment. The balance is the principal-agent problem in procurement form.

Platform team and service teams. The platform team (agent) provides infrastructure to service teams (principals). The service teams cannot observe the platform team's effort. The platform team may optimize for its own interests — interesting technical work, clean architecture — rather than service team needs. The contract: SLAs with penalties, internal billing (chargebacks), user satisfaction surveys. The mechanisms align the platform team's incentives with service team outcomes.

Open-source maintainer and corporate user. The maintainer (agent) produces a library. The corporation (principal) depends on it. The maintainer's effort is unobservable. The corporation cannot compel the maintainer to fix bugs or accept patches. The contract is social, not legal — reputation, sponsorship, contribution guidelines. The principal-agent problem in open source is acute because the mechanisms are weak. The weakness is why maintainers burn out.

The alignment problem

The principal-agent problem is the alignment problem in economic form. Align the agent's incentives with the principal's goals. The alignment is never perfect because information is never perfect. The residual misalignment is the agency cost. The cost is irreducible. The mechanism designer's job is to minimize it. Minimize it by making outcomes observable, linking rewards to observables, and accepting that some misalignment will remain. The acceptance is realism. The minimization is engineering.


References:

  • Bengt Holmström, "Moral Hazard and Observability," Bell Journal of Economics, 1979.
  • Oliver Hart and Bengt Holmström, "The Theory of Contracts," 1987.
  • Michael Jensen and William Meckling, "Theory of the Firm: Managerial Behavior, Agency Costs and Ownership Structure," Journal of Financial Economics, 1976.
  • Related posts: Design the Game, Mechanism Design

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

Mechanism Design

Mechanism design is reverse game theory. Start with the outcome you want. Design the rules that produce it. It explains SLAs, automated contract testing, deployment gates, code review requirements, and every organizational rule that works. It was worth a Nobel Prize in 2007. It is the highest-leverage activity in software engineering.

game-theorymechanism-designincentiveshurwiczmaskinmyerson

Mechanism design is game theory in reverse. In standard game theory, you are given the rules and you solve for the equilibrium. In mechanism design, you are given the desired outcome and you design the rules that produce it as an equilibrium. You don't ask "what will happen given these incentives?" You ask "what incentives will produce what we want to happen?"

The field was founded by Leonid Hurwicz, Eric Maskin, and Roger Myerson. They shared the 2007 Nobel Prize. Hurwicz asked the founding question in 1960: how do you design an allocation mechanism that works even when participants have private information and act strategically? The answer: you design the rules so that truthful revelation of private information is the best strategy for each participant, and the outcome given truthful revelation is the one you want.

The mechanism has four components: a set of participants, each with private information (their type). A set of possible outcomes. A rule that maps reported types to outcomes. And a solution concept — usually dominant-strategy incentive compatibility or Bayesian incentive compatibility. The mechanism is incentive-compatible if truthful reporting is an equilibrium. The mechanism implements the desired outcome if the equilibrium produces it.

Interpretations from different branches

Economics (Hurwicz, 1960). The fundamental question: can a central planner achieve an efficient allocation without knowing individuals' private valuations? The answer: yes, under certain conditions. The Vickrey-Clarke-Groves (VCG) mechanism achieves efficient allocation in quasi-linear environments. Participants report their valuations. The mechanism allocates goods to maximize total reported value and charges each participant the externality they impose on others. Truthful reporting is a dominant strategy. The VCG mechanism is the theoretical foundation of spectrum auctions, online advertising auctions, and compute resource allocation.

Implementation theory (Maskin, 1977 Nobel 2007). Maskin answered: which social choice rules can be implemented in Nash equilibrium? The answer involves Monotonicity — if an outcome is selected under one preference profile, and the outcome moves up in everyone's ranking under a new profile, it must still be selected. Monotonicity is necessary and, with no veto power, sufficient for Nash implementation. The mathematics is abstract. The implication is practical: not every desirable outcome can be implemented. The constraints are mathematical, not political.

Auction theory (Myerson, 1981 Nobel 2007). Myerson characterized optimal auctions. The revenue-equivalence theorem: any auction that allocates to the highest bidder and gives zero surplus to the lowest type yields the same expected revenue. The optimal auction sets a reserve price and allocates to the highest bidder above the reserve. The reserve price is the mechanism designer's tool for extracting surplus. Myerson applied mechanism design to auctions and showed that auction design is mechanism design.

Market design (Roth, 2012 Nobel). Alvin Roth applied mechanism design to markets that didn't exist: matching medical residents to hospitals, matching students to schools, matching kidney donors to recipients. The mechanisms are algorithms that take reported preferences and produce stable matches. Stability means no pair would prefer each other over their current match. The deferred acceptance algorithm (Gale-Shapley) produces stable matches. Roth made it work in practice. Market design is mechanism design implemented.

Software engineering interpretations

Automated contract testing. The desired outcome: services maintain stable API contracts. The mechanism: every CI build runs contract tests. Breaking the contract fails the build. The build failure is immediate, visible, and costly. The cost of defection — changing the API without updating callers — is brought forward to the moment of the change. The mechanism implements the outcome. Truthful revelation — "I changed the API" — is enforced by the test.

SLAs with penalty clauses. The desired outcome: services maintain availability targets. The mechanism: an SLA defines the target and the penalty for missing it. The penalty makes degradation costly to the provider. The cost aligns the provider's incentive with the consumer's need. The mechanism implements the outcome. The SLA is the contract. The penalty is the enforcement.

Deployment gates. The desired outcome: only tested, reviewed code reaches production. The mechanism: the deployment pipeline requires passing tests, code review approval, and a staging verification period. Each gate is a rule. The rules collectively implement the outcome. Bypassing a gate is possible but visible. Visibility creates accountability. Accountability enforces compliance.

Code review requirements. The desired outcome: all code is reviewed before merge. The mechanism: the repository requires an approving review. The mechanism is enforced by the platform. The enforcement is automatic. The automatic enforcement removes the human decision. The removal is mechanism design — design the rules so the desired behavior is the only possible behavior.

The designer's question

Mechanism design gives the software engineer a question to ask: what outcome do I want, and what rules would produce it? The question is more powerful than "what should we do?" because it acknowledges that people respond to incentives. Telling people to cooperate produces cooperation if people are cooperative. Designing a mechanism where cooperation is the dominant strategy produces cooperation regardless. The mechanism doesn't require virtue. It requires structure. Structure is more reliable than virtue. Design the structure.


References:

  • Leonid Hurwicz, "Optimality and Informational Efficiency in Resource Allocation Processes," 1960.
  • Eric Maskin, "Nash Equilibrium and Welfare Optimality," Review of Economic Studies, 1977.
  • Roger Myerson, "Optimal Auction Design," Mathematics of Operations Research, 1981.
  • Related posts: Design the Game, Field Guide to Scarcity Games

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

Evolutionary Game Theory

Maynard Smith asked: what if players don't choose strategies rationally but inherit them genetically? Evolutionary game theory replaces rational choice with replicator dynamics. Strategies that perform well reproduce. Strategies that perform poorly die out. The Evolutionarily Stable Strategy cannot be invaded by mutants. This is why microservices survived and monoliths didn't.

game-theoryevolutionarymaynard-smithreplicator-dynamicsESS

John Maynard Smith published Evolution and the Theory of Games in 1982. He asked: what if players don't choose strategies rationally but inherit them genetically? The question launched evolutionary game theory. The key concepts: replicator dynamics and the Evolutionarily Stable Strategy (ESS).

Replicator dynamics describe how the proportions of different strategies in a population change over time. Strategies that earn above-average payoffs grow. Strategies that earn below-average payoffs shrink. The growth rate is proportional to the difference between the strategy's payoff and the population average. The dynamics are deterministic. Given the initial proportions and the payoff matrix, the trajectory is determined.

An Evolutionarily Stable Strategy is a strategy that, if adopted by the entire population, cannot be invaded by any small group of mutants playing a different strategy. The ESS is a refinement of Nash equilibrium for biological contexts. Every ESS is a Nash equilibrium. Not every Nash equilibrium is an ESS. The ESS requires that if a mutant strategy appears, it does worse against the existing population than the existing strategy does. The condition is stricter than Nash. The strictness is appropriate for evolution, where stability means resistance to invasion, not just mutual best response.

Interpretations from different branches

Biology (Maynard Smith, 1982). The hawk-dove game models animal conflict. Hawks fight for resources. Doves display but retreat if attacked. Hawk vs. Hawk: one wins, one is injured. Hawk vs. Dove: Hawk wins. Dove vs. Dove: both share. The ESS is a mixed population — some proportion of Hawks, some of Doves. The proportion depends on the value of the resource relative to the cost of injury. The prediction matches observed behavior in multiple species. Evolution is a game. The game has an equilibrium. The equilibrium is the observed behavior.

Economics (evolutionary game theory, Nelson and Winter, 1982). Firms don't optimize. They follow routines — organizational habits inherited from the past. Routines that produce profits survive. Routines that produce losses are replaced. The market is the selection environment. The replicator is the firm's growth rate. The ESS is the industry equilibrium. The dynamics explain why industries converge on similar practices. The practices are not optimal in any absolute sense. They are stable against invasion by alternatives.

Anthropology (Boyd and Richerson, 1985). Culture evolves through imitation and social learning. Cultural variants — beliefs, practices, technologies — are strategies. Successful variants are copied. Unsuccessful variants are abandoned. The replicator dynamics are cultural transmission. The ESS is a cultural equilibrium. The dynamics explain why some cultural practices persist despite being individually costly — they are stable against invasion by alternatives, even if alternatives would be better for individuals.

Computer science (genetic algorithms, classifier systems). Evolutionary computation uses replicator dynamics to solve optimization problems. A population of candidate solutions. Fitness evaluation. Selection. Recombination. Mutation. Repeat. The algorithm is replicator dynamics implemented in code. The solutions evolve. The evolution finds optima that gradient-based methods miss. The search is stochastic. The convergence is evolutionary.

Software engineering interpretations

Architecture pattern evolution. Microservices emerged as a mutant strategy. The monolith was the incumbent. Early microservices adopters demonstrated advantages — independent deployability, team autonomy, fault isolation. Other teams observed the success and adopted. The replicator dynamics: adoption rate proportional to observed advantage. The ESS: microservices are now the default for new systems. The monolith couldn't resist the invasion. Not because microservices are universally better. Because they are stable against re-invasion by the monolith pattern. Once adopted, the switching cost back to monolith is high. The switching cost is the invasion barrier.

Language and framework adoption. React emerged as a mutant in a jQuery-dominant ecosystem. Early adopters demonstrated advantages — component model, virtual DOM, unidirectional data flow. Adoption accelerated as the advantages became visible. jQuery declined. The replicator dynamics: framework popularity follows relative fitness. The fitness landscape changes as tooling, community, and hiring markets co-evolve. Today's ESS may not be tomorrow's. The dynamics continue.

Process evolution. Agile emerged as a mutant in a waterfall-dominant ecosystem. Early adopters demonstrated faster delivery, better responsiveness to change. Adoption accelerated. Waterfall declined — not eliminated, but no longer the default. The replicator dynamics: process fitness is measured by organizational survival. Organizations that adopted agile survived at higher rates. The surviving organizations shaped the hiring market. The hiring market shaped the training pipeline. The pipeline reinforced agile. The equilibrium is self-reinforcing.

The persistence of patterns. Some architecture patterns persist despite being widely criticized. The distributed monolith — microservices with tight coupling — persists because it is an ESS in certain environments. A team that inherits a distributed monolith can't unilaterally refactor to clean boundaries. The refactoring requires coordination across teams. The coordination is a public good. The individually rational choice is to work within the existing architecture. The equilibrium is suboptimal but stable. Stability doesn't mean goodness. It means resistance to change.

The evolutionary lens

Evolutionary game theory gives the software engineer a lens: what strategies are growing in the population? What strategies are stable against invasion? The dominant architecture, the dominant language, the dominant process — they are not necessarily optimal. They are evolutionary equilibria. They persist because they resist invasion. Understanding the invasion barrier — switching cost, coordination cost, network effects — explains why they persist. Changing them requires lowering the barrier or raising the fitness of the alternative. The lowering is mechanism design. The raising is engineering. Both are evolutionary.


References:

  • John Maynard Smith, Evolution and the Theory of Games, Cambridge University Press, 1982.
  • John Maynard Smith and George Price, "The Logic of Animal Conflict," Nature, 1973.
  • Richard Nelson and Sidney Winter, An Evolutionary Theory of Economic Change, Harvard University Press, 1982.
  • Related posts: Scarcity and Games, Field Guide to Scarcity Games

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

Chicken

Two drivers speed toward each other. The first to swerve loses. If neither swerves, both die. Chicken models brinkmanship — the game where the worst outcome is mutual stubbornness. In software: conflicting rewrites, deploy races, and the organizational politics of who yields.

game-theorychickenbrinkmanshipcredible-commitmentorganizational-politics

Two drivers speed toward each other on a narrow road. The first to swerve is the chicken — loses status, loses the game. If both swerve, both lose a little. If neither swerves, both crash. The payoff matrix:

Swerve Straight
Swerve (0, 0) (-1, +1)
Straight (+1, -1) (-10, -10)

There are two pure-strategy Nash equilibria: (Straight, Swerve) and (Swerve, Straight). Both are asymmetric — one player wins, one loses. The symmetric outcome (Swerve, Swerve) is not an equilibrium because each player would prefer to deviate to Straight. The symmetric outcome (Straight, Straight) is disaster.

The game is about who commits first. If you can credibly commit to going straight — by throwing your steering wheel out the window, visibly, where the other driver can see — the other driver must swerve. The commitment must be credible. A verbal threat is not credible. The thrown steering wheel is. The commitment changes the game from simultaneous to sequential. The sequential game has a unique outcome: the committed player goes straight, the uncommitted player swerves.

Interpretations from different branches

Classical game theory. Chicken has two asymmetric Nash equilibria. The equilibrium selection problem is about who can credibly commit first. The commitment must be observable and irreversible. Schelling analyzed this extensively in The Strategy of Conflict — the ability to worsen one's own options can be strategically advantageous because it forces the opponent to concede.

Nuclear strategy (Schelling, RAND). Chicken was the dominant metaphor for Cold War nuclear brinkmanship. Two superpowers, each threatening mutual destruction. The one that could credibly commit to retaliation — by making retaliation automatic, by removing the human decision from the loop — gained strategic advantage. "Threats that leave something to chance" was Schelling's phrase. The threat of mutual destruction must be credible. If it's credible, the other side swerves.

Behavioral game theory. In laboratory play, Chicken produces higher rates of "swerve" than Nash predicts, especially among players who have played before and learned that mutual stubbornness produces disaster. Experience teaches swerving. The learning is expensive. The disaster teaches it.

Organizational theory. Chicken models inter-team conflict. Two teams both want to rewrite the same shared service. Both have started work. Neither wants to abandon their effort. If both continue, the organization gets two incompatible rewrites and a migration mess. Someone must swerve. Who swerves is determined by organizational hierarchy, political capital, or the urgency of each team's deliverable. The hierarchy is a mechanism for resolving Chicken. The mechanism is informal. The crashes are frequent.

Software engineering interpretations

Friday deploy races. Two teams both want to deploy Friday afternoon. Both know Friday deploys are risky. Both want their feature in. If both deploy and nothing breaks, both win. If both deploy and something breaks, both lose their weekend. The equilibrium before the "no Friday deploys" rule was Friday deploys by the team with the most political capital. The rule resolved the game by removing the choice. Mechanism design.

Conflicting rewrites. Two teams independently decide to rewrite the same legacy service. Both have invested weeks. Both present their work at demo day. Neither wants to be the team that wasted effort. The resolution: one rewrite is adopted, the other is shelved. The shelved team swerved. The swerve was forced by the architect. The architect is the mechanism.

Competing architecture proposals. Two senior engineers propose incompatible architectures for the same system. Both have strong opinions. Both have supporters. The debate is unresolved. Someone must swerve. If neither does, the organization forks the system or deadlocks. The CTO resolves it. The CTO is the mechanism. The mechanism is hierarchical. The hierarchy exists partly to resolve Chicken games.

On-call escalation. An incident is ongoing. Two engineers disagree about the fix. The clock is ticking. The disagreement is Chicken in miniature — each sticking to their approach risks extended downtime. The escalation policy — "after 10 minutes of disagreement, escalate to the on-call lead" — is a mechanism for resolving Chicken. The policy predetermines who has the authority. The predetermination prevents the debate.

Resolving chicken

Chicken is resolved by mechanisms that predetermine who yields. Pre-commitment — "I will not swerve" — works if credible. Organizational hierarchy — "the architect decides" — works if respected. Pre-agreed rules — "no Friday deploys" — work if enforced. The mechanism changes the game from a simultaneous contest of wills to a sequential game where the first mover's commitment is binding. The binding commitment selects the equilibrium. The equilibrium prevents the crash.


References:

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

The Battle of the Sexes

Two players want to coordinate but disagree on which coordinated outcome to choose. Both prefer coordination to miscoordination. Both prefer their own choice. The Battle of the Sexes models every argument about which technology to standardize on, which queue to use, and which convention to adopt. The answer isn't better arguments. It's a selection mechanism.

game-theorybattle-of-sexescoordinationfocal-pointsstandards

A couple wants to spend the evening together. One prefers the opera. The other prefers the football game. Both prefer being together at the less-preferred event to being apart at the preferred one. The payoff matrix:

Opera Football
Opera (3, 2) (0, 0)
Football (0, 0) (2, 3)

Two pure-strategy Nash equilibria: both go to Opera, both go to Football. Both are Pareto-efficient — neither can be improved without harming the other. Neither equilibrium is obviously better than the other without a way to weigh the two players' preferences. The game has a mixed-strategy equilibrium as well, but it is inefficient — sometimes they miscoordinate. The challenge is selecting which coordinated outcome without an external authority.

Interpretations from different branches

Classical game theory. The Battle of the Sexes is a coordination game with conflicting interests. The equilibria are Pareto-rankable only by interpersonal utility comparison, which classical game theory avoids. Equilibrium selection requires something beyond the payoff matrix: a focal point, a convention, a first-mover advantage, or a bargaining process.

Focal points (Schelling). In the absence of communication, players coordinate by finding what is obvious. "We'll go to the opera because it's Tuesday and the opera is on Tuesdays." The day is a focal point. It is arbitrary. It works because both players know the other is looking for it. The power of the focal point is not in its logic. It is in its obviousness.

Bargaining theory (Nash). If players can communicate, they can bargain. The Nash bargaining solution splits the surplus according to relative bargaining power. The player with better outside options — could go to the opera alone and enjoy it more than the other would enjoy football alone — has more bargaining power. The solution predicts the outcome based on the disagreement point.

Evolutionary game theory. In a population playing Battle of the Sexes, which equilibrium emerges depends on initial conditions and the speed of adaptation. If a slight majority initially goes to Opera, the minority has incentive to switch. The equilibrium is path-dependent. History matters. The equilibrium that emerges is the one that got a small initial advantage.

Feminist economics (Akerlof, Sen). The Battle of the Sexes models the gendered division of labor. In traditional households, the wife's preferences are systematically discounted. The equilibrium that emerges — "his" choice — is not a reflection of equal bargaining. It is a reflection of unequal outside options. The wife cannot credibly threaten to go alone because her outside option is worse. The bargaining power is structural. The structure is social.

Software engineering interpretations

Message queue choice. Team A wants NATS. Team B wants Kafka. Both prefer either queue to no queue. Both prefer their own choice. The game has two Nash equilibria: both NATS or both Kafka. Which equilibrium is selected depends on who has more organizational power, who moves first, or who cares more. The selection mechanism is political. The politics are game-theoretic.

Language choice for a new service. The team is split between Go and Rust. Both prefer a unified language to fragmentation. Both prefer their own. The equilibrium: the language chosen by the tech lead, or the language chosen by the team that builds the first service (first-mover advantage), or the language that has the most internal library support (focal point).

Meeting time selection. Team members across time zones need a recurring meeting slot. Everyone prefers any slot to no meeting. Everyone prefers their own working hours. The equilibrium: the slot that works for the person with the most organizational power, or the slot suggested first, or the slot that's "obvious" — Tuesday 10am, the universal default.

Deploy schedule. Three teams share a deploy window. Each prefers a different day. All prefer a coordinated schedule to conflicting deploys. The equilibrium: the day claimed by the team that shipped first, or the day assigned by the release manager (hierarchy), or Monday (focal point — start of the week).

Resolving the Battle of the Sexes

The Battle of the Sexes is resolved by a selection mechanism. Any mechanism will do. The mechanism must be accepted as legitimate by both players, or it won't be accepted. Hierarchy: the architect decides. Convention: we always use what the first team chose. Rotation: we alternate. External authority: the CTO mandated NATS. Market: whichever technology has the most internal tooling wins. Commitment: Team A already deployed to production with NATS; Team B can join or build their own queue.

The mechanism that works is the one both players will accept. Acceptance is the constraint. The constraint is social. The social is game-theoretic.


References:

  • R. Duncan Luce and Howard Raiffa, Games and Decisions, Wiley, 1957.
  • Thomas Schelling, The Strategy of Conflict, Harvard University Press, 1960.
  • John Nash, "The Bargaining Problem," Econometrica, 1950.
  • Related posts: Scarcity and Games, Field Guide to Scarcity Games

Game theory is engineering when applied to systems design. The players are components. The strategies are behaviors. The payoffs are performance metrics. The equilibrium is the system's steady state. The mechanism designer is the engineer — designing rules that produce desired outcomes without controlling individual decisions. Every protocol, every API contract, every rate limiter is mechanism design in code. The game is the system. The rules are the architecture.

Field Guide to Scarcity Games

An exhaustive taxonomy of game types, organized by cooperation, conflict, timing, information, symmetry, player count, determinism, strategy, state space, time horizon, and special classes. Every category has a concrete software example. A reference for recognizing game structures in architecture.

game-theorytaxonomyreferencesoftware-engineering

Every software situation is a game. The game has a type. The type determines the appropriate analysis. This catalog classifies games along eleven independent dimensions. Each category includes a concrete software example. Use it to locate any situation in game-theoretic space. The location tells you what you're dealing with. What you're dealing with determines what you should do.

This catalog is a diagnostic tool. When a situation feels stuck — the migration isn't happening, the standard isn't being adopted, the teams keep breaking each other's APIs — locate it here. The name of the game tells you why it's stuck. The why tells you what to change. Most organizational interventions fail because they treat every stuck situation as a communication problem. Some are stag hunts. Some are chicken. Some are commons tragedies. The solution to a stag hunt is visible early adopters. The solution to chicken is a pre-committed rule. The solution to a commons tragedy is a governance mechanism. The solutions are different because the games are different. The catalog tells you which one you're in.

"Simplicity does not precede complexity, but follows it." — Alan Perlis

The catalog that follows is a map of the complexity. Use it to find the simplicity on the other side.

By cooperation

Category Description
Cooperative Players can form binding agreements. — SLA with penalty clauses. The SLA is the binding agreement.
Non-cooperative No binding agreements. — Teams without API contracts. Communication is cheap talk.
Cheap talk Communication allowed but unenforceable. — "We promise not to break the API." Without testing, this is cheap talk.
Bargaining Two-player cooperative game over surplus division. — Two teams negotiating shared infrastructure costs.
Team problems Non-cooperative structure, identical payoffs. — Team sharing the same OKRs. Incentives aligned by design.

By conflict of interest

Category Description
Zero-sum One's gain = another's loss. — Fixed headcount allocation. Every hire for Team A is one Team B doesn't get.
Non-zero-sum Players can both gain or both lose. — API design. Both gain from clean contract. Both lose from broken one.
Coordination No conflict. Players need to align. — Choosing a shared logging format. Everyone wants the same thing.
Mixed-motive Both conflict and cooperation incentives. — Most real situations. Shared goals but private priorities.

By timing

Category Description
Simultaneous Players move without observing others. — Teams making independent tech choices in the same quarter.
Sequential Players move in turn. — Deployment ordering. Stackelberg: first-mover advantage.
Repeated Same game multiple times. — Sprint planning. Repetition enables reputation and reciprocity.
One-shot Played exactly once. — A rewrite decision. No repetition.
Stochastic State evolves probabilistically. — Incident response. Alerts arrive randomly.

By information

Category Description
Complete information All players know all payoffs. — Open-source. Everyone sees the code, issues, priorities.
Incomplete information Private information. Bayesian games. — Team proposing rewrite has private info about true motivation.
Perfect information All previous moves known. — Monolith. Every module observes every other module's state.
Imperfect information Some moves unknown. — Microservices. Service A can't observe Service B's internal state.
Symmetric information Same uncertainty. — Both teams uncertain about new CTO's priorities.
Asymmetric information Different knowledge. — Senior engineer knows legacy weaknesses. New hire doesn't.

By symmetry

Category Description
Symmetric Payoffs depend only on strategies, not identity. — Two identical microservices with the same SLA.
Asymmetric Changing identities changes payoffs. — Frontend vs. backend team. Different constraints, different strategies.

By player count

Category Description
1-player Decision theory. No strategic interaction. — Choosing an algorithm. Nature is the constraint.
2-player Classic case. — Two teams negotiating an API contract.
N-player Three or more. Coalitions possible. — Organization with many teams. Alliances form. Politics emerges.
Large/Many-player Continuum. Atomic vs. non-atomic. — Open-source ecosystem. No single contributor changes equilibrium.
Mean field Players interact through state distribution. — Microservices at scale. Each service interacts with aggregate behavior.

By determinism

Category Description
Deterministic No chance elements. — Deterministic deployment pipeline.
Stochastic Some moves by nature/chance. — System with probabilistic failures.
Games of chance All moves by one player and chance. — A/B testing. Nature randomizes users.

By strategy type

Category Description
Pure strategy Single deterministic action. — Always use the same database for every service.
Mixed strategy Randomize over actions. — Random on-call assignment. Prevents predictable exploitation.

By state/action space

Category Description
Finite Finite actions and states. — Choosing between exactly three database options.
Infinite/Continuous Continuous spaces. — Allocating compute resources. Continuous variable.
Discrete-time Decisions at discrete intervals. — Sprint planning. Every two weeks.
Continuous-time Decisions at every instant. — Real-time auto-scaling. Differential equations.

By time horizon

Category Description
Finite horizon Known number of periods. — Project with fixed deadline and milestones.
Infinite horizon No predetermined end. — Ongoing system maintenance. Cooperation sustainable.
Discounted Future payoffs weighted less. — Quarterly planning. This quarter > next year.

Special game classes

Category Description
Signaling Informed player acts to reveal info. — Detailed RFC signals competence and seriousness.
Screening Uninformed player moves first. — Requiring prototype before architecture review.
Stackelberg Leader-follower. Leader first. — Platform team sets API standard. Services build against it.
Pursuit-evasion Zero-sum differential. — Intrusion detection. Attacker evades. Defender pursues.
Mechanism design Reverse game theory. — SLAs, contract testing, deployment gates, code review requirements.
Global games Noisy private signals of underlying state. — Teams deciding on new tech based on private maturity signals.
Combinatorial Finite, deterministic, perfect-info, 2P, zero-sum. — Automated theorem proving. Compiler optimization.
Evolutionary Fit strategies survive. Replicator dynamics. — Architecture patterns that persist. Microservices resist re-monolithing.
Partizan Moves differ per player. — Frontend and backend teams have different available moves.
Impartial Moves depend only on position, not player. — Identical services with identical deployment options.

This is part 7 of a 7-part series on scarcity and software.

Scarcity is the universal engineering constraint. Time, attention, compute, complexity — every engineering decision is made within a budget. The budget is economic. The engineer who doesn't track the budget makes decisions blind. The engineer who tracks it makes decisions with full knowledge of the trade-off. The trade-off is the decision. The budget is the constraint. Scarcity is the unifying principle.

Five Habits

How to apply scarcity thinking, game theory, and mechanism design to daily software engineering decisions. Five frameworks, five questions, five habits.

scarcitygame-theorymechanism-designpracticesoftware-engineering

The theory is useful. The practice is harder. Here are five frameworks for applying scarcity thinking, game theory, and mechanism design to daily software engineering decisions. The difference between knowing a principle and applying it is the difference between owning a cookbook and cooking dinner. Most engineers who read Robbins will agree that scarcity matters. Most will return to work and debate architectures without naming the scarce resource. The knowing is easy. The doing is hard. The gap between knowing and doing is a habit. Habits are built by repetition, not by insight. The five habits below are designed to be repeated until they're automatic. The goal is not to understand scarcity. The goal is to feel its absence like a missing step on a staircase.

Alan Perlis, the first Turing Award winner, wrote: "Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it." Most software teams are pragmatists. They suffer complexity because avoiding it requires upfront investment and removing it requires genius. The five frameworks that follow are for pragmatists who want to suffer less. Genius is not required. Discipline is.

1. Name the scarce resource

Before every architectural decision, ask: what is the scarce resource? Is it developer time? Optimize for simplicity of change. Is it compute? Optimize for efficiency. Is it attention? Optimize for clarity. Is it coordination capacity? Optimize for independent deployability. The scarce resource determines the optimal trade.

Most architectural debates skip this step. Two engineers debate two architectures. Neither names the constraint. The debate is unresolvable because the constraint is unstated. State the constraint. "We are optimizing for developer time, not compute." "We are optimizing for change velocity, not operational simplicity." The debate resolves. The constraint determines the answer. Name the constraint.

2. Identify the game

Every situation involving multiple players is a game. Ask: who are the players? What are their strategies? What are their payoffs? What do they know that others don't? Is the game cooperative or non-cooperative? Zero-sum or non-zero-sum? Simultaneous or sequential? One-shot or repeated?

Naming the game changes how you think about it. "This is a stag hunt — we all benefit if we coordinate, but if anyone defects we all lose" is different from "this is chicken — someone needs to swerve or we crash." The game type suggests the solution. Stag hunts need coordination mechanisms. Chicken needs a pre-committed rule about who yields. Prisoner's Dilemmas need repeated interaction and reputation. Battle of the sexes needs a selection mechanism. Name the game. The solution follows.

3. Design the mechanism

If you don't like the equilibrium, change the game. Mechanism design works backward from desired outcomes to the rules that produce them. Ask: what outcome do I want? What rules would produce it? What information do players need? What incentives do they face? How do I make defection visible and costly?

Automated contract testing is mechanism design. "No Friday deploys" is mechanism design. A booking calendar for staging is mechanism design. A test budget per service is mechanism design. Each mechanism changes the payoffs. Changed payoffs change behavior. Design the mechanism. Don't plead for the behavior.

4. Account for complexity

Complexity is a cost. It consumes attention, time, and future change capacity. Every feature adds complexity. The complexity has a present cost and a future cost — the accumulated drag on every subsequent change. The future cost is invisible in the current sprint. It is visible in year three.

Account for it. When estimating a feature, include the complexity cost. "This feature will take two weeks to build and will add complexity equivalent to 5% of the current system, which will cost approximately one week per quarter in reduced velocity. The NPV at our discount rate is negative. Don't build it, or build it simpler." Most teams don't do this math. The math exists. Do it.

5. Use the Robbins test

For any decision, ask Robbins's four questions: What is the end? What are the means? Are they scarce? Do they have alternative uses? If yes — and it always is — the decision is economic. Name the opportunity cost. "Building this feature means not building that one." Name the trade. Make the choice explicit. Explicit choices are better than implicit ones. Implicit choices are still choices. They're just choices made without awareness that a choice was being made.

Five habits

Habit 1: Before debating architecture, name the scarce resource. If you can't name it, you don't understand the trade.

Habit 2: Before negotiating with another team, model their scarcity. What are they optimizing for? What are they giving up? If you don't know, ask. Their answer will explain their behavior more than any personality trait.

Habit 3: Before adding a feature, calculate the complexity budget. How much does this cost in future velocity? Is the return worth the cost? Most features pass the "is it useful?" test. Few pass the "is it worth the complexity?" test.

Habit 4: Before accepting a broken process, ask what game it's producing. The process is a mechanism. The mechanism produces an equilibrium. If the equilibrium is bad, the mechanism is wrong. Change the mechanism. Don't blame the players.

Habit 5: Before choosing a technology, ask what game it enables. REST produces spatial coupling. NATS produces spatial decoupling. The technology is not neutral. It embeds assumptions about how players will interact. Choose the technology that produces the game you want to play.

The meta-habit

Scarcity thinking becomes a habit. You start seeing opportunity costs everywhere. You start modeling other teams' incentives before meeting them. You start recognizing game structures in organizational conflicts. You start designing mechanisms instead of pleading for behavior. The habit is the point. The tools — economics, game theory, mechanism design — are lenses. The lenses change what you see. What you see changes what you do.


This is part 6 of a 7-part series on scarcity and software.

References:

  • Lionel Robbins, An Essay on the Nature and Significance of Economic Science, Macmillan, 1932.
  • Barry W. Boehm, Software Engineering Economics, Prentice-Hall, 1981.

Scarcity is the universal engineering constraint. Time, attention, compute, complexity — every engineering decision is made within a budget. The budget is economic. The engineer who doesn't track the budget makes decisions blind. The engineer who tracks it makes decisions with full knowledge of the trade-off. The trade-off is the decision. The budget is the constraint. Scarcity is the unifying principle.

Design the Game

AI agents are players in games. Mechanism design is how we govern them. From deep RL auction design to LLM-based economic simulacra, the convergence of AI and mechanism design is the most important software engineering development of this decade.

aimechanism-designagentsreinforcement-learningauctionsgovernance

AI agents are becoming players in the games that software systems constitute. They make choices under scarcity. Their choices affect other agents. Other agents' choices affect them. This is game theory. Governing these agent interactions requires mechanism design — the branch of game theory that works backward from desired outcomes to the rules that produce them.

Mechanism design is the highest-leverage activity in software engineering. A line of code changes behavior once. A mechanism changes behavior every time the game is played. The "no Friday deploys" rule took one minute to write and has prevented a thousand weekend incidents. The automated contract test took a day to set up and has caught every breaking API change since. Mechanisms compound. Code depreciates. The mechanism is the investment. The behavior change is the return. The best engineers design mechanisms. The rest fix the same bug in a different context every sprint.

The convergence of AI and mechanism design is not a future trend. It is happening now. Deep learning is being used to design mechanisms. Mechanisms are being used to govern AI agents. The two fields are merging. Software engineers who understand both will build the infrastructure. Those who don't will consume it without understanding why it behaves as it does.

AI as mechanism designer

Traditionally, mechanism design was analytical. You proved that a given mechanism — an auction format, a voting rule, a matching algorithm — had certain properties: strategy-proofness, efficiency, individual rationality. The proofs were mathematical. The mechanisms were simple enough to analyze by hand.

Deep learning changed this. A mechanism is a function from reported preferences to outcomes. A neural network is a function approximator. Train a neural network to maximize a social objective — revenue, welfare, fairness — subject to incentive constraints, and you have a learned mechanism. The mechanism is a neural network. The properties are learned, not proved.

RegretNet (Google DeepMind, 2019) learns auction mechanisms for multi-bidder, multi-item settings where optimal mechanisms are analytically unknown. The network learns allocation and payment rules that are provably truthful and revenue-maximizing. The proof is not analytical. It is computational — the network's regret (the maximum gain from misreporting) is bounded during training. If regret is near zero, the mechanism is approximately strategy-proof. The approximation is good enough for practical use.

AI Economist (Salesforce, 2020-2022) uses deep RL to design tax policies. A social planner (the mechanism designer) and economic agents (workers) are trained in simulated economies. The planner learns tax schedules that balance productivity and equality. The learned policies recover classic theoretical results — the Saez optimal tax formula — while discovering novel hybrid policies that analytical methods missed. The planner is a neural network. The economy is a simulation. The tax policy is learned.

HCMD-zero (Google DeepMind, 2025) collects human preference data, trains neural models to imitate human voting behavior, and optimizes mechanisms against simulated human proxies. The mechanisms achieve high approval from real participants in public goods games. The humans never interact with the mechanism during training. The mechanism is designed against a model of humans. The model is learned from data. The design is computational.

The implication for software: any resource allocation problem — compute, bandwidth, storage, deployment slots, review capacity — can be framed as mechanism design. If the problem is too complex for analytical solution, deep learning can approximate the optimal mechanism. The mechanism is a model. The model allocates resources. The allocation is fair, efficient, and incentive-compatible by construction.

AI as game player

AI agents are not just designed by mechanisms. They are players in games. Multi-agent reinforcement learning (MARL) studies how learning agents interact in shared environments. The interactions produce emergent behavior. The behavior can be cooperative, competitive, or catastrophic.

The collusion problem. Kolumbus, Halpern, and Tardos (2024) showed that when RL agents in an auction are allowed to make side payments to each other outside the mechanism, they learn to collude. The auctioneer's revenue drops to near zero. The agents didn't communicate. They didn't coordinate explicitly. They learned that mutual restraint produced higher individual returns. The collusion was emergent. The emergence was game-theoretic. The mechanism designer must anticipate collusion and design against it.

The alignment problem as mechanism design. Aligning AI agents with human values is a mechanism design problem. The human is the principal. The agent is the agent. The principal wants the agent to take actions aligned with the principal's interests. The agent has private information — its capabilities, its true objective, its understanding of the task. The principal designs incentives — reward functions, oversight mechanisms, kill switches — to align the agent's behavior. The design is mechanism design. The principal is the mechanism designer. The agent is the strategic player.

LLM-based economic simulacra. Karten et al. (Princeton, 2025) framed optimal taxation as a Stackelberg game between an LLM planner and 100 LLM workers. Workers have census-calibrated skill distributions. The planner learns tax schedules by exploring bracket adjustments. Workers periodically vote to retain or replace the planner based on platform proposals. The governance is emergent. The voting is democratic. The planner is an LLM. The workers are LLMs. The economy is simulated. The tax policy is learned. This is mechanism design with AI agents on both sides.

Mechanism design for software infrastructure

Hayek's central insight was that prices communicate scarcity. "The price system is a mechanism for communicating information. The most significant fact about this system is the economy of knowledge with which it operates." A Vickrey auction for compute does what prices do in markets: it elicits truthful information about private valuations without requiring anyone to reveal anything beyond their bid. The bid is the price. The price communicates the scarcity. The mechanism processes the prices. The allocation emerges. No central planner knows the true value of compute to each team. The auction discovers it. Hayek would approve.

The principles apply directly to software infrastructure:

Compute allocation as auction design. Multiple teams compete for a shared compute cluster. Each team has private information about the value of its jobs. A central scheduler allocates compute. If the scheduler uses first-come-first-served, teams have incentive to misreport urgency. If the scheduler uses a Vickrey auction — second-price sealed-bid — truthful reporting is a dominant strategy. The auction is mechanism design. The scheduler is the mechanism. The teams are the bidders. The compute is the good.

API rate limiting as mechanism design. An API gateway limits requests per client. If the limit is fixed, clients have incentive to request the maximum regardless of need. If the limit uses a token bucket with rollover, clients smooth their usage. The token bucket is a mechanism. It incentivizes efficient use without requiring clients to report their true needs. The mechanism works because it aligns individual incentives with system-wide efficiency. The alignment is the point.

Service mesh traffic shaping as mechanism design. A service mesh routes traffic between services. If routing is round-robin, overloaded services receive as much traffic as idle ones. If routing uses least-connections with circuit breaking, traffic shifts away from degraded services. The routing policy is a mechanism. It incentivizes services to report their true state — by becoming slow when overloaded, they naturally receive less traffic. The mechanism is self-regulating. The regulation is emergent.

Code review allocation as mechanism design. Pull requests compete for reviewer attention. Reviewers are a scarce resource. If assignment is ad-hoc, authors lobby reviewers directly — costly signaling and political gaming. If assignment uses a queue with SLAs and automatic escalation, the mechanism allocates reviewer attention without requiring authors to compete. The queue is the mechanism. The SLA is the contract. The escalation is the enforcement. The allocation is fair by design.

The governance of agent fleets

As software systems become populated by AI agents — coding agents, testing agents, deployment agents, monitoring agents — the governance problem becomes acute. Each agent has objectives. The objectives may conflict. The agents may learn to collude, compete, or exploit vulnerabilities in the governance mechanism.

Agent Governance Toolkit (AGT). Microsoft's AGT, discussed earlier on this blog, provides policy evaluation, identity and trust primitives, execution sandboxes, and audit trails for agent fleets. This is mechanism design implemented as infrastructure. The policy engine is the mechanism. The agents are the players. The sandbox is the enforcement. The audit trail is the monitoring. The system treats agents as strategic actors operating under scarcity — of permissions, of compute, of access to resources. The scarcity is real. The mechanism governs it.

Task automation factories. The task automation economics paper argues that the economic unit is the verified automation asset — a released object with specification, evidence, and a defined interface. Assets are produced by agents, verified by agents, consumed by agents. The factory is a marketplace of agents exchanging verified assets. The marketplace needs mechanism design: how are assets priced? How is quality ensured? How are malicious or incompetent agents excluded? The answers are auction theory, reputation systems, and entry barriers. The questions are economic. The answers are mechanism design.

The convergence

Mechanism design and AI are converging because the problems they solve are the same problem: how to achieve desired outcomes when the agents producing the outcomes have their own objectives, private information, and strategic incentives. Robbins defined the problem in 1932: choice under scarcity. Von Neumann gave it mathematics in 1944: game theory. Mechanism design gave it engineering: design the game to produce the outcome. AI gave it scale: the agents are now software, the mechanisms are learned, the games are played at machine speed.

The software engineer who understands this convergence will design systems where agents cooperate by default, where incentives align with system goals, where emergent behavior is anticipated rather than discovered in production. The software engineer who doesn't will build platforms where agents collude, commons are overgrazed, and the system's behavior is a surprise. The surprise will be expensive. The theory is free.


This is part 5 of a 7-part series on scarcity and software.

References:

  • Tacchetti et al., "Deep Mechanism Design," PNAS, 2025.
  • Tonghan Wang, "Advancing Deep Learning for Multiagent AI," PhD thesis, Harvard, 2025.
  • Kolumbus, Halpern & Tardos, "Paying to Do Better: Games with Payments between Learning Agents," 2024.
  • Karten et al., "LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacra," Princeton, 2025.
  • Related posts: Agent Governance Toolkit, Task Automation Economics

Scarcity is the universal engineering constraint. Time, attention, compute, complexity — every engineering decision is made within a budget. The budget is economic. The engineer who doesn't track the budget makes decisions blind. The engineer who tracks it makes decisions with full knowledge of the trade-off. The trade-off is the decision. The budget is the constraint. Scarcity is the unifying principle.

The Architecture Is a Game

Every software situation is a game. The service boundary game, the microservices migration game, the deployment chicken game, the logging stag hunt — each has a game-theoretic structure. Recognizing it tells you what to do.

game-theorysoftware-architecturemicroservicescoordination

The most dangerous games are the ones you don't know you're playing. Nobody told you that logging format choice was a stag hunt. Nobody announced that the rewrite proposal was a signaling game. The games are invisible. Their outcomes are visible — the broken APIs, the fragmented standards, the stalled migrations. The outcomes are blamed on individuals. "Team B should have communicated better." "The SRE team should have enforced the standard." But the individuals were playing rationally given the game they were in. Blaming the player is easier than recognizing the game. Recognizing the game is easier than changing it. Change the game.

Software engineering is a multiplayer game. The players are teams, services, organizations, companies. Each player makes choices under conditions of scarcity. Each player's outcome depends on the choices of others. Recognizing the game you're in is the first step to playing it well.

The service boundary game

Thomas Schelling observed that "what makes many agreements enforceable is only the recognition of future opportunities for agreement that will be eliminated if mutual trust is not created and maintained, and whose value outweighs the momentary gain from cheating in the present instance." API contracts are agreements. Automated testing makes cheating visible. Visibility eliminates the momentary gain. The contract becomes enforceable because the future cost of breaking it exceeds the present benefit. This is mechanism design as Schelling described it before the term existed.

Team A builds Service A. Team B builds Service B. Service A depends on Service B. Team B changes Service B's API without telling Team A. Service A breaks. Team A is angry. Team B is surprised.

This is a coordination game with asymmetric information. Team B didn't know what Team A depended on. Team A assumed Team B wouldn't change the API without notice. The assumptions were incompatible. The failure is a Nash equilibrium — neither team can unilaterally improve their outcome given what the other is doing. Team A can't make Team B communicate better. Team B can't make Team A depend on fewer things. The equilibrium is suboptimal. The solution is mechanism design: automated contract testing that makes breaking changes immediately visible. Visibility changes the payoff. The equilibrium shifts.

The microservices migration game

The monolith works but is increasingly expensive to change. Each team wants to extract their service. Extraction requires coordination with other teams. Coordination is costly. The cost falls on the team doing the extraction. The benefit accrues to all teams.

This is a public goods game. Each team would benefit if everyone extracted. Each team would prefer someone else pay the coordination cost. Individually rational: wait. Collectively optimal: coordinate. The gap is the Prisoner's Dilemma scaled to organizational architecture. The dilemma produced the distributed monolith — services extracted without clean boundaries, communicating through a complete call graph, with all the monolith's coupling and all the network's latency. Nobody wanted this. The game produced it.

The deployment chicken game

Two teams both want to deploy Friday afternoon. Both know Friday deploys are risky — if something breaks, on-call spends the weekend fixing it. Both want their feature in before the weekend. If both deploy and nothing breaks, both win. If both deploy and something breaks, both lose. If one deploys and one waits, the deployer wins and the waiter deploys Monday.

Both deploying is the crash outcome. The organizational rule "no Friday deploys" is mechanism design. It removes the choice. Before the rule: Friday deploys. After: Monday deploys. The mechanism changed the equilibrium.

The staging environment battle of the sexes

Team A wants staging for integration testing. Team B wants staging for customer demos. Both want staging. Both prefer any coordinated solution to constant conflict. Both prefer their own preferred time.

Battle of the sexes. Two Nash equilibria: Team A's schedule or Team B's schedule. The solution is a coordination mechanism — a booking calendar, a dedicated demo environment, a policy. The mechanism selects an equilibrium. Before: conflict. After: coordination. The mechanism worked because it made defection visible.

The logging library stag hunt

Twelve services. Each uses its own logging format. SRE proposes a standard library. If all adopt, logs become queryable across services. If some adopt and others don't, adopters get no benefit — their logs are standardized but they still can't query across services. If nobody adopts, nothing changes.

Stag hunt. The stag is cross-service observability. The rabbit is keeping your own format. The stag hunt succeeds when early adopters reach critical mass. Once enough services adopt, the benefit to remaining services exceeds switching cost. The equilibrium tips. The tipping point is a property of network effects.

The rewrite signaling game

Team A proposes rewriting a legacy service. The proposal is costly to evaluate — architecture reviews, specs, stakeholder meetings. Team A has private information: genuine belief the rewrite is necessary, or boredom with legacy work.

Signaling game. A costly signal separates sincere from insincere. Requiring a working prototype before the architecture review is a costly signal. Sincere teams pay the cost. Insincere teams won't. The signal screens. Pay the price. Get the review.

The regression test commons game

The regression test suite is a common-pool resource. Everyone benefits from tests. Everyone benefits from fast tests. Adding tests makes the suite slower. The cost is shared. The benefit of your test is yours alone.

Individually rational: add tests. Collectively optimal: add only high-value tests. The commons is overgrazed. The suite grows. Build times increase. Hardin described the tragedy in 1968. Your CI pipeline is living it. The solution: a test budget per service, periodic culling of low-value tests, a rule requiring historical failure catch rate to justify additions. The mechanism is the institutional response to a tragedy of the commons.

The open-source game

Everyone benefits from open-source software. Contributing costs time. Using costs nothing. Individually rational: use without contributing. Collectively optimal: everyone contributes. Public goods game. Resolved by reputation, corporate sponsorship, intrinsic motivation. The free-rider problem is managed, never solved. GitHub sponsorships, open-source foundations, corporate OSPOs — the institutions exist because the game exists. The game exists because scarcity exists.

The platform pricing Stackelberg game

A platform team sets API pricing for internal services. Service teams respond by choosing how much to consume. The platform team moves first. Service teams move second.

Stackelberg game. The leader (platform) chooses price anticipating the followers' (services') responses. Set price too high: services build their own. Too low: platform is underfunded. The optimal price is where marginal cost of providing the service equals the marginal value to consumers. The calculation is economic. The implementation is an internal API with metered billing. The billing is mechanism design.

Recognize the game

Every situation is a game. The game has a structure. The structure determines the likely outcome. If you don't like the outcome, change the game. Mechanism design is the tool for changing games. Automated contract testing changes the service boundary game. "No Friday deploys" changes the deployment chicken game. A booking calendar changes the staging environment game. A test budget changes the regression test commons game. Each mechanism changes the payoffs. Changed payoffs change behavior. Changed behavior is the point.


This is part 4 of a 7-part series on scarcity and software.

References:

Scarcity is the universal engineering constraint. Time, attention, compute, complexity — every engineering decision is made within a budget. The budget is economic. The engineer who doesn't track the budget makes decisions blind. The engineer who tracks it makes decisions with full knowledge of the trade-off. The trade-off is the decision. The budget is the constraint. Scarcity is the unifying principle.

No Free Lunch

Barry Boehm named the field in 1981. This post defines every economic concept that applies to software — opportunity cost, sunk cost, NPV, option value, technical debt, build vs. buy — and shows how Brooks, Parnas, and Lehman were economists before the field existed.

software-economicsboehmopportunity-costtechnical-debtnpv

Barry Boehm published Software Engineering Economics in 1981. The book applied cost estimation, net present value, and decision analysis to software projects. It named a field that had been practiced without a name since the first programmer decided which feature to build first. The vocabulary is precise. Most engineers use it without knowing the definitions. The definitions matter.

Milton Friedman distilled economics to one sentence: "There's no such thing as a free lunch." Every feature has a cost. The cost is not just the time to build it. It is the time plus the complexity it adds plus the features you didn't build instead. The lunch appears free. The bill arrives in year three.

Every organization has a discount rate — the rate at which it discounts the future relative to the present. The discount rate is the most important number in your technical culture and the one nobody knows. A high discount rate means next quarter matters more than next year. Features ship. Refactors don't. Complexity accumulates. A low discount rate means sustainability matters. The refactor gets done. The feature waits. Neither rate is correct in the abstract. The correct rate depends on whether the company will exist in five years. The discount rate should be a conscious choice. It never is. It is set by the urgency of the nearest deadline. The deadline is a discount rate of nearly infinity. Infinity is too high.

The terminology

Opportunity cost. The value of the best alternative foregone. Every hour on Feature A is an hour not on Feature B. The cost of Feature A is the time to build it plus the value of whatever you would have built instead. Opportunity cost is invisible. It appears on no invoice. It is the largest cost in software engineering. The features you didn't build are the cost of the features you did.

Sunk cost. A cost already incurred and unrecoverable. The three years on the monolith are sunk. They should not influence the migration decision. They do. The influence is irrational. The irrationality is human. The only defense is to externalize the decision to a process that doesn't know the sunk cost.

Marginal cost. The cost of one additional unit. The marginal cost of a user is near zero for software. The marginal cost of complexity is not zero — adding a feature to an already-complex system costs more than adding it to a simple one. Software has decreasing marginal cost of serving users and increasing marginal cost of adding features. The first makes software businesses attractive. The second makes them eventually unmaintainable.

Comparative advantage. Produce what you're relatively better at. Trade for the rest. Ricardo's logic, applied to microservices. A team better at both frontend and backend should specialize in whichever it has the greater relative advantage in. Absolute advantage doesn't matter. Comparative advantage does. Most teams don't know theirs. They assume they should build everything. They are wrong.

Economies of scale. Cost per unit decreases as volume increases. The monolith has economies of scale: one build pipeline, one deployment. Microservices lose these in exchange for independent deployability. The optimal number of services is where the marginal benefit of independence equals the marginal cost of lost scale. Few calculate this. Most guess. The guess is usually wrong.

Diseconomies of scale. Cost per unit increases as volume increases. Brooks's Law is a diseconomy: adding people increases output by less than the increase in coordination cost. Net effect negative. Diseconomies of scale are why organizations don't grow infinitely. At some size, internal coordination cost exceeds external transaction cost. Coase (1937): firms exist because internal transactions are cheaper than market transactions. The firm's boundary is where internal cost equals market cost. The service's boundary is where building equals buying.

Net present value (NPV). The current value of future cash flows, discounted by the time value of money. A refactoring costing $100K now and saving $20K/year for ten years has NPV dependent on the discount rate. At 5%, positive. At 15%, negative. The discount rate is the organization's preference for present over future. Organizations with high discount rates don't refactor. The rate is set by quarterly earnings pressure. The pressure is economic. The decision not to refactor is economic, stated in technical language.

Option value. The value of keeping a choice available. A clean interface has option value: change the implementation later without changing callers. The option costs more upfront. The option is valuable — being able to change without coordination. Financial options have a market price (Black-Scholes). Software options don't. They have an implicit value estimated by the architect. Good architects price options correctly. Bad architects don't know they're pricing options.

Technical debt as economic debt. Borrowing future productivity to increase present velocity. Principal: the cleanup work required. Interest: reduced velocity from uncleaned code. Interest compounds. Compounding interest eventually consumes all available velocity. The system becomes unchangeable. The debt must be repaid or defaulted on. Default is a rewrite. The rewrite is bankruptcy. The accounting that didn't track the debt was wrong.

Cost of delay. Revenue or value lost per unit time by not shipping. If Feature A generates $10K/week and takes 10 weeks, cost of delay is $100K. If Feature B generates $2K/week, cost of delay is $20K. Build A first. Most prioritization ignores cost of delay. They prioritize by effort, by intuition, by the loudest stakeholder. That framework is not a framework. It is ritualized negotiation.

Build vs. buy. Make-or-buy. Build if internal cost < market cost, adjusted for risk, control, and strategic value. Coase: the boundary of the firm is where internal cost equals market cost. Teams that build everything have not calculated the boundary. They have assumed it.

The economists who didn't know they were economists

Tony Hoare observed: "There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." The difficult method costs more upfront. The easy method costs more over time. The economics of software is the economics of choosing which cost to pay. Most choose the easy method. The easy method is why refactoring exists.

Brooks. Conceptual integrity is an economic argument. One mind controls the design because coordination cost exceeds the benefit of additional designers. The n(n−1)/2 communication paths make adding designers counterproductive. Brooks's Law is a statement about the diseconomy of scale in software teams. Brooks was doing economics without the vocabulary.

Parnas. Information hiding is an economic strategy. Invest in a stable interface. The return is reduced propagation of change. The investment costs more upfront. The return accrues over time. The net present value is positive if the decision is sufficiently volatile. Parnas was pricing options without Black-Scholes.

Lehman. E-type systems must evolve. Evolution increases complexity unless work is done to reduce it. The work costs time and attention — both scarce. The complexity budget is finite. The accounting that ignores it produces systems cheap to build and expensive to maintain. Lehman was doing cost accounting without the ledger.

Christensen. Incumbents are held captive by their customers. The captivity is rational under conditions of scarcity. The resources that could fund the disruption are allocated to sustaining innovations for existing customers. The allocation is optimal in the short run. Fatal in the long run. Christensen was describing capital allocation under asymmetric constraints without calling it that.


This is part 3 of a 7-part series on scarcity and software.

References:

  • Barry W. Boehm, Software Engineering Economics, Prentice-Hall, 1981.
  • R.H. Coase, "The Nature of the Firm," Economica, 1937.
  • Frederick P. Brooks, Jr., The Mythical Man-Month, Addison-Wesley, 1975.
  • David L. Parnas, "On the Criteria to Be Used in Decomposing Systems into Modules," Communications of the ACM, 1972.
  • M.M. Lehman, "Programs, Life Cycles, and Laws of Software Evolution," Proceedings of the IEEE, 1980.
  • Clayton M. Christensen, The Innovator's Dilemma, Harvard Business School Press, 1997.

Scarcity is the universal engineering constraint. Time, attention, compute, complexity — every engineering decision is made within a budget. The budget is economic. The engineer who doesn't track the budget makes decisions blind. The engineer who tracks it makes decisions with full knowledge of the trade-off. The trade-off is the decision. The budget is the constraint. Scarcity is the unifying principle.

The Game You're Already Playing

Von Neumann gave scarcity mathematics in 1944: choice under scarcity, when strategic, is a game. This post defines every game theory concept that matters to software, from zero-sum to mechanism design.

game-theoryvon-neumannnashsoftware-engineering

If Robbins defined the problem, John von Neumann gave it mathematics. In 1928, he published "On the Theory of Parlor Games." In 1944, with Oskar Morgenstern, he published Theory of Games and Economic Behavior. The connection to Robbins is direct. Robbins said economics is choice under scarcity. Von Neumann said: choice under scarcity, when the outcome depends on the choices of others, is a game.

If your choice were independent of others' choices, it would be optimization. Because it depends on others' choices, it is strategy. Strategy is optimization under uncertainty about what other players will do. Von Neumann's first contribution was the minimax theorem for zero-sum games: every two-player zero-sum game has an optimal strategy — minimize your maximum possible loss. In purely competitive situations, there is a rational strategy. Compute it. Follow it.

But most real situations are not zero-sum. Nash extended the framework in 1950 with the Nash equilibrium — a state where no player can benefit by unilaterally changing strategy, given what everyone else is doing. The equilibrium is not necessarily optimal. The Prisoner's Dilemma, formalized by Flood, Dresher, and Tucker, shows that individually rational strategies can produce collectively worse outcomes than cooperation. The gap between individual rationality and collective optimality is where economics lives. It is also where software architecture lives.

Schelling won his Nobel for a deceptively simple observation: in games without communication, people coordinate by finding what is obvious. The power is not in being right. It is in being obvious. The team that picks the most obvious API convention doesn't win because the convention is best. They win because everyone else predicted they would pick it. Obviousness is a coordination technology. The best architects don't design the optimal interface. They design the interface everyone will predict they designed. The prediction does the coordination. The interface just has to be what was predicted.

The definitions

Game. A set of players, strategies for each, and a payoff function mapping strategy combinations to outcomes. Your microservices architecture is a game. Players: services. Strategies: API designs, deployment schedules, dependency choices. Payoffs: uptime, throughput, maintainability.

Zero-sum game. One player's gain is another's loss. Total payoff constant. Fixed headcount across teams is zero-sum: every hire for Team A is a hire Team B doesn't get.

Non-zero-sum game. Players can both gain or both lose. Total payoff varies with cooperation. Most software situations are non-zero-sum. Both teams benefit from a clean API. Both lose from a broken one.

Cooperative game. Players can form binding agreements. An SLA with penalty clauses is a binding agreement. Non-cooperative: no binding agreements. API contracts without automated testing are cheap talk. Cheap talk doesn't change equilibria.

Perfect information. Each player knows all previous moves. Monolith: every module can see every other module's state. Imperfect information: players don't know all previous moves. Microservices: Service A doesn't know Service B's internal state. Imperfect information produces coordination failures. The failures are not bugs. They are properties of the information structure.

Simultaneous game. Players choose without observing others. Teams making independent technology choices in the same quarter. Sequential game: players move in turn, observing previous moves. Deployment ordering is sequential. The first mover sets the environment. The Stackelberg leader has advantage.

Dominant strategy. Best regardless of what others do. Rare in real situations. When it exists, decision-making simplifies to triviality. When it doesn't, you need a model of the other player.

Pareto optimality. No player can be made better off without making another worse off. The Nash equilibrium of the Prisoner's Dilemma is not Pareto optimal. Many architectures are stuck in Pareto-suboptimal equilibria. The system is at a Nash equilibrium. The equilibrium is suboptimal. Moving requires coordination. Coordination is costly. The cost keeps the system where it is.

Stag hunt. Everyone benefits from cooperation but only if everyone cooperates. Hunting a stag requires all hunters. Hunting a rabbit can be done alone. The stag is worth more. If any hunter defects, the stag escapes and cooperators get nothing. API standardization is a stag hunt. If all teams adopt, everyone benefits. If some defect, the standard fragments. The stag was worth more. Nobody got it.

Chicken. Two players drive toward each other. First to swerve loses. If neither swerves, both crash. Two teams both rewriting the same service. Neither backs down. Both ship incompatible rewrites. The system breaks. Someone must swerve. Organizational hierarchy determines who. The hierarchy is a mechanism for resolving Chicken.

Battle of the sexes. Two players want to coordinate but prefer different coordinated outcomes. Both prefer coordination to miscoordination. Choosing a shared message queue: both prefer either queue to no queue. Both prefer their own choice. Two Nash equilibria. The selected equilibrium depends on who moves first or has more power. The selection is political. The politics are game-theoretic.

Mechanism design. Reverse game theory. Start with the desired outcome. Design the rules that produce it. If you want teams to keep API contracts stable, design a system where breaking a contract is immediately visible and costly. Automated contract testing is mechanism design. Deployment friction is mechanism design. Design the rules. Don't plead for the outcome.

Evolutionary game theory. Strategies that perform well survive. Strategies that perform poorly die. John Maynard Smith (1982): an Evolutionarily Stable Strategy cannot be invaded by a mutant. Microservices survived because they resist being re-monolithed. The monolith didn't survive because a small team could extract a service and demonstrate value. The extraction was the mutation. The mutation spread. The population shifted. The shift was evolutionary.

Signaling game. One player has private information and takes an action that may reveal it. Writing a detailed RFC signals competence. A costly signal separates sincere from insincere types. Requiring a prototype before architecture review is a screening mechanism. The cost of the signal is the price of credibility.

Repeated game. The same game played multiple times. Sprints are repeated games. Repeated games enable reputation and reciprocity. The shadow of the future disciplines present behavior. Cooperation can be sustained in repeated games even when it collapses in one-shot games. This is why teams that work together for years develop trust. The trust is game-theoretic. The game is repeated.

Robert Aumann, who shared the 2005 Nobel with Schelling, proved the Folk Theorem: any feasible, individually rational outcome can be sustained as an equilibrium when the game is repeated infinitely. In his own words: "In a single encounter, confrontation is the logical move; but when the interaction will occur repeatedly, cooperation is the logical behavior." The shadow of the future disciplines the present. Teams that will work together for years develop trust not because they are virtuous but because defection today costs cooperation tomorrow. The trust is game-theoretic. The game is repeated.


This is part 2 of a 7-part series on scarcity and software.

References:

  • John von Neumann and Oskar Morgenstern, Theory of Games and Economic Behavior, Princeton University Press, 1944.
  • John Nash, "Equilibrium Points in N-Person Games," Proceedings of the National Academy of Sciences, 1950.
  • John Maynard Smith, Evolution and the Theory of Games, Cambridge University Press, 1982.

Scarcity is the universal engineering constraint. Time, attention, compute, complexity — every engineering decision is made within a budget. The budget is economic. The engineer who doesn't track the budget makes decisions blind. The engineer who tracks it makes decisions with full knowledge of the trade-off. The trade-off is the decision. The budget is the constraint. Scarcity is the unifying principle.

The First Lesson

Robbins defined economics as choice under scarcity in 1932: ends, means, scarcity, alternative uses. Software engineering is economics by other means. Time, attention, complexity, and compute are the four scarcities that shape every architecture decision.

scarcityeconomicssoftware-engineeringrobbins

In 1932, Lionel Robbins published An Essay on the Nature and Significance of Economic Science. On page 15, he wrote:

"Economics is the science which studies human behaviour as a relationship between ends and scarce means which have alternative uses."

Four concepts. Ends — what you want. Means — what you have. Scarcity — means are finite. Alternative uses — means can be deployed in different ways, forcing choice. Every human activity involving these four conditions is economic. Robbins made this explicit:

"Insofar as it deals with the influence of scarcity, any kind of human behaviour falls within the scope of Economic Generalisations. There are no limitations on the subject-matter of Economic Science save this."

There are no limitations on the subject-matter. Software engineering is human behaviour under conditions of scarcity. Time is scarce. Attention is scarce. Compute is scarce. Complexity budget is scarce. You have finite means — developer hours, cognitive capacity, hardware, money — and infinite ends — features to build, bugs to fix, systems to improve, customers to satisfy. The means have alternative uses. Every hour spent on one feature is an hour not spent on another. Every dollar spent on infrastructure is a dollar not spent on hiring. Every ounce of cognitive capacity spent on one problem is an ounce not available for another. This is economics. The subject-matter is software. The structure is scarcity.

Thomas Sowell sharpened Robbins: "The first lesson of economics is scarcity: There is never enough of anything to satisfy all those who want it. The first lesson of politics is to disregard the first lesson of economics." Milton Friedman made it simpler: "There's no such thing as a free lunch." Every resource has an alternative use. Every choice has a cost. The cost is whatever you gave up.

Most people think the opposite of scarcity is abundance. It isn't. The opposite of scarcity is waste. Abundance without discipline produces waste. Scarcity with discipline produces focus. The difference between a great engineering team and a mediocre one is not how many resources they have. It is how they choose under scarcity. Great teams make the scarcity explicit. Mediocre teams pretend it doesn't exist. Both face the same constraints. One names them. The other pretends.

The four scarcities of software

Time is scarce. You have finite developer hours. Every feature you build is a feature you didn't build. Every refactor you do is a refactor you didn't do. Every meeting you attend is code you didn't write. The scarcity of time forces prioritization. Prioritization is an economic act. The framework for prioritization — what to build now, what to build later, what to never build — is capital budgeting applied to code. The budget is time. The investments are features. The returns are user value, revenue, reduced maintenance cost. The same mathematics that determines whether to build a factory determines whether to build a microservice. The logic is identical.

Attention is scarce. A developer can hold one complex problem in their head at a time. Two if exceptional. Three is impossible. The scarcity of attention is the binding constraint on software complexity. Brooks's Law — adding people to a late project makes it later — is a statement about attention scarcity. Each new person must learn the system. The teaching consumes the attention of those who already know it. Communication overhead grows quadratically. Attention per person shrinks. The project gets later. Brooks's argument for conceptual integrity — one mind controlling the design — is also about attention scarcity. The design must fit in one mind because only one mind can hold it. When the design exceeds one mind's capacity, it must be split. The split requires coordination. Coordination consumes attention. Attention consumed by coordination is not available for design. Design quality degrades.

Complexity budget is scarce. Lehman's Second Law: complexity increases unless work is done to reduce it. The work requires time and attention, both scarce. The complexity budget is the total complexity the system can absorb before becoming unmaintainable. Every feature adds to the budget. Every quick fix adds. Every workaround adds. The budget is finite. When exhausted, the system must be rewritten. The rewrite is expensive. The expense is the cost of exceeding the budget. The budget was always finite. The accounting ignored it.

Parnas's information hiding is an economic strategy. Hide volatile decisions behind stable interfaces. The interface is the investment. The hiding is the return. When the volatile decision changes, the change is contained within the module. The containment saves time, attention, and complexity budget. The investment costs more upfront. The return accrues over time — every change that doesn't propagate is a cost avoided. The net present value is positive if the decision is sufficiently volatile. Parnas didn't state it in economic terms. The economics are implicit. The economics are correct.

Compute is scarce. Moore's Law made compute cheap but not free. Cloud computing made it elastic but not infinite. Every algorithm choice trades compute for something else — development time, code simplicity, latency. The trade is economic. The price of compute determines which side is optimal. When compute was expensive, developers optimized for cycles. When compute became cheap, they optimized for developer time. The price changed. The optimal trade changed. The change was economic. The economics were invisible to the developers. They thought they were making technical decisions. They were making economic decisions with technical parameters.

The Robbins test

Robbins gives a test for any decision. Ask four questions:

  1. What is the end? (What are you trying to achieve?)
  2. What are the means? (What resources do you have?)
  3. Are the means scarce? (Could you use them for something else?)
  4. Do they have alternative uses? (What are you giving up?)

If the answer to 3 and 4 is yes — and it always is — the decision is economic. Treat it as such. Name the scarce resource. Name the alternative foregone. Calculate the trade. Make the choice. The choice will be better for having been made explicitly.

Most software decisions are made without naming the scarcity. Two engineers debate two architectures. Neither names the constraint. The debate is unresolvable because the constraint is unstated. State the constraint. The debate resolves. This feature or that feature? State the scarce resource. This architecture or that architecture? State the trade. The Robbins test makes implicit economics explicit. Explicit economics are debatable. Implicit economics are invisible. Invisible economics produce worse decisions.

The unifying principle

Resources are finite. Ends are infinite. Means have alternative uses. Every decision is a choice under scarcity. The choice has consequences. The consequences propagate. The propagation is the system's behavior. The behavior is emergent from the choices. The choices are economic. Software engineering is economics.


This is part 1 of a 7-part series on scarcity and software.

References:

  • Lionel Robbins, An Essay on the Nature and Significance of Economic Science, Macmillan, 1932.
  • Frederick P. Brooks, Jr., The Mythical Man-Month, Addison-Wesley, 1975.
  • David L. Parnas, "On the Criteria to Be Used in Decomposing Systems into Modules," Communications of the ACM, 1972.
  • M.M. Lehman, "Programs, Life Cycles, and Laws of Software Evolution," Proceedings of the IEEE, 1980.

Scarcity is the universal engineering constraint. Time, attention, compute, complexity — every engineering decision is made within a budget. The budget is economic. The engineer who doesn't track the budget makes decisions blind. The engineer who tracks it makes decisions with full knowledge of the trade-off. The trade-off is the decision. The budget is the constraint. Scarcity is the unifying principle.

Governance Tokens and DAOs

A governance token gives its holder the right to vote on a protocol's decisions. The functional origin is the joint-stock company — invented in 1602 by the Dutch East India Company. Shareholders owned the company. Shareholders voted. The DAO is the joint-stock company, automated. The governance token is the share. The vote is the product.

defidaogovernancetokensjoint-stock

The governance token is the most abstract financial instrument in DeFi. It confers no claim on cash flows. It confers no ownership of assets. It confers the right to vote on protocol parameters — fee levels, collateral types, treasury allocations. The token is a share in the protocol's governance, not in its profits. The distinction is the subject of regulatory uncertainty, legal innovation, and philosophical debate about the nature of ownership in decentralized systems.

The first governance token was COMP, launched by Compound in May 2020. COMP holders vote on Compound's interest rate models, collateral factors, and treasury expenditures. The token was airdropped to users of the protocol — lenders and borrowers. The airdrop was the reward for early adoption. The reward created a constituency of token holders with a stake in the protocol's success. The constituency was the governance community. The community was the innovation.

The functional origin: the joint-stock company

The joint-stock company was invented by the Dutch East India Company (VOC) in 1602. Before the VOC, business enterprises were partnerships — small groups of investors who pooled capital for a single voyage. The partnership dissolved when the voyage ended. The VOC introduced permanent capital: investors bought shares that could be held indefinitely and traded on the Amsterdam Stock Exchange. The shares conferred ownership of the company's assets and a claim on its profits — dividends. The shares also conferred voting rights — the right to elect the company's directors. The voting right was proportional to share ownership. One share, one vote.

The joint-stock company was the institutional innovation that enabled European colonialism. It pooled capital from thousands of investors. It operated at a scale that no partnership could match. It outlasted individual voyages, individual governors, individual lifetimes. The company was immortal. The shares were the mechanism of immortality.

The governance token is the joint-stock share, stripped of the dividend right and amplified in the voting right. The token holder votes on protocol parameters. The vote is on-chain, transparent, and binding — the winning proposal is executed automatically by the protocol's smart contracts. The automation is the innovation. The innovation eliminates the management layer. The shareholders are the managers. The code executes their decisions.

The DAO

A Decentralized Autonomous Organization (DAO) is a governance token plus a treasury plus a decision-making process. The token holders propose and vote on how to spend the treasury. The treasury is funded by protocol revenue — trading fees, lending interest, liquidation penalties. The revenue flows to the treasury, not to token holders directly. The token holders decide how to allocate it: development grants, liquidity incentives, bug bounties, token buybacks.

The DAO is the organizational form of DeFi protocols. Uniswap is governed by a DAO. Compound is governed by a DAO. MakerDAO is the original DAO, governing DAI since 2017. The DAO form is evolving. Early DAOs were plutocratic — one token, one vote. Wealth determined control. The concentration of token ownership produced concentration of governance power. The concentration is the subject of experimentation: quadratic voting, delegation, conviction voting, reputation-weighted voting. The experiments are attempts to solve the plutocracy problem. The problem is inherent in token-weighted voting. The solutions are partial.

The regulatory question

Governance tokens exist in a regulatory gray zone. They confer voting rights without dividend rights. The SEC has indicated that tokens with governance rights may still be securities if they are sold to raise capital for a common enterprise with the expectation of profit derived from the efforts of others — the Howey test. The profit expectation is created by token buybacks, fee switches that redirect protocol revenue to token holders, and the appreciation of the token's market price driven by protocol growth. The appreciation is the profit. The profit is the security.

The DAO's legal status is also uncertain. A DAO is not a corporation. It has no legal personality. It cannot sign contracts, sue, or be sued. The lack of legal personality protects token holders from personal liability but also limits the DAO's ability to interact with the traditional legal system. Several jurisdictions — Wyoming, Vermont, the Marshall Islands — have created DAO-specific legal entities. The entities provide limited liability to token holders while preserving on-chain governance. The entities are the bridge between DAO governance and legal recognition.

The reference

Adam Smith, The Wealth of Nations (1776). Smith's analysis of the joint-stock company identified the agency problem that DAOs attempt to solve: the separation of ownership and control. In a joint-stock company, shareholders own the company but directors control it. The directors may pursue their own interests rather than the shareholders'. The agency problem is the cost of professional management. DAOs eliminate the management layer. The shareholders are the directors. The elimination of the agency problem is the theoretical advantage of DAOs. The practical disadvantage is the plutocracy problem and the inefficiency of collective decision-making. The trade-off is the subject of DAO governance research. The research is ongoing.

The engineering connection

A DAO is a distributed system where the state is the treasury, the transactions are governance proposals, and the consensus mechanism is token-weighted voting. The architecture is identical to a blockchain: proposals are transactions, votes are signatures, execution is a state transition. The DAO is a blockchain with a single application — treasury management — built on top of an existing blockchain. The recursion is elegant: a governance system implemented as smart contracts on a chain that is itself governed by a similar mechanism.

The DAO's core engineering problem is the same as any access control system: who can do what, under what conditions, with what checks and balances. Token-weighted voting is one access control policy — more tokens, more power. Timelocks on execution are another — proposals pass, but execution is delayed, giving stakeholders time to exit if they disagree. Multi-signature execution is a third — multiple parties must approve. These are the same patterns as role-based access control (RBAC), change management windows, and approval workflows in enterprise software. The domain is governance. The patterns are authorization.


References:

Structured Products

A structured product packages multiple financial instruments into a single product with a predefined payoff structure. Principal-protected notes, yield enhancement, range-bound strategies. The functional origin is the structured note — invented in the 1980s to give retail investors access to derivatives strategies. DeFi structured products automate these strategies on-chain.

defistructured-productsoptionsyieldribeye

A structured product is a packaged investment strategy. It combines multiple financial instruments — typically a bond and one or more derivatives — to produce a predefined payoff profile. The investor deposits capital. The product returns the capital plus a return that depends on the performance of an underlying asset, subject to conditions. If the conditions are met, the investor earns an enhanced yield. If they are not, the investor may earn nothing, lose some principal, or receive a minimum guaranteed return.

The simplest example is a principal-protected note: the investor deposits $100. The product invests $95 in a zero-coupon bond that matures at $100 in one year. The remaining $5 buys a call option on the S&P 500. If the S&P rises, the investor participates in the upside. If it falls, the investor gets their $100 back at maturity. The principal is protected. The upside is capped by the cost of the option. The structure is a bond plus a call option. The structure is a structured product.

The functional origin: structured notes

Structured notes were developed by investment banks in the 1980s and 1990s to provide retail investors with access to derivatives strategies without requiring them to understand or trade derivatives directly. The bank packaged the derivatives into a note — a debt instrument — and sold it through brokers. The investor bought the note. The bank managed the underlying strategy. The bank earned fees. The investor earned a return linked to the strategy's performance.

The market grew rapidly. By the 2000s, structured notes were a multi-trillion-dollar market. The products ranged from simple (principal-protected equity notes) to incomprehensible (tranched CDOs-squared). The complexity was the vulnerability. When the underlying assets — subprime mortgages — defaulted in 2008, the structured products collapsed. The collapse was the financial crisis. The crisis was the lesson: structured products are only as safe as their underlying assets and the transparency of their structure.

DeFi structured products

DeFi structured products automate the same strategies on-chain with full transparency. The types:

Covered call vaults. Ribbon Finance and Thetanuts sell out-of-the-money call options against deposited assets. The premiums generate yield. If the underlying asset stays below the strike price, the options expire worthless and the depositor keeps the premium plus the asset. If the asset rises above the strike, the depositor's asset is called away — they receive the strike price instead of the asset. The upside is capped. The premium is the compensation for the capped upside. The vault automates the option selling and compounding. The depositor earns yield without managing strikes or expiries.

Principal-protected products. Cega and Friktion offer principal-protected notes on crypto assets. The investor deposits stablecoins. The vault invests most in lending protocols (the bond) and uses the remainder to buy options (the upside). If the options pay off, the investor earns enhanced yield. If they don't, the investor gets their principal back. The structure is transparent — all positions are on-chain. The transparency is the improvement over traditional structured notes, where the underlying positions were opaque.

Range-bound strategies. The investor bets that an asset will stay within a price range. If it does, they earn yield. If it breaks out, they may lose principal or have their asset converted. The strategy is implemented by selling a strangle — a call and a put at different strikes. The premium is the yield. The conversion is the risk.

Basis trade vaults. The vault executes the cash-and-carry trade — buying spot and shorting perpetual futures — to earn the funding rate. The trade is market-neutral. The yield is the funding rate. The vault automates position management, rebalancing, and compounding.

The reference

Peter Bernstein, Against the Gods: The Remarkable Story of Risk (1996). Bernstein's book is a history of risk management from the Renaissance to modern finance. The central argument: the quantification of risk — probability theory, statistics, derivatives pricing — is the defining intellectual achievement of modern capitalism. Structured products are the application of that achievement to retail investment products. DeFi structured products are the next iteration. The math is the same. The execution is on-chain. The transparency is the improvement.

The engineering connection

A structured product is a function composition. A principal-protected note is bond() + call_option(). A covered call vault is hold_asset() + sell_call(). The product's payoff is the sum of its components' payoffs. The engineering insight: complex financial products can be built by composing simple primitives, the same way complex software systems are built by composing simple functions. The composition is the architecture. The primitives are the modules. The structured product is the application.

The DeFi structured product automates the composition. The vault rebalances, compounds, and rolls positions without human intervention. This is the same automation pattern as a Kubernetes operator: observe state, compare to desired state, execute reconciliation. The vault observes the option's time to expiry, compares to the roll threshold, executes the roll. The control loop is identical. The domain is different. The pattern is the same.


References:

Insurance Protocols

Insurance began at Lloyd's coffee house in 1688, where merchants pooled risk to protect their ships. DeFi insurance protocols — Nexus Mutual, Unslashed, InsurAce — automate risk pooling on-chain. Members contribute capital to a mutual. Claims are assessed by members. The model is the same as Lloyd's. The execution is code.

defiinsurancenexus-mutuallloydsrisk

Insurance is the oldest risk management tool. A group of people faces a common risk — shipwreck, fire, death. Each contributes to a pool. When one suffers a loss, the pool compensates them. The contributions are premiums. The payouts are claims. The pool is the mutual. The mutual is the insurance company.

The first modern insurance market was Lloyd's of London, which began at Edward Lloyd's coffee house on Tower Street in 1688. Merchants, shipowners, and underwriters gathered at Lloyd's to share news of shipping, negotiate insurance contracts, and pool risk. A shipowner seeking insurance would pass a slip around the coffee house. Underwriters would sign their names under the risk they were willing to accept, each taking a fraction. The slip was the contract. The signature was the commitment. The coffee house was the exchange.

The Lloyd's model persists today. Lloyd's is not an insurance company. It is a market where syndicates of underwriters compete to accept risk. The syndicates are backed by capital providers — "Names" — who put up their personal wealth as collateral. The Names earn premiums in good years and lose their fortunes in bad ones. The unlimited liability of Names was the mechanism that aligned incentives: the underwriter had skin in the game.

DeFi insurance

DeFi insurance protocols apply the mutual model to crypto risks. The risks: smart contract bugs, oracle manipulation, stablecoin depegs, exchange hacks. The protocols: Nexus Mutual, InsurAce, Unslashed. The mechanism: members deposit capital into a mutual. Members purchase coverage against specific risks. When a claim is filed, members vote on whether to pay it. The vote is the claims assessment. The assessment is decentralized.

Nexus Mutual, launched in 2019, is the largest DeFi insurance protocol. It has paid claims for the UST depeg, the FTX collapse, and multiple protocol hacks. The claims process is governed by NXM token holders, who stake their tokens on the outcome of claims assessments. If a claim is valid and the assessors correctly vote to pay, they earn rewards. If a claim is fraudulent and assessors vote to deny, they also earn rewards. If assessors vote incorrectly — paying a fraudulent claim or denying a valid one — they lose their stake. The staking mechanism aligns incentives. The alignment is the mechanism design.

The challenge: claims assessment requires expertise. A smart contract bug may have been exploited by an attacker. Was the loss caused by a bug in the covered protocol or by user error? The distinction requires technical analysis. The analysis is provided by claims assessors. The assessors are compensated for their work. The compensation attracts expertise. The expertise is the product.

The risk

Mutual insurance is capital-constrained. The capital in the mutual must cover all potential claims. If claims exceed capital, the mutual is insolvent. The insolvency risk is managed by risk-based pricing: higher-risk protocols pay higher premiums. The premium is the price of risk. The price is set by the market — by the willingness of capital providers to accept the risk at a given premium. The market for risk is the innovation. The innovation is the same as Lloyd's in 1688. The venue is different. The principle is the same.

The reference

Lloyd's of London, A History of Lloyd's (various editions). The history of Lloyd's is the history of insurance. The coffee house. The slip. The Name. The syndicate. The innovation of Lloyd's was the market for risk — bringing together those who had risk and those who would accept it for a price. DeFi insurance protocols are recreating that market on-chain. The market is the same. The technology is different. The principles — pooling, diversification, skin in the game — are unchanged.

The engineering connection

DeFi insurance is a distributed consensus protocol applied to claims assessment. A claim is filed. Assessors vote. The vote must reach consensus. If the assessors are honest, the correct outcome is reached. If assessors collude, the outcome can be manipulated. The mechanism — stake-weighted voting with economic penalties for incorrect votes — is the same mechanism as proof-of-stake consensus. The validators (assessors) stake tokens on their assessment. If the majority is honest, the honest assessors earn rewards and the dishonest lose their stake. The protocol is a blockchain consensus algorithm applied to a different type of state transition — a claim rather than a block.

The capital constraint — the mutual must hold enough capital to cover all potential claims — is the same constraint as any capacity-planning problem. You have a pool of resources (capital) and a stream of demands (claims). You must size the pool so that the probability of exhaustion is below a threshold. The math is queuing theory. The same math that determines how many servers you need for a web application determines how much capital an insurance mutual needs. The domain is insurance. The math is capacity planning.


References:

  • Lloyd's of London, historical archives.
  • Nexus Mutual, "Nexus Mutual Documentation."
  • Hugh Eaves, "The History of Insurance," Lloyd's Library.
  • Related posts: Design the Game, On Scarcity

Tokenized Real-World Assets

Tokenized real-world assets — Treasuries, real estate, private credit — bring off-chain assets on-chain. BlackRock's BUIDL fund tokenized $500M in Treasuries in 2024. The functional origin is securitization: pooling assets and issuing claims against the pool. The innovation is doing it on-chain, with programmatic settlement and fractional ownership.

defirwatokenizationblackrocktreasuries

Tokenized real-world assets (RWAs) are claims on off-chain assets, issued as on-chain tokens. A Treasury bond held by a custodian bank. A token representing fractional ownership of the bond, issued on Ethereum. The token trades on-chain. The bond's interest payments flow to token holders. The token is the bridge between traditional financial assets and DeFi infrastructure.

In 2024, RWAs became the fastest-growing sector in DeFi. BlackRock launched BUIDL, a tokenized Treasury fund, in March 2024. By year-end, it held over $500 million in assets. Franklin Templeton's OnChain U.S. Government Money Fund reached $400 million. Ondo Finance, a DeFi-native RWA protocol, tokenized over $200 million in Treasuries. The total market for tokenized Treasuries exceeded $2 billion. The growth was driven by demand for yield-bearing assets in a high-interest-rate environment. U.S. Treasuries yield 4-5%. DeFi stablecoin lending yields 2-3%. The arbitrage is obvious. The tokenization enables the arbitrage.

The functional origin: securitization

Securitization is the process of pooling financial assets and issuing securities backed by the pool. The most famous example is the mortgage-backed security (MBS), developed by Ginnie Mae in 1970. A bank originates mortgages. It sells them to a trust. The trust issues securities — bonds — backed by the mortgage payments. The security holders receive the interest and principal from the mortgage pool. The bank removes the mortgages from its balance sheet. The risk is transferred to the security holders.

Securitization transformed finance. It enabled banks to originate loans without holding them to maturity. It created liquid markets for illiquid assets. It spread risk across the financial system. It also enabled the 2008 financial crisis: when the underlying mortgages defaulted, the securities collapsed. The model worked until it didn't.

Tokenization is securitization on a blockchain. The token is the security. The smart contract is the trust. The blockchain is the settlement layer. The difference: tokenization enables programmatic issuance, fractional ownership, and composable integration with DeFi protocols. A tokenized Treasury can be used as collateral on Aave. It can be traded on Uniswap. It can be wrapped into a yield-bearing stablecoin. The composability is the innovation. The innovation multiplies the utility of the underlying asset.

The infrastructure

Tokenized RWAs require off-chain infrastructure. A custodian holds the underlying asset. An issuer creates the token. An oracle reports the net asset value. A transfer agent manages the token holder registry. A legal framework governs the relationship between token holders and the underlying asset. The infrastructure is the same as traditional securitization, with the blockchain replacing the central securities depository.

The regulatory status of tokenized RWAs is uncertain. The SEC has indicated that many tokenized assets are securities. The classification triggers registration, disclosure, and compliance obligations. The obligations are the cost of legitimacy. The cost is paid by the protocols that pursue institutional adoption. The protocols that don't pursue institutional adoption operate in regulatory gray zones. The zones are shrinking. The shrinkage is the trend. The trend is toward regulated, institutionally-compliant tokenization. The early DeFi ethos of permissionless, anonymous access to financial instruments is colliding with the reality of securities law. The collision is the defining regulatory question of the RWA sector.

The reference

BlackRock, "BUIDL: BlackRock USD Institutional Digital Liquidity Fund," 2024. The launch of BUIDL by the world's largest asset manager was the signal that tokenized RWAs had arrived as an institutional product. The fund holds U.S. Treasury bills, repurchase agreements, and cash. It issues BUIDL tokens on Ethereum. It pays daily dividends to token holders. It is regulated. It is audited. It is the bridge between TradFi and DeFi. The bridge is built. The traffic is increasing.

The engineering connection

Tokenization is an adapter pattern. The traditional financial asset (Treasury bond, real estate deed) speaks one protocol — legal ownership, paper settlement, business hours. The DeFi ecosystem speaks another — programmatic transfer, atomic settlement, 24/7. The tokenization protocol is the adapter between them. The adapter translates: legal ownership → on-chain token, interest payment → token distribution, redemption → token burn. The same pattern as any API adapter, database connector, or protocol bridge. The adapter's job is to make two incompatible systems interoperate. The job is pure software engineering.

The oracle that reports the net asset value is the adapter's most critical component. If the oracle reports incorrectly, the token's value diverges from the underlying. The divergence creates arbitrage or insolvency. The oracle is a trust boundary — the point where the on-chain system must trust an off-chain data source. Engineering trust boundaries is the core problem of distributed systems. The solutions — multiple data sources, stake-based validation, challenge periods — are the same whether the oracle reports a stock price or a server health check. The domain changes. The trust architecture doesn't.


References:

  • BlackRock, "BUIDL Fund," 2024.
  • Ondo Finance, "Ondo: Institutional-Grade Onchain Finance," Ondo Documentation.
  • Related posts: Stablecoins, Synthetic Assets

Synthetic Assets

A synthetic asset is a token that tracks the price of something without requiring custody of that thing. Synthetix lets you trade synthetic gold, synthetic Apple stock, synthetic Bitcoin — all on-chain, collateralized by crypto. The functional origin is the contract for difference (CFD), invented in London in the 1950s. The execution is on Ethereum.

defisyntheticssynthetixcfdsderivatives

A synthetic asset is a financial instrument that simulates the payoff of another asset without requiring ownership of that asset. A synthetic S&P 500 token tracks the price of the S&P 500 index without the holder owning any of the underlying stocks. The synthetic is created by collateralization: a user deposits crypto as collateral and mints the synthetic token. The token's value is maintained by an oracle that reports the price of the underlying asset. If the synthetic token's price diverges from the underlying, arbitrageurs trade the divergence back to parity.

Synthetix, launched in 2018, is the dominant synthetic asset protocol. Users deposit SNX (the Synthetix governance token) as collateral and mint synthetic assets — synths — that track currencies, commodities, cryptocurrencies, and equities. The synths trade on Synthetix's own exchange. The exchange uses a pooled collateral model: all synths are backed by the collective SNX collateral, not by individual positions. The pooling enables infinite liquidity — any synth can be traded against any other synth at the oracle price, with zero slippage. The zero slippage is the key innovation. The key innovation is funded by SNX stakers, who bear the risk of the pooled collateral.

The functional origin: contracts for difference

The contract for difference (CFD) was invented in London in the 1950s by hedge funds seeking to trade equities on margin without triggering stamp duty — a tax on share transactions. The CFD is an agreement between two parties to exchange the difference between the opening and closing price of an underlying asset. The buyer doesn't own the asset. The seller doesn't deliver it. They settle the price difference in cash.

CFDs became popular among retail traders in the 1990s, offered by spread-betting firms like IG Index. The appeal: leveraged exposure to any asset class without owning the underlying, without paying stamp duty, and without the complexity of futures or options. The risk: the leverage amplifies losses. The risk materialized in 2015 when the Swiss National Bank unpegged the franc from the euro. EUR/CHF collapsed. Retail CFD traders were wiped out. Several brokers went bankrupt.

Synthetic assets are CFDs on a blockchain. The collateral is on-chain. The settlement is automatic. The counterparty is the protocol, not a broker. The protocol's solvency depends on the collateralization ratio. If the ratio falls below the threshold, the protocol must liquidate positions or mint new tokens. The mechanism is the same as MakerDAO's DAI, generalized to any asset.

The pooled collateral model

Synthetix's pooled collateral model is the key architectural difference from MakerDAO. In MakerDAO, each vault is independent — if one vault is undercollateralized, only that vault is liquidated. In Synthetix, all synths are backed by a single pool of SNX collateral. The pool absorbs gains and losses collectively. If the value of outstanding synths exceeds the value of the collateral pool, the protocol is insolvent. The insolvency risk is collective. The collective risk requires a higher collateralization ratio — Synthetix targets 400-500%, compared to MakerDAO's 150%.

The pooled model enables infinite liquidity. Because all synths are fungible against the collateral pool, any synth can be traded for any other synth at the oracle price. There is no order book. There is no AMM curve. There is no slippage. The trade is executed against the pool at the oracle price plus a fee. The fee goes to SNX stakers. The model is elegant. The elegance is the attraction. The attraction is counterbalanced by the complexity of managing a pooled collateral system with exposure to dozens of synthetic assets.

The frontier

Synthetic assets enable on-chain exposure to off-chain assets. The oracle reports the price of Apple stock. The protocol mints sAAPL. The user trades sAAPL on-chain. The exposure is synthetic. The custody is unnecessary. The border between traditional finance and DeFi is the oracle. The oracle is the bridge. The bridge enables a future where any asset can be traded on-chain without the asset ever touching a blockchain. The future is synthetic. The synthesis is the product.

The reference

Synthetix, "Synthetix Litepaper" (2018). The original description of the pooled collateral model and the infinite liquidity exchange. The litepaper is 12 pages. The key insight — pooled collateral enables zero-slippage trading between any synthetic assets — is on page 3. The rest is implementation. The implementation is now managing billions in synthetic asset value. The value is the proof of the concept.

The engineering connection

A synthetic asset is a data pipeline. The oracle is the data source. The protocol is the transformation layer. The synthetic token is the output. The pipeline ingests a price feed, transforms it into a token with the same economic exposure, and delivers it to the user. The same pattern as an ETL pipeline: extract (oracle price), transform (mint synth at oracle price), load (deliver token to user). The domain is finance. The architecture is data engineering.

The pooled collateral model is a shared-nothing architecture applied to risk. Each synth is backed by the entire collateral pool, not by individual positions. The risk is shared. The sharing eliminates counterparty risk between synth holders but creates systemic risk — if the collateral value crashes, all synths are affected simultaneously. This is the same trade-off as a monolith vs. microservices. The monolith (pooled collateral) is efficient and simple until it fails globally. Microservices (isolated vaults, like MakerDAO) are resilient to individual failures but less capital-efficient. The trade-off is architectural. The domain determines which side is correct.


References:

  • Synthetix, "Synthetix Litepaper," 2018.
  • IG Group, "Contracts for Difference," IG Documentation.
  • Related posts: Stablecoins, Perpetual Futures

Liquid Staking

Liquid staking solves the capital efficiency problem of proof-of-stake: you stake ETH to secure the network, and you get a liquid token (stETH) that you can use in DeFi while your ETH is locked. The functional origin is the depository receipt — a claim on a deposited asset that trades freely. Lido now holds over 30% of all staked ETH.

defiliquid-stakinglidostethproof-of-stake

Proof-of-stake requires validators to lock capital. On Ethereum, a validator must stake 32 ETH. The staked ETH is locked — it cannot be transferred, traded, or used as collateral. The lockup secures the network: validators who misbehave lose their stake. The lockup also creates an opportunity cost. Staked ETH cannot earn yield in DeFi. The opportunity cost is the return the staker could earn by deploying the ETH elsewhere.

Liquid staking eliminates the opportunity cost. A liquid staking protocol accepts ETH deposits, stakes them with validators, and issues a liquid token — stETH for Lido, rETH for Rocket Pool, cbETH for Coinbase — representing the deposited ETH plus accrued staking rewards. The liquid token can be traded, lent, or used as collateral in DeFi. The staker earns staking rewards plus whatever DeFi yield they generate with the liquid token. The capital does double duty. The efficiency is the innovation.

The functional origin: depository receipts

A depository receipt is a financial instrument that represents ownership of an underlying asset held in custody. The most famous example is the American Depositary Receipt (ADR), introduced by J.P. Morgan in 1927. A U.S. bank holds shares of a foreign company in custody. It issues ADRs that trade on U.S. exchanges. The ADR holder receives dividends, votes, and price exposure without directly owning the foreign shares. The ADR solves the problem of cross-border investment: the foreign shares never leave their home market, but the ADR trades freely in the U.S.

The first depository receipt predates the ADR by centuries. The Dutch East India Company, founded in 1602, issued negotiable share certificates that could be transferred without altering the company's share register. The certificates were depository receipts in function if not in name. The principle — a claim on a deposited asset that trades independently — is the same.

Liquid staking tokens are depository receipts for staked ETH. The ETH is deposited with validators. The stETH is issued to the depositor. The stETH accrues staking rewards through a rebasing mechanism — the balance of stETH in the holder's wallet increases daily to reflect staking rewards. The stETH trades on AMMs, is accepted as collateral on lending protocols, and can be deployed in yield farming strategies. The deposited ETH secures the network. The stETH circulates in DeFi. The separation of security provision from capital deployment is the innovation.

The risk

Liquid staking concentrates stake. Lido holds over 30% of all staked ETH. The concentration is a systemic risk to Ethereum: if Lido's validators collude or are compromised, the network's security is threatened. Lido distributes stake across multiple independent node operators to mitigate this risk. The distribution is a governance mechanism. The mechanism is imperfect. The imperfection is the subject of ongoing protocol development — distributed validator technology (DVT), stake capping, and validator set rotation.

The secondary risk: stETH can depeg from ETH. During market stress, stETH holders may want to exit to ETH directly rather than wait for the unstaking period. The selling pressure pushes stETH below its 1:1 peg. The discount creates an arbitrage opportunity — buy stETH at a discount, redeem for ETH after the unstaking period, profit. The arbitrage requires capital and patience. The capital must be locked during the unstaking period. The patience is tested during extended drawdowns. The discount is the market's price for immediate liquidity. The price fluctuates with market stress.

The reference

J.P. Morgan, "American Depositary Receipts," 1927. Morgan's innovation was financial infrastructure: the ADR created a mechanism for U.S. investors to hold foreign equities without navigating foreign custody, settlement, and currency conversion. The ADR was a bridge between national financial systems. Liquid staking tokens are a bridge between the staking layer and the DeFi layer. The bridge is the infrastructure. The infrastructure enables capital to flow between layers. The flow is the efficiency. The efficiency is the value.

The engineering connection

Liquid staking is a delegation pattern. The staker delegates capital to a validator. The validator does work (proposes blocks). The staker receives a receipt token (stETH) representing the delegated capital plus accrued rewards. The receipt token is a proxy object — the same pattern as a remote proxy in distributed systems, a lazy-loading proxy in ORM design, or a future/promise in async programming. The holder interacts with the proxy as if it were the underlying. The proxy handles the indirection. The pattern is universal. The domain is staking. The pattern is delegation.

The depeg risk — stETH trading below ETH during market stress — is the same class of failure as any system where a proxy gets out of sync with its underlying. The proxy's price should track the underlying. When it doesn't, arbitrage should restore the peg. The arbitrage requires time and capital. The time is the unstaking period. The capital must be locked during that period. The mechanism design challenge: how to keep the proxy in sync when the sync mechanism has inherent latency. The same challenge appears in cache coherence, database replication, and distributed consensus. The solutions are different. The problem structure is identical.


References:

  • Lido, "Lido: Ethereum Liquid Staking," Lido Documentation.
  • Rocket Pool, "Rocket Pool: Decentralised Ethereum Liquid Staking," Rocket Pool Documentation.
  • Ethereum Foundation, "Proof of Stake," Ethereum Documentation.
  • Related posts: Stablecoins, AMMs

Yield Farming

Yield farming is the practice of deploying capital across DeFi protocols to earn returns. The functional origin is sharecropping — providing land (capital) in exchange for a share of the harvest (yield). The innovation is liquidity mining — protocols paying users in governance tokens to bootstrap liquidity. The economics are incentive design. The risks are impermanent loss, smart contract bugs, and the inevitable decline of unsustainable yields.

defiyield-farmingliquidity-miningincentivescompound

Yield farming is the practice of deploying capital across DeFi protocols to earn the highest available return. The farmer deposits tokens into lending pools, AMM liquidity pools, or options vaults. The protocols pay interest, trading fees, or incentive rewards. The farmer monitors returns and reallocates capital as yields change. The activity is called farming because the capital is the seed and the yield is the harvest.

Yield farming emerged in the summer of 2020 — "DeFi Summer" — when Compound introduced liquidity mining. Compound distributed COMP governance tokens to users who borrowed and lent on the protocol. The distribution turned lending from a low-yield activity into a high-yield one. Users borrowed assets they didn't need, lent them back, and earned COMP on both sides. The activity was economically circular — it didn't increase lending efficiency, it harvested tokens. The harvest was profitable while COMP traded at high valuations. The valuations were sustained by the expectation of future protocol revenue. The expectation was speculative. The speculation was the yield.

The functional origin: sharecropping

Sharecropping is an agricultural arrangement where a landowner provides land to a farmer in exchange for a share of the crop. The arrangement emerged after the American Civil War, when former slaves had labor but no land, and former plantation owners had land but no labor. The sharecropper worked the land. The landowner provided the capital — land, tools, seed. The harvest was split. The split was the return on capital.

The arrangement was exploitative in practice — landowners often manipulated accounts to keep sharecroppers in debt — but the economic logic was sound: capital and labor combine to produce output. The capital provider earns a return proportional to the capital's contribution. The labor provider earns a return proportional to the labor's contribution. The split is the subject of bargaining.

Yield farming is sharecropping on blockchain rails. The farmer provides capital — tokens deposited into a protocol. The protocol provides the "land" — the smart contract infrastructure that generates fees. The harvest is the fees plus the incentive tokens. The split is determined by the protocol's parameters, not by a landowner's accounting. The transparency eliminates the exploitation vector. The automation eliminates the bargaining. The code is the landlord. The code is neutral.

Liquidity mining

Liquidity mining is the distribution of protocol governance tokens to users who provide liquidity. The mechanism: the protocol allocates a percentage of its token supply to liquidity providers. The tokens are distributed pro-rata based on each provider's share of the pool. The tokens have value if the protocol has value. The value of the tokens is the incentive to provide liquidity.

Liquidity mining is an incentive design problem. The protocol wants to attract liquidity — deeper pools mean lower slippage, which attracts traders, which generates fees, which attracts more liquidity. The incentive tokens are the subsidy. The subsidy is paid to early liquidity providers to overcome the chicken-and-egg problem: no one wants to provide liquidity to an empty pool, and no one wants to trade in a pool with no liquidity. The subsidy breaks the equilibrium. The subsidy costs the protocol dilution of its token. The dilution is the cost of bootstrapping.

The problem: liquidity mining attracts mercenary capital. When yields are high, capital floods in. When yields decline — because the token price falls, or the subsidy ends, or a competing protocol offers higher yields — capital floods out. The mercenary capital is not loyal. The loyalty must be earned by the protocol's fundamentals: fee generation, user growth, sustainable economics. The protocols that achieve fundamentals survive the end of the subsidy. The protocols that don't die when the subsidies stop. The death is the market's verdict on the protocol's underlying value.

The strategies

Simple lending. Deposit stablecoins into Aave or Compound. Earn the lending interest rate. The rate fluctuates with utilization. The yield is modest — typically 2-10% APY for stablecoins. The risk is low — the protocols are battle-tested, the collateral is overcollateralized. Simple lending is the baseline. The baseline is the risk-free rate of DeFi.

LP farming. Provide liquidity to an AMM pool. Earn trading fees plus incentive tokens. The yield is higher than lending. The risk is higher — impermanent loss, smart contract risk, incentive token price risk. LP farming is the most common yield farming strategy. The commonness is evidence of the risk-return trade-off. The trade-off is the farmer's decision.

Leveraged farming (yield looping). Deposit collateral. Borrow against it. Deposit the borrowed funds. Repeat. The leverage multiplies the yield and the risk. If the lending rate exceeds the farming yield, the position loses money. If the collateral value falls, the position is liquidated. Leveraged farming is the riskiest yield strategy. The riskiest strategy attracts the most sophisticated farmers. The sophistication is the barrier to entry. The barrier protects the yields of those who cross it.

Auto-compounding. Protocols like Yearn and Beefy automatically compound yields — harvesting rewards, selling them for the underlying asset, redepositing. The compounding increases APY. The automation saves gas. The protocol takes a fee. The fee is the price of convenience. The convenience is the product.

The reference

Vitalik Buterin, "On Liquidity Mining" (2020). Buterin argued that liquidity mining is only sustainable if the distributed tokens confer genuine governance rights over a protocol that generates genuine revenue. Otherwise, liquidity mining is a wealth transfer from late buyers of the token to early farmers. The wealth transfer is zero-sum. The zero-sum game ends when the music stops. The music stopped for many protocols in 2022. The protocols that survived had fundamentals. The fundamentals were revenue. The revenue was from fees. The fees were from users. The users were real.

The engineering connection

Liquidity mining is an incentive design problem — the same class of problem as designing a recommendation algorithm or a caching policy. You have a desired behavior (provide liquidity). You have a budget (protocol tokens). You need to allocate the budget to maximize the behavior while minimizing cost. The solution space is mechanism design. The constraints are Sybil resistance, capital efficiency, and retention of liquidity after the subsidy ends. The engineer who designs a liquidity mining program is designing a market. The same skills — optimization under constraints, adversarial thinking, measurement of outcomes — apply to any incentive system.

The auto-compounding vault (Yearn, Beefy) is an automation pattern: harvest rewards, sell for the underlying, reinvest, repeat. The loop is a cron job implemented as a smart contract. The gas cost per harvest must be less than the rewards gained, or the strategy loses money. The optimization problem is: given gas costs, reward rates, and pool sizes, what is the optimal harvest frequency? The answer is a function of the same variables that determine cache invalidation intervals, batch sizes, and polling frequencies in any distributed system. The domain is yield. The math is operations research.


References:

  • Vitalik Buterin, "On Liquidity Mining," 2020.
  • Compound, "Compound Governance," 2020.
  • Yearn Finance, "Yearn Vaults," Yearn Documentation.
  • Related posts: Lending Protocols, AMMs

Lending Protocols

Lending is the oldest financial instrument. The Code of Hammurabi regulated interest rates in 1754 BC. Compound and Aave automated lending on-chain with overcollateralized pools and algorithmically set interest rates. The functional origin is 3,800 years old. The code is seven years old. The principle is the same.

defilendingcompoundaavecredit

Lending is the oldest financial instrument. Before there were equities, before there were bonds, before there were derivatives, there were loans. A farmer borrows seed grain after a bad harvest. A merchant borrows capital for a voyage. A king borrows to finance a war. The lender provides resources now in exchange for repayment with interest later. The interest compensates for the time value of money and the risk of default.

The Code of Hammurabi, inscribed in Babylon around 1754 BC, regulated lending. It set maximum interest rates — 33.3% for grain loans, 20% for silver loans. It required loans to be witnessed. It specified penalties for fraudulent lending practices. The code is the oldest surviving legal text. Its largest single subject is lending. Lending is that old. Lending is that central to civilization.

The functional origin: the moneylender

In medieval Europe, lending was constrained by usury laws — the Catholic Church prohibited charging interest on loans. The prohibition created a market niche filled by Jews, who were not subject to canon law, and by Lombards, Italian merchants who developed bills of exchange that disguised interest as exchange-rate differentials. The moneylender lent at interest and was vilified for it. The vilification was moral. The function was economic. The economy needed credit. The moneylender provided it. The moral opposition was overcome by economic necessity.

The Medici family, in 15th-century Florence, systematized lending into banking. They took deposits, made loans, transferred funds across Europe through bills of exchange, and financed trade, government, and the arts. The Medici bank was the largest financial institution in Europe. Its innovation was scale: lending as an institution rather than an individual activity. Deposits funded loans. Loans generated interest. Interest funded expansion. The virtuous cycle was the model for modern banking.

The model had a structural vulnerability: loans were not fully collateralized. The Medici bank lent to kings and princes who could not be compelled to repay. The bank failed in 1494 when its largest borrowers defaulted. The failure was the consequence of unsecured lending to sovereigns. The lesson: collateral matters. The lesson was learned by DeFi lending protocols 500 years later.

DeFi lending: Compound and Aave

Compound, launched in 2018, introduced the pool-based lending model to DeFi. Lenders deposit tokens into a pool. Borrowers borrow from the pool. Interest rates are set algorithmically: as utilization (the percentage of the pool that is borrowed) increases, the interest rate increases. The rate curve incentivizes equilibrium — when borrowing demand is high, rates rise, attracting more deposits and discouraging borrowing. When demand is low, rates fall. The algorithm is the market maker for credit.

All loans are overcollateralized. A borrower must deposit more value than they borrow. If the collateral value falls below the liquidation threshold, the position is liquidated: anyone can repay the loan and claim the collateral at a discount. The liquidation incentive ensures that positions are closed before they become undercollateralized. The overcollateralization eliminates credit risk. The trade-off: capital efficiency. Overcollateralized lending cannot expand the credit supply. It can only recycle existing capital. The limitation is the subject of undercollateralized lending protocols, which are still experimental.

Aave, launched in 2020, extended Compound's model with flash loans (borrow and repay in one transaction, zero collateral), rate switching (stable vs. variable), and multi-asset pools. Aave is now the largest DeFi lending protocol, with billions in total value locked.

The interest rate model

The core innovation of DeFi lending is the algorithmic interest rate. Traditional lending uses credit scores, relationship banking, and manual underwriting. DeFi lending uses a utilization curve: borrow rate = base rate + (utilization rate × multiplier), with a "kink" at optimal utilization where the slope steepens. The curve is transparent. The curve is enforced by code. The curve eliminates the need for credit assessment. The collateral substitutes for the credit score.

The utilization curve is a market mechanism. When utilization is low, rates are low — capital is abundant, borrowers are scarce. When utilization is high, rates rise sharply — capital is scarce, borrowers are abundant. The sharp rise above the kink prevents the pool from being fully utilized, which would prevent depositors from withdrawing. The kink is the safety valve. The safety valve is algorithmic. The algorithm is the lender of last resort.

The reference

Sidney Homer and Richard Sylla, A History of Interest Rates (1963, 4th edition 2005). The definitive history of lending from ancient Mesopotamia to the modern era. The book documents 5,000 years of interest rates across civilizations, tracing the evolution of credit from temple loans in Babylon to the Eurodollar market. The data shows that interest rates reflect the intersection of time preference, risk, and institutional structure. DeFi lending changes the institutional structure. Time preference and risk remain. The book is the context. The context is essential.

The engineering connection

DeFi lending is a state machine. The pool is the state. Deposits, borrows, repays, liquidations are the transitions. The utilization curve is the control law. The entire protocol is a finite state machine with economic incentives as the transition guards. The engineering pattern is the same as any event-sourced system: each transaction is an event, the pool state is the projection, the interest rate is a derived value computed from the utilization ratio. The architecture would be familiar to any engineer who has built a CQRS system. The domain is lending. The pattern is event sourcing.

The liquidation mechanism is a watchdog timer. If collateral value falls below threshold, liquidate. The watchdog is decentralized — anyone can trigger it. The incentive is the liquidation discount. The mechanism is the same as a circuit breaker in a distributed system: detect the fault condition, trigger the protective action, compensate the responder. The domain is different. The control flow is identical.


References:

  • Sidney Homer and Richard Sylla, A History of Interest Rates, Wiley, 4th edition, 2005.
  • Robert Leshner and Geoffrey Hayes, "Compound: The Money Market Protocol," 2019.
  • Aave, "Aave Protocol Whitepaper," 2020.
  • Related posts: Flash Loans, Stablecoins

Stablecoins

A stablecoin is a token that holds its value relative to a reference asset. The idea is older than crypto: David Ricardo proposed a gold-backed currency in 1816. The implementation is newer: fiat-backed (USDT), crypto-overcollateralized (DAI), and algorithmic (UST, which failed). The story of stablecoins is the story of money itself, repeatedly reinvented.

defistablecoinsdaiusdtmonetary-history

The desire for stable value is as old as money. Every monetary innovation — coinage, paper currency, the gold standard, fiat money — was an attempt to create a medium of exchange whose value was predictable. Stablecoins are the latest iteration. They are tokens designed to maintain a peg to a reference asset, typically the U.S. dollar. They are the bridge between the volatile world of crypto assets and the stable world of everyday commerce.

The three types of stablecoins — fiat-backed, crypto-overcollateralized, and algorithmic — each have a different mechanism for maintaining the peg. Each mechanism has a different failure mode. Each failure mode has been demonstrated in production. The demonstrations were expensive. The lessons are public.

The functional origin: David Ricardo and the gold standard

In 1816, David Ricardo published Proposals for an Economical and Secure Currency. Ricardo, the greatest economist of his generation, argued that the Bank of England should stop issuing gold coins and instead issue paper notes fully backed by gold bullion. The public would hold notes, not coins. The notes would be redeemable for gold at a fixed rate. The gold would sit in the Bank's vaults. The system would be more efficient — paper is cheaper to produce and easier to transport than gold — while maintaining the stability of the gold standard.

Ricardo's proposal was not adopted in his lifetime. It was adopted after his death, in the Bank Charter Act of 1844. The act centralized note issuance in the Bank of England and required new notes to be fully backed by gold. The gold standard, in this form, persisted until 1931. The principle — a paper claim on a reserve asset, redeemable at a fixed rate — is the principle of the fiat-backed stablecoin. Tether (USDT) is a Ricardo note. Circle (USDC) is a Ricardo note. The notes are issued on a blockchain instead of on paper. The reserve is held in a bank account instead of a vault. The principle is the same.

The failure mode is also the same. If the reserve is not fully backing the notes — if the issuer issues more notes than it holds reserves — the peg is vulnerable to a run. The Bank of England suspended convertibility in 1797, during the Napoleonic Wars, and again in 1914, during World War I. Tether has never suspended convertibility, but it has never been fully audited. The opacity is the vulnerability. The vulnerability is the subject of ongoing regulatory attention.

Crypto-overcollateralized: DAI

DAI, launched by MakerDAO in 2017, is a stablecoin backed by crypto assets rather than fiat reserves. The mechanism: users deposit crypto collateral (ETH, WBTC, other approved assets) into a vault. They mint DAI against the collateral. The collateralization ratio must exceed a minimum — typically 150%. If the value of the collateral falls below the liquidation threshold, the vault is liquidated: the collateral is sold, the DAI is repaid, and the remaining collateral is returned to the user. The overcollateralization absorbs price volatility. The liquidation mechanism enforces the peg.

DAI is the first decentralized stablecoin that achieved scale. It survived the March 2020 crash — when ETH fell 50% in a day, Maker's liquidation system failed to keep up, and DAI briefly traded above its peg — and emerged with improved mechanisms. It survived the Luna collapse in May 2022 — when UST imploded and algorithmic stablecoins were discredited — and DAI's overcollateralized model was vindicated by contrast. DAI is now the dominant decentralized stablecoin, with a market cap exceeding $5 billion.

The innovation: a stablecoin backed by volatile assets, stabilized by overcollateralization and automated liquidation. The risk: a black swan event that crashes collateral faster than the liquidation system can respond. The risk is managed by collateral diversification, liquidation parameter tuning, and an emergency shutdown mechanism. The risk is not eliminated. It is priced.

Algorithmic: UST and the lesson

TerraUSD (UST), launched in 2020, was an algorithmic stablecoin. It had no reserves. It maintained its peg through a seigniorage mechanism: UST could be redeemed for $1 worth of LUNA, Terra's governance token, at any time. If UST traded below $1, arbitrageurs would buy UST and redeem it for LUNA, profiting from the difference and pushing UST back to $1. If UST traded above $1, arbitrageurs would mint UST with LUNA and sell UST, pushing it back down. The mechanism relied on LUNA having value. In May 2022, confidence in LUNA collapsed. UST lost its peg. The death spiral: UST below peg → arbitrageurs redeem for LUNA → LUNA supply expands → LUNA price falls → confidence falls further → more redemptions. Within days, $40 billion in market value was destroyed.

The UST collapse was the most significant event in stablecoin history. It demonstrated that algorithmic stablecoins without exogenous collateral are vulnerable to death spirals. The vulnerability is not a bug. It is a property of the mechanism. The mechanism works when confidence holds. Confidence is reflexive — it holds when people believe it will hold. When belief cracks, the mechanism accelerates the collapse. The acceleration is the death spiral. The death spiral is a feature of the design.

The lesson: stablecoins require backing. The backing can be fiat (USDT, USDC), crypto overcollateralized (DAI), or a basket of assets. It cannot be pure belief. Belief is not capital. Capital is the foundation. The foundation matters when the market turns.

The reference

Friedrich Hayek, The Denationalisation of Money (1976). Hayek argued that the government monopoly on currency issuance should be abolished and replaced with competing private currencies. Each issuer would maintain its currency's value by promising redeemability and managing supply. The market would select the currencies that maintained stable purchasing power. Hayek's vision was dismissed as utopian. Stablecoins are the partial realization of that vision. Competing private currencies. Redeemability as the mechanism. Market selection as the arbiter. UST failed the market test. DAI passed it. The market is the selector. The selection is ongoing.

The engineering connection

A stablecoin is a control system. The peg is the setpoint. The collateral is the actuator. The oracle is the sensor. When the price deviates from the peg, the mechanism acts: mint, burn, liquidate. The control loop is the same as a thermostat, an autoscaler, a PID controller. The domain is monetary. The engineering is control theory.

DAI and USDC are different control strategies for the same objective — stability. DAI uses crypto collateral and automated liquidation, a reactive strategy. USDC uses fiat reserves and legal redemption, a preventive strategy. The choice between reactive and preventive control is an engineering trade-off between capital efficiency and tail risk. The engineer who understands control theory can evaluate the trade-off. The investor who doesn't is evaluating a black box.


References:

  • David Ricardo, Proposals for an Economical and Secure Currency, 1816.
  • Friedrich Hayek, The Denationalisation of Money, Institute of Economic Affairs, 1976.
  • MakerDAO, "The Maker Protocol: MakerDAO's Multi-Collateral Dai System," 2019.
  • Related posts: AMMs, Scarcity Rules Everything

Options and DeFi Derivatives

Thales of Miletus traded the first recorded option in the 6th century BC — a bet on the olive harvest. The Black-Scholes formula made options mathematically tractable in 1973. DeFi options protocols — Hegic, Opyn, Ribbon — are automating option writing, selling, and settlement on-chain. The instrument is 2,600 years old. The execution is new.

defioptionsblack-scholesderivativeshegic

Aristotle, in the Politics, tells the story of Thales of Miletus. Thales was a philosopher. His critics mocked him for his poverty, arguing that philosophy was useless because it couldn't make money. Thales, using his knowledge of astronomy, predicted a particularly abundant olive harvest. During the winter, when demand for olive presses was low, he paid small deposits to reserve the use of all the olive presses in Miletus and Chios for the following autumn. When the harvest arrived — abundant, as he predicted — demand for olive presses surged. Thales sold his reservations at a premium. He made a fortune.

The transaction was a call option. Thales paid a premium for the right, but not the obligation, to use the olive presses at a future date. If the harvest had been poor, he would have let the option expire. His loss would have been limited to the premium. The structure — limited downside, asymmetric upside, premium paid upfront — is the structure of every option contract since. Thales invented the option. He also proved that philosophy could make money. The two achievements are connected.

The functional origin: Black-Scholes

The modern options market was created by a formula. Fischer Black and Myron Scholes published "The Pricing of Options and Corporate Liabilities" in 1973. The Black-Scholes formula calculates the theoretical price of a European call option as a function of the underlying asset price, the strike price, the time to expiration, the risk-free interest rate, and the asset's volatility. The formula assumes continuous trading, no transaction costs, and log-normal price distributions. The assumptions are false. The formula is still useful. The usefulness derives from the formula's insight: an option can be replicated by dynamically hedging a position in the underlying asset. The replication argument is the foundation of all derivatives pricing.

The Chicago Board Options Exchange opened in April 1973, the same year Black-Scholes was published. The coincidence was not planned. The coincidence was catalytic. Traders had a formula for pricing options. The formula gave them confidence to trade. The trading created liquidity. The liquidity attracted more traders. The virtuous cycle created the modern options market, which now trades trillions in notional value annually.

DeFi options

DeFi options protocols are automating the options market on-chain. The approaches:

Order book options. Hegic, Opyn, and Lyra use on-chain order books or request-for-quote systems for options trading. Buyers and sellers match on-chain. Settlement is automated. The challenge: options are complex instruments with many parameters (strike, expiry, type). The order book for any specific option is thin. The thinness produces wide spreads. The spreads limit adoption.

Automated options vaults. Ribbon Finance and ThetaNuts sell options automatically on behalf of depositors. A depositor deposits ETH into a covered call vault. The vault sells call options against the ETH, collecting premiums. The premiums are distributed to depositors. The vault automates the option selling strategy. The depositor earns yield without managing strikes, expirations, or greeks. The automation is the product. The product is yield.

Structured products. Cega and Friktion combine options into structured products — principal-protected notes, yield enhancement, volatility trading. The investor deposits capital. The protocol executes a strategy involving multiple options positions. The strategy generates yield in most market conditions and loses principal in tail events. The risk is the tail. The tail is priced into the premium. The pricing is the challenge.

Perpetual options. Paradigm and Panoptic are developing perpetual options — options with no expiry, analogous to perpetual futures. The mechanism is a funding rate between option buyers and sellers. The perpetual option eliminates expiration management. The perpetual option is the next frontier.

The reference

Nassim Nicholas Taleb, Dynamic Hedging: Managing Vanilla and Exotic Options (1997). Taleb was a options market maker before he was an author. His book is the practitioner's guide to the reality of options trading — the greeks, the hedging, the tail risks that Black-Scholes assumes away. Taleb's later book, The Black Swan (2007), is about the consequences of those tail risks. The DeFi options market is still young. The tail risks have not yet materialized at scale. When they do, Taleb's framework will be the guide to understanding them. The framework is ready. The market is not.

The engineering connection

An option is a function from an underlying price to a payoff. Black-Scholes is the algorithm that prices that function. DeFi options vaults automate the execution of that algorithm on-chain. The automation is software engineering applied to financial engineering. The vault deposits collateral, sells options, collects premiums, reinvests — the same loop as a CI/CD pipeline: trigger, execute, verify, repeat. The domain is different. The control flow is the same.

The DeFi options stack also illustrates a recurring engineering pattern: complexity compression. Black-Scholes is a partial differential equation. A covered call vault presents it as "deposit ETH, earn yield." The vault compresses the complexity of options pricing, Greeks management, and position monitoring into a single user action. The compression is the product. The product is an abstraction. The abstraction is good if it hides the right things and bad if it hides the wrong things. The vault that hides tail risk from the user is a bad abstraction. The vault that surfaces tail risk is a good one. The engineer's judgment is knowing which is which.


References:

  • Aristotle, Politics, Book I, Chapter XI (Thales and the olive presses).
  • Fischer Black and Myron Scholes, "The Pricing of Options and Corporate Liabilities," Journal of Political Economy, 1973.
  • Nassim Nicholas Taleb, Dynamic Hedging, Wiley, 1997.
  • Related posts: Perpetual Futures, Market Making

Prediction Markets

A prediction market lets people bet on the outcome of future events. The market price is a probability estimate. The functional origin is 16th-century papal conclave betting. The modern incarnation is Polymarket, which correctly called the 2024 U.S. election while polls were uncertain. Markets aggregate dispersed knowledge. The aggregation is the product.

defiprediction-marketspolymarketinformationhayek

A prediction market is a market where participants trade contracts that pay out based on the outcome of a future event. A contract that pays $1 if a candidate wins an election and $0 if they lose. If the contract trades at $0.60, the market's implied probability of the candidate winning is 60%. The price is a probability. The market is a forecasting tool.

The key insight, formalized by Friedrich Hayek in "The Use of Knowledge in Society" (1945), is that markets aggregate dispersed information. No single individual knows the true probability of an event. Each individual holds fragments of relevant knowledge — a pollster has survey data, a journalist has sources, a trader has a model, a local observer has on-the-ground impressions. The market price aggregates these fragments into a single number. The number is more accurate than any individual's estimate. The accuracy is the market's product.

The functional origin: papal conclave betting

Betting on papal elections was common in 16th-century Rome. Cardinals gathered in conclave. The outside world speculated on the outcome. Bookmakers offered odds. The odds fluctuated as news leaked from the conclave — a cardinal was seen visiting another's cell, a delegation arrived with a message from a foreign monarch. The betting markets aggregated the leaks into a probability estimate. The estimate was often more accurate than the assessments of diplomats and ambassadors, who had access to more formal information but lacked the market's ability to weigh competing signals.

The tradition continued. Betting on elections was widespread in the United States in the 19th and early 20th centuries, conducted through organized exchanges and informal bookmaking. The markets were suppressed by anti-gambling laws in the mid-20th century. They reemerged in the 1980s with the Iowa Electronic Markets, a small-scale academic prediction market for U.S. elections. The IEM demonstrated that prediction markets could forecast elections more accurately than polls. The demonstration was academic. The adoption was limited by regulatory constraints.

Polymarket and the 2024 election

Polymarket, launched in 2020, is a decentralized prediction market built on Polygon. Users deposit USDC and trade shares in event outcomes. The market resolves when an oracle — currently UMA's optimistic oracle — reports the outcome. The oracle is the source of truth. The trust in the oracle is the trust in the market.

The 2024 U.S. presidential election was Polymarket's breakthrough moment. The market correctly called the election outcome while traditional polls showed a statistical tie. The market's implied probability moved sharply in the final weeks before the election, aggregating signals that polls were missing or misweighting. The market's accuracy generated mainstream attention. Trading volume exceeded $1 billion. The market became a news source in its own right — journalists cited Polymarket odds alongside polling averages.

The success validated the Hayekian thesis: markets aggregate knowledge that no single institution possesses. Polls ask people what they think. Markets ask people what they'll bet on. The difference is skin in the game. Skin in the game improves accuracy. The improvement is the efficiency of markets.

The mechanism

Prediction markets use a simple mechanism. A binary outcome market has two tokens: YES and NO. Each pays $1 if correct. The tokens trade freely. The price of YES is the market's probability estimate. A trader who believes the true probability is higher than the market price buys YES. A trader who believes it's lower buys NO. The trading moves the price toward the traders' collective estimate. At resolution, the correct token is redeemable for $1. The incorrect token is worthless.

The mechanism is incentive-compatible. Traders profit from correcting the market's errors. The profit motive drives information into prices. The information is the traders' private knowledge. The private knowledge becomes public through the price. The price is the public good. The market produces the public good as a byproduct of private profit-seeking. The byproduct is the innovation.

The reference

Robin Hanson, "Shall We Vote on Values, But Bet on Beliefs?" (2013). Hanson is the leading academic advocate for prediction markets. His proposal: futarchy — a form of government where elected officials set goals and prediction markets determine which policies will achieve them. The proposal is radical. The underlying logic is mainstream: markets aggregate information better than committees. Polymarket is the partial implementation of Hanson's vision. The implementation is for elections, not policy. The extension to policy is the next frontier.

The engineering connection

Prediction markets are information aggregation mechanisms. The price is the signal. The traders are the sensors. The market is the aggregator. This is the same architecture as a distributed monitoring system: each node observes a fragment of the system state, reports its observation, and the aggregator produces a unified view. The market does this without a central aggregator. The price does the aggregation. The price is the emergent output of millions of independent observations, weighted by conviction (position size).

The oracle that resolves the market is a trust-minimized data feed — the same pattern as an API contract. The oracle reports the outcome. The market settles. If the oracle is wrong, the market produces wrong outcomes. The oracle's reliability is the system's reliability. Engineering reliable oracles — UMA's optimistic oracle, Chainlink's decentralized oracle networks — is a distributed systems problem. The problem is: how do you get truthful reports from potentially adversarial reporters? The solution is mechanism design: stake, challenge periods, economic incentives for honesty. The mechanism is the same as staking in proof-of-stake. The engineering is the same. The domain is different.

Prediction markets also teach a software engineering lesson about decision-making. A team estimating a project timeline is a prediction market without prices. Each person has private information. The information is aggregated through discussion, not through trading. Discussion is a worse aggregator than prices because discussion rewards confidence, not accuracy. The person who speaks most confidently shapes the estimate. The person who knows the truth but speaks quietly is ignored. A prediction market would surface the quiet person's knowledge through their willingness to bet on it. The bet is the signal. The price is the aggregation. The engineering team that replaces estimation meetings with internal prediction markets would produce better estimates. No team does this. The technology exists. The adoption is zero.


References:

  • Robin Hanson, "Shall We Vote on Values, But Bet on Beliefs?" 2013.
  • Friedrich Hayek, "The Use of Knowledge in Society," American Economic Review, 1945.
  • Polymarket, "Polymarket Documentation."
  • Related posts: The knowledge is dispersed, On Scarcity

Perpetual Futures

The perpetual futures contract — a futures contract with no expiry — is the most traded financial instrument in crypto. It was invented by Robert Shiller in 1992 for real estate indices. It was adapted by BitMEX in 2016 for Bitcoin. The funding rate mechanism that keeps the contract price close to the spot price is a work of economic engineering.

defiperpetualsfuturesfunding-ratebitmex

The perpetual futures contract is a futures contract with no expiration date. Unlike a traditional futures contract — which settles on a specific date, at which point the contract converges to the spot price — a perpetual contract never settles. The position can be held indefinitely. The mechanism that keeps the perpetual price close to the spot price is the funding rate: a periodic payment between long and short positions. If the perpetual trades above spot, longs pay shorts. The payment incentivizes selling the perpetual and buying spot, which pushes the perpetual price down toward spot. If the perpetual trades below spot, shorts pay longs. The payment incentivizes the reverse. The funding rate is the mechanism. The mechanism replaces expiration as the convergence force.

The perpetual is the most traded instrument in crypto. Daily volumes regularly exceed $100 billion across centralized exchanges. The volume exceeds spot trading volume by a factor of 3-5×. The perpetual is the dominant instrument for speculation, hedging, and leveraged trading. Its dominance is a function of its design: no expiration means no roll costs, no settlement dates to manage, no term structure to model. The perpetual is the simplest possible futures contract. The simplicity drove adoption.

The functional origin: Japanese rice futures

The first futures contracts were traded on the Dōjima Rice Exchange in Osaka, Japan, in the 17th century. Rice was the currency of feudal Japan — samurai were paid in rice stipends, taxes were collected in rice, wealth was measured in rice. The price of rice fluctuated with the harvest. Rice futures allowed merchants and samurai to lock in prices in advance. The contracts had expiration dates. At expiration, the contract converged to the spot price of rice in Osaka.

The Dōjima exchange was the first organized futures market in the world. It predated the Chicago Board of Trade by 150 years. It had standardized contracts, a clearinghouse, and mark-to-market settlement. The infrastructure was sophisticated. The principle — a contract for future delivery at a price agreed today — was the foundation of all derivatives markets that followed.

The perpetual futures contract eliminates the expiration that the Dōjima exchange institutionalized. The elimination is an innovation. The innovation changes the market structure: without expiration, the cost of maintaining a position is reduced to the funding rate. The funding rate is paid periodically — typically every 8 hours. The rate is determined by the market, not by a counterparty. The rate is transparent. The transparency enables algorithmic trading strategies that would be impractical with traditional futures.

The funding rate mechanism

The funding rate is the economic engine of the perpetual. It is calculated as the difference between the perpetual price and the spot price, scaled by a funding interval. The formula: funding rate = (perpetual price - spot price) / spot price / funding intervals per day. If the perpetual trades at a 0.1% premium, longs pay 0.1% per 8-hour period to shorts. The payment is direct — no intermediary, no clearinghouse fee. The payment is enforced by the exchange's smart contract or matching engine.

The funding rate serves three functions. It anchors the perpetual price to spot. It indicates market sentiment — positive funding means the market is bullish (longs are paying to maintain their positions), negative funding means bearish. It generates yield for market-neutral strategies — a trader who buys spot and shorts the perpetual earns the funding rate without directional exposure. The yield is the basis trade. The basis trade is the subject of quantitative strategy research.

During bull markets, funding rates can reach extreme levels — 0.1% per 8 hours, which annualizes to over 100%. The extreme rates reflect extreme demand for leverage. The demand is self-limiting: high funding rates attract basis traders who short the perpetual and buy spot, earning the funding premium while pushing the perpetual price toward spot. The basis traders are the mechanism for market efficiency. The mechanism works. The speed at which it works depends on the capital available for basis trading.

The reference

Robert Shiller, "Measuring Asset Values for Cash Settlement in Derivative Markets: Hedonic Repeated Measures Indices and Perpetual Futures" (1992). Shiller proposed the perpetual futures contract as a mechanism for creating derivative markets on illiquid assets — specifically, real estate indices. The contract would have no expiration and would use a "dividend" payment (what we now call the funding rate) to keep the contract price close to the index value. Shiller's proposal was academic. BitMEX's implementation in 2016 was commercial. The academic proposal became a $100 billion daily market. The market is larger than Shiller's wildest expectation. The mechanism is exactly as he described it.

The engineering connection

The perpetual contract is mechanism design implemented in code. The funding rate replaces the expiration date. Expiration is a hard constraint — a date on a calendar. The funding rate is a soft constraint — an incentive that nudges the market toward equilibrium. The engineering insight: when you can replace a hard constraint with an incentive, replace it. The hard constraint requires enforcement, monitoring, penalties. The incentive is self-enforcing. The market enforces it. The enforcement is free.

This is the same principle as automated contract testing replacing manual API review. Manual review is a hard constraint — someone must check that the API hasn't changed. Automated testing is an incentive — breaking the contract fails the build immediately. The build failure is the funding rate. It nudges the developer toward the equilibrium of stable contracts. The market of developers enforces it. The enforcement is automated.

The perpetual contract is also an example of emergent stability. No central authority sets the funding rate. The rate emerges from the difference between the perpetual price and the spot price. The difference is measured. The rate is calculated. The payment is executed. The loop is a feedback system — Lehman's Eighth Law in financial form. The software engineer who understands feedback systems can build them in any domain. The domain changes. The feedback structure doesn't.


References:

  • Robert Shiller, "Measuring Asset Values for Cash Settlement in Derivative Markets," 1992.
  • BitMEX, "Perpetual Contracts," BitMEX Documentation, 2016.
  • Related posts: Arbitrage, Market Making

Parnas's Information Hiding

David Parnas's 1972 paper introduced information hiding: modularize around design decisions likely to change, not around processing steps. The interface reveals as little as possible.

designparnasinformation-hidingmodulessoftware-architecture

In 1972, David Parnas published a paper that changed how programmers think about modules. The title is dry — "On the Criteria to Be Used in Decomposing Systems into Modules." The content is a controlled explosion.

"The effectiveness of a 'modularization' is dependent upon the criteria used in dividing the system into modules."

Before Parnas, modularization meant dividing the program by what it did, in the order it did it. Step one, module one. Step two, module two. The flowchart was the architecture. Parnas pointed out that this criterion — "major steps in the processing" — is the least stable thing about any system.

"Note, however, that nothing is said about the criteria to use in dividing the system into modules. Because the decision to divide a system into n modules of a given size does not determine the decomposition, this paper will discuss that issue."

Having n modules is not enough. The criterion for drawing the boundaries is everything. The same system with the same number of modules can be flexible or brittle depending on where you put the cuts. Most programmers were putting the cuts in the wrong place. Most still are.

The KWIC index

Parnas illustrates with a KWIC (Key Word In Context) index — a system that takes lines of text, produces all circular shifts, sorts them, and formats output. Small enough to understand in one sitting. Complex enough to teach a fifty-year lesson.

Two modularizations, same functionality:

Modularization 1: conventional. Five modules by processing step. Input. Circular Shift. Alphabetizer. Output. Master Control. The data flows predictably. The interfaces expose data structures: character packing, pointer conventions, core formats. Every module knows how the data is stored. This is how most programmers would build it.

"In the first decomposition the criterion used was make each 'major step' in the processing a module."

Modularization 2: information hiding. Also five modules. Line Storage hides character packing behind access functions. Input hides the input format. Circular Shifter hides the shift algorithm behind access functions identical to Line Storage. Alphabetizer hides the sort. Output hides the format. Each module's interface reveals nothing about how it works inside.

"The second decomposition was made using 'information hiding' as a criteria. The modules no longer correspond to steps in the processing."

"Every module in the second decomposition is characterized by its knowledge of a design decision which it hides from all others. Its interface or definition was chosen to reveal as little as possible about its inner workings."

Interface as minimum revelation. Implementation as concealed volatility. A module is not a step in a process. It is a secret keeper. It knows something the others don't, and it tells them only what they need.

The change propagation test

Parnas applies the test that matters: what modules change when a single design decision changes?

Input format changes. Modularization 1: every module touches the data. Input, Circular Shift, Alphabetizer, Output, Master Control — all break. Modularization 2: one module. The Input module's interface is stable. Nobody else knows the format changed.

Sort algorithm changes. Modularization 1: Alphabetizer and Output break. Modularization 2: one module. The Alphabetizer's interface holds. The rest of the system hasn't heard of the sort algorithm it was using and doesn't need to.

Storage representation changes — packed characters four to a word, or linked lists, or a tree. Modularization 1: every module breaks. Modularization 2: one module. Line Storage. Nothing else even knows how characters are stored.

"It is by looking at changes such as these that we can see the differences between the two modularizations."

The criterion is not function. It is volatility. Ask not what the module does. Ask what it protects you from when the world changes.

Why this was radical

Parnas inverted the logic of decomposition. You don't modularize by what the system does. You modularize by what you want to be able to change without telling anyone.

"We propose instead that one begins with a list of difficult design decisions or design decisions which are likely to change. Each module is then designed to hide such a decision from the others. Since, in most cases, design decisions transcend time of execution, modules will not correspond to steps in the processing."

This is the core of the paper. Begin with the volatile. Wrap each volatile decision in a module whose interface survives the change. The execution order is secondary. The change isolation is primary.

Dijkstra, Parnas's contemporary, had already argued for separation of concerns as a structuring principle. But Dijkstra was thinking about intellectual manageability — can you reason about one part without holding the whole in your head? Parnas added the operational criterion: can you change one part without touching the others? The two criteria converge. What you can change without touching is what you can reason about in isolation. What you can reason about in isolation is what you can build independently.

"The major progress in the area of modular programming has been the development of coding techniques and assemblers which allow one module to be written with little knowledge of the code used in another module... but its use has not resulted in the expected benefits."

The tools existed. The assemblers could do separate compilation. Modules could be swapped without full rebuilds. The mechanics worked. The designs didn't. Because the criterion was wrong.

The efficiency objection

Parnas was honest about the cost. Information-hiding modularizations, implemented conventionally as subroutines, would be less efficient than the conventional decomposition. More function calls. More indirection. More boundaries to cross.

"The unconventional decomposition, if implemented with the conventional assumption that a module consists of one or more subroutines, will be less efficient in most cases."

He sketched an alternative: a preprocessor that inlines module accesses at compile time. The modularization stays clean. The runtime code stays fast. This was 1971. He was describing what we now call zero-cost abstractions. C++ would take twenty more years to get there. Rust would take forty.

Hoare, in his work on data abstraction, was developing compatible ideas from a different direction — proving modules correct through their specifications. Parnas was less interested in proof than in adaptability. Hoare wanted to know the module was right. Parnas wanted to know you could change your mind about what was right without rewriting everything. Both were necessary. Neither was sufficient alone.

What "information hiding" actually means

The term has been so thoroughly absorbed that it now means almost nothing. Most "encapsulation" in production code is theater. A getter that returns the internal data structure is not hiding anything. It is exposing the decision with extra syntax.

"Its interface or definition was chosen to reveal as little as possible about its inner workings."

As little as possible. Not "as little as convenient." Not "as little as the framework supports." As little as the problem allows while still being useful. If you change the data structure and the caller's code still compiles, you did information hiding. If the caller imports the type, you didn't. If the caller casts to the internal type, you really didn't. If the caller wrote their own parser for your output format, you have lost control of your own design decision and someone else is now coupled to a choice you didn't even know you were making.

Wirth, with stepwise refinement, had given programmers a method for designing top-down. Parnas gave them a reason to design bottom-up — from the volatile decisions outward. The two methods are compatible but the emphasis is different. Wirth asked: what are the steps? Parnas asked: what are the secrets? The steps change. The secrets change too, but when they do, you want them contained.

The connection to Brooks

This is the paper referenced throughout the Brooks on Software Design series. Parnas and Brooks worked the same era and reached compatible conclusions from different premises.

Brooks: "The building of a design is the forcing of the will of one upon the stuff of the world." One mind controls the interfaces. Parnas: those interfaces exist to hide the volatile decisions. One mind decides what to hide. The rest of the system builds against stable interfaces and doesn't need to know.

Brooks without Parnas: one mind, but no criterion for where to draw the modular boundaries. Parnas without Brooks: the right criterion, but no mechanism for enforcing it across a system. Together: one designer, one set of secrets, one set of stable interfaces. That is the architecture of every system that has aged well. That is also the architecture of almost no system you actually work on.

The paper is from 1971. It is fifty-five years old. It was right then. It is right now. Most production code still violates its central principle — modularizing by step, not by secret, and wondering why changes ripple. This is either comforting or damning. Parnas would probably note that both reactions are consistent with his theory. He'd hidden which one he meant. That's how you know the man understood his own idea.


Reference: David L. Parnas, "On the Criteria to Be Used in Decomposing Systems into Modules," Communications of the ACM, Vol. 15, No. 12, December 1972, pp. 1053-1058. PDF

Infrastructure choice is engineering choice. The protocol, the format, the runtime — each is a decision that shapes everything built on top of it. DOT for pipelines, Temporal vs DBOS for durable execution, NATS for messaging. The choice determines the coupling, the scaling, the failure modes. The engineer who treats infrastructure as a commodity gets the failure modes of the default. The engineer who treats it as a design decision gets the failure modes they chose.

A module is not a step in a process. It is a secret keeper. It knows something the other modules don't and reveals only what they need. The interface is the revelation. The implementation is the secret.

McCulloch's question: what is a number, that a man may know it?

Warren McCulloch's 1960 lecture traces a single question — asked at age 19 — through Augustine, Duns Scotus, Hume, Russell, Turing, and the neural circuitry of the brain, ending with a probabilistic logic for fallible neurons.

mccullochcyberneticsneural-networkslogicepistemologyhistory-of-science

In 1917, a nineteen-year-old Warren McCulloch was called into the office of the Quaker philosopher Rufus Jones at Haverford College.

"Warren," said Jones, "what is thee going to be?"

"I don't know."

"And what is thee going to do?"

"I have no idea; but there is one question I would like to answer: What is a number, that a man may know it, and a man, that he may know a number?"

Jones smiled. "Friend, thee will be busy as long as thee lives."

Forty-three years later, McCulloch delivered the Alfred Korzybski Memorial Lecture under that same title. He had been busy. The lecture is a compressed intellectual autobiography — part history of philosophy, part neuroscience manifesto, part mathematical logic — and it traces how one man spent a lifetime trying to ground epistemology in the physics and chemistry of the brain.

The two halves of the question

The title is recursive for a reason. McCulloch's argument is that you cannot answer "what is a number?" without also answering "what is a knower?" The two questions are one question. Any theory of number that does not account for the biological system that apprehends number is incomplete. Any theory of mind that cannot account for how a physical system grasps mathematical truth is insufficient.

This is not a casual framing. It is the central methodological commitment of McCulloch's career: reduce epistemology to an experimental science. Not by philosophizing about knowledge, but by understanding the neural circuitry that produces it. The lecture is the story of that attempt.

The philosophical arc: from Augustine to Hume

McCulloch traces the Western approach to number through four theological principles, each of which maps onto an epistemological position.

The eternal verities. Augustine, around 500 AD: "7 and 3 are 10; 7 and 3 have always been 10; 7 and 3 at no time and in no way have ever been anything but 10; 7 and 3 will always be 10." These are truths independent of time and place — ideas in the Mind of God, which we can understand but never fully comprehend. Augustine's examples are drawn from arithmetic, geometry, and logic, but he includes what we would now call the laws of Nature. The history of Western science — Galileo, Newton, Einstein, the tensor invariant — would have been no shock to his theology.

Authority. The scholastic reliance on texts and their interpretations. McCulloch moves quickly through this, noting the persistent questions of textual corruption and translation, but his real interest is in what came next.

The shift to experiment. Roger Bacon insisted that the eternal verities and the authorities must be tested not only by reason — the old meaning of "experiment" — but by looking again at Nature. Natural law began to grow. Duns Scotus, the last great scholastic defender of realistic logic, formalized the three roads to truth: deduction (from rules and cases to facts), induction (from cases and facts to rules), and abduction (from rules and facts to the hypothesis that the fact is a case under the rule). Abduction is the breeding place of scientific ideas, of intuition, of insight. McCulloch would return to this at the end of his life as the unsolved problem.

The nominalist break. William of Ockham, "greatest of nominalists," severed logic from empirical science. His demand that no conclusion contain what was not in the premises barred two of the three roads to truth and reduced the third — deduction — to vacuous tautology. Logic decayed. Science was born. Law usurped the throne of Theology, and Science began to usurp the throne of Law.

Then came Hume. At twenty-three, Hume had shown that only in logic and arithmetic can we argue through any number of steps, because only here do we have the proper test of equality: "When a number hath a unit answering to an unit of the other we pronounce them equal." One-to-one correspondence.

What a number is

Bertrand Russell, whom McCulloch credits as the first to thank Hume properly, gave the definition: a number is the class of all classes that can be put into one-to-one correspondence with it. The number 7 is the class of all classes that can be put into one-to-one correspondence with the days of the week.

McCulloch accepts this definition and then makes a critical observation: the numbers 1 through 6 are perceptibles. Experiments on many animals — birds, rats, primates — show this. A creature can distinguish 3 from 4 without counting. These are, in Ockham's phrase, natural terms — shared with the beasts. All larger integers are countables, arrived at by a symbolic process: putting pebbles in pots, cutting notches in sticks, establishing one-to-one correspondences. These are conventional terms — "tricks for setting things into one-to-one correspondence" — that grew out of communication, out of logos.

The definition of number depends on two foundations: the perception of small whole numbers (a biological given) and a symbolic process of one-to-one correspondence (a cultural achievement). This is the answer to the first half of the question.

What a man is

The second half is harder. McCulloch's attempt to answer it consumed his career.

In 1923, he attempted to write a logic of transitive verbs and failed. The problem was too hard, the available logic too primitive. He pivoted to a different approach: invent a "least psychic event" — a psychon — with four properties. It either happened or it didn't. It happened only if its bound cause had happened. It proposed this fact to subsequent psychons. And these could be compounded to produce more complex propositions.

In 1929 it dawned on him: these events might be the all-or-none impulses of neurons, combined by convergence onto the next neuron to yield propositional complexes. A neuron fires if and only if its input conditions are met. That firing implies its antecedent. It signals this to downstream neurons. Networks of such units compute logical functions of their inputs.

This was the seed of the 1943 paper with Walter Pitts, "A Logical Calculus of the Ideas Immanent in Nervous Activity" — one of the founding documents of neural network theory, artificial intelligence, and computational neuroscience.

The 1943 paper and its consequences

The paper proved several things simultaneously. First, that networks of neurons computing simple logical functions could extract any configuration of signals from their input. Second, because Gödel had arithmetized logic, and Turing had shown that a simple machine could compute any computable number, nets of neurons were equivalent to Turing machines. The brain was a computer, and computers could be brains — not metaphorically, but formally, in the structure of the proof.

But the paper did more. Pitts's modulo mathematics allowed them to analyze circuits with closed paths — reverberating loops — and to set up a theory of memory. A memory is a temporal invariant: given an event at one time, and its regeneration at later times, one knows that there was an event of that kind. In logical notation: (∃x)(ψx). There exists some x such that x was a ψ. Given this and negation — for which neural inhibition suffices — you get the lower predicate calculus with equality, which was recently proved to be a sufficient logical framework for all of mathematics.

The brain, in other words, contains the logical machinery for mathematics. This is not a metaphor. It is a claim about circuit structure.

McCulloch and Pitts followed this with "How We Know Universals" (1947), which generalized the mechanism. Any object — any universal — is an invariant under some group of transformations. A square is an invariant under 90-degree rotations. A face is an invariant under changes in expression, angle, and lighting. The neural net need only compute averages over the group of transformations to recognize the universal. The mechanism is general.

Von Neumann's problems: reliability from unreliable parts

The second half of the lecture concerns a set of problems posed by John von Neumann, who had absorbed the McCulloch-Pitts model and used it in teaching computing machine theory. Von Neumann wanted to know: how can a brain made of fallible neurons compute reliably?

His attempt — "Toward a Probabilistic Logic" — made three assumptions, each of which was fatal. He assumed failures were absolute (not dependent on signal strength or threshold). He assumed neurons had only two inputs. He assumed each computed the same single function. Under these constraints, reliable computation from unreliable parts is nearly impossible.

McCulloch and his collaborators spent years dismantling these assumptions one by one. With Leo Verbeek, he showed that the error probability of a net can be reduced to the error probability of a single output neuron, and that this can be further reduced by parallel output neurons. With Manuel Blum, he proved that excitation, inhibition, and inhibitory interaction between afferent fibers are necessary and sufficient for neurons to compute their logical functions across the full range from coma to convulsion. With Eugene Prange, he showed that neurons with multiple inputs and controlling signals can compute the vast majority of possible logical functions — the great logical redundancy of the system buying reliability.

The brain does not work despite its unreliable components. It works because of the redundancy that unreliable components make necessary. The logic is probabilistic — not a logic of probabilities (where the arguments are uncertain but the logical operations are certain), but a probabilistic logic (where the functions themselves are infected by chance). This is a thing "of which Aristotle never dreamed."

The Venn functions

McCulloch closes with a practical tool he developed for teaching this logic to neurologists and psychiatrists. Using a simplified Venn diagram notation — an X with four quadrants, each marked with a jot (true), blank (false), or probability p — he created a visual calculus for probabilistic logic that a twelve-year-old could learn in minutes.

The point is not the notation. The point is that the logic of fallible neurons is tractable. It can be taught. It can be computed. It can be programmed. The tools exist to reason rigorously about systems whose components misbehave.

The unanswered question

McCulloch ends with a confession. The problem of insight — of intuition, of invention, of abduction — is still unsolved. A child learns at least one logical particle ("neither" or "not both") from ostension alone. How? We do not know. Tarski thought we lacked a fertile calculus of relations of more than two relata. McCulloch, at sixty-two, felt too old to tackle it: "Too bad — I'm too old. I may live to see the youngsters do it."

He did not. The problem of how a physical system generates insight — how it abduces a hypothesis from a rule and a fact — remains open. McCulloch would have been delighted by the progress in machine learning, but he would also have noted that we still do not have a satisfactory theory of abduction stated in terms of neural circuitry. The question he asked Rufus Jones in 1917 is not fully answered.

Why this lecture matters

McCulloch's lecture is easy to misread. It looks like a ramble through the history of philosophy, some personal anecdotes, and a technical appendix on probabilistic logic. It is actually a unified argument:

  1. Numbers are relations of one-to-one correspondence, built on a biological foundation (the perception of small quantities) and a cultural one (the symbolic process of counting).
  2. A knower is a neural system whose circuitry implements the lower predicate calculus with equality — sufficient for all of mathematics — using neurons as propositional units and reverberating loops as memory.
  3. The logic must be probabilistic because the components are fallible, but the system can be made arbitrarily reliable through redundancy.
  4. The unsolved problem is abduction — how a physical system generates hypotheses — and it is the problem that connects epistemology to neuroscience.

The lecture is McCulloch's intellectual testament. He died nine years later. The question in its title is still worth sitting with: not because we lack answers, but because the answers we have reveal how much we still do not understand about how a physical system — a brain, a body, a network of fallible cells — comes to know that 7 and 3 are 10, and always have been, and always will be.

The question, sixty-five years later

In February 2025, a paper appeared on arXiv with a title that would have made McCulloch smile: "What is a Number, That a Large Language Model May Know It?" Raja Marjieh, Veniamin Veselovsky, Thomas Griffiths, and Ilia Sucholutsky picked up McCulloch's question and pointed it at the system that has come closest to realizing his vision of a thinking machine.

Their finding is both elegant and troubling. LLMs represent numbers through an entangled representation — a blend of string-based similarity (Levenshtein edit distance) and numerical magnitude (log-linear distance). The digit sequence "911" activates both its string properties (it looks like "191" and "119") and its quantity properties (it is close to 910 and 912). These two representations are not cleanly separable in the model's latent embeddings. Context can reduce the entanglement but cannot eliminate it.

This is not how humans know numbers. McCulloch's distinction between perceptibles (1–6, shared with beasts, grounded in dedicated neural circuitry for quantity) and countables (larger integers, requiring symbolic convention and one-to-one correspondence) describes a cognitive architecture with clean boundaries. The biological brain has specialized circuits for numerosity. The LLM has a single set of weights that must simultaneously learn that digits are symbols with edit distances and quantities with magnitudes. The two forms of knowledge bleed into each other.

The practical consequences are real. The paper shows that representational confusion propagates into downstream decisions. Ask an LLM to reason about numbers and it may be influenced by how similar the digit strings look rather than what they mean. This is a category error in silicon — exactly the kind of failure that McCulloch's careful distinction between natural terms and conventional terms was designed to prevent.

The deeper point is that McCulloch's question was always about architecture. What kind of system can know a number? His answer was specific: a system with propositional units (neurons), temporal invariants (memory loops), and a mechanism for computing invariants under groups of transformations (universals). The LLM is a different architecture — a stack of attention layers trained on next-token prediction — and it knows numbers differently as a result. The entanglement that Marjieh et al. document is not a bug in the training data. It is a consequence of the architecture.

McCulloch would have been fascinated. He would have recognized the paper as empirical epistemology in his own tradition — asking, experimentally, how a particular kind of knower represents number. And he would have noted, with some satisfaction, that sixty-five years after his lecture, the question he asked Rufus Jones still organizes a research program.

"What is a number, that a man may know it, and a man, that he may know a number?" — Warren McCulloch (1961)


Reference: Warren S. McCulloch, "What Is a Number, that a Man May Know It, and a Man, that He May Know a Number?" Alfred Korzybski Memorial Lecture, 1960. Published in General Semantics Bulletin, No. 26/27, 1960, pp. 7–18. Full text (PDF).

Related: Raja Marjieh, Veniamin Veselovsky, Thomas L. Griffiths, Ilia Sucholutsky. "What is a Number, That a Large Language Model May Know It?" arXiv:2502.01540, February 2025.

Infrastructure choice is engineering choice. The protocol, the format, the runtime — each is a decision that shapes everything built on top of it. DOT for pipelines, Temporal vs DBOS for durable execution, NATS for messaging. The choice determines the coupling, the scaling, the failure modes. The engineer who treats infrastructure as a commodity gets the failure modes of the default. The engineer who treats it as a design decision gets the failure modes they chose.

A number is not a thing. It is a relation. The relation is one-to-one correspondence. The brain that perceives the relation is a neural circuit. The circuit can be understood. The understanding is epistemology reduced to physiology.

Henney's Microservices

Kevlin Henney never wrote a microservices book. He wrote the books you should have read before adopting microservices. Coupling is shared knowledge. Architecture is measured by cost of change. Modularity is in the relationships, not the services.

kevlin-henneymicroservicesmodularitycouplingsoftware-architecture

Kevlin Henney never wrote a book called Building Microservices. He wrote the books you should have read before adopting microservices — the POSA pattern volumes, 97 Things Every Programmer Should Know, and two decades of columns and talks on modularity, coupling, and the cost of change.

The microservices movement adopted Conway's Law and forgot Parnas. Adopted bounded contexts and forgot coupling. Adopted independent deployability and forgot that deployment independence requires interface stability — and interface stability requires getting the boundaries right. Henney's work explains what went wrong. It also explains what to do instead.

What a module is

Before microservices, there were modules. Before modules were a deployment unit, they were a design decision. Henney, drawing on Parnas and the pattern literature, defines modularity in terms of knowledge — not code, not deployment, not team boundaries.

"The modularity of a system cannot be evaluated by examining designs of individual modules in isolation. The goal of modular design is to simplify the relationships between components of a system."

This sentence indicts most microservices architectures. Teams design services in isolation — one team per service, optimizing locally — and wonder why the system is a distributed mess. You cannot evaluate modularity by looking at one service. You evaluate it by looking at what happens when one service changes. If the change stays inside the service, the boundary is in the right place. If it propagates, the boundary is decoration. The modularity is not in the service. It is in the relationships between services.

"Coupling is the aspect of a system that defines what knowledge is shared between components of a system. Different ways of coupling components share different types and amounts of knowledge. Some will increase complexity, while others will contribute to modularity."

Coupling is shared knowledge. Not shared libraries. Not shared databases. Not shared deployment pipelines. Shared knowledge. When Service A knows the internal data format of Service B, they are coupled — regardless of whether they communicate via HTTP, gRPC, or a message queue. The transport is irrelevant. The knowledge is the coupling. If Service B changes its schema and Service A breaks, the coupling is real. If Service B changes and Service A doesn't notice, the coupling was imaginary. Most organizations don't know which kind they have. They find out in production. The finding is expensive.

The test of architecture

Henney channels Grady Booch for the definition that should be printed above every microservices whiteboard:

"All architecture is design but not all design is architecture. Architecture represents the significant design decisions that shape a system, where significant is measured by cost of change."

Cost of change. That is the criterion. If changing a decision requires rewriting one service, it was local design. If it ripples across seven services, it was architectural — and you got the boundary wrong. Most microservices failures are boundary failures. The services work individually. The boundaries between them are in the wrong places. When the business requirement changes, the change crosses five service boundaries and requires coordinated deploys. The architecture didn't reduce the cost of change. It increased it. You distributed a system to make changing your mind more expensive. That is not progress.

"The aggressive pursuit of LCHC (low coupling and high cohesion) can ensure that the effect of change is simplified and isolated, rather than traumatic and global. LCHC also simplifies testing, building, versioning, experimentation, optimization, team organization, and pretty much any other development activity you can think of."

LCHC is the goal. Microservices are one mechanism for achieving it — but only if the boundaries are right. LCHC simplifies everything Henney lists: testing, building, versioning, experimentation, optimization, team organization. If your microservices migration made any of these harder, you didn't achieve LCHC. You achieved distribution without modularity. That is the worst of both worlds: the complexity of a distributed system with the coupling of a monolith. You kept the monolith's problems and added network latency.

"It is possible to simplify the structure of software without losing effective options. It is even possible to do so and increase your options. Now, that sounds worthwhile: simpler and more flexible."

Simpler. More flexible. After your migration, is the system both? If yes, the boundaries are correct. If no — if it's more complex, harder to change, harder to reason about — the modularity didn't improve. The distribution was cosmetic. The coupling was the real structure all along.

Boundaries under uncertainty

Henney's deepest architectural insight is about decisions you don't yet know how to make:

"When a design decision can reasonably go one of two ways, an architect needs to take a step back. Instead of trying to decide between options A and B, the question becomes 'How do I design so that the choice between A and B is less significant?' The most interesting thing is not actually the choice between A and B, but the fact that there is a choice between A and B."

The existence of a choice is a signal. It means the boundary is volatile. Don't pick A or B. Design the interface so that A and B are interchangeable behind it. This is Parnas's information hiding applied to architectural decisions: hide the decision behind a stable interface, and the choice becomes less significant because you can change your mind later without telling anyone.

Applied to microservices: don't ask "should this be one service or two?" Ask "how do I design the interface so that whether it's one or two doesn't matter to the callers?" If you can split a service without the callers updating, the interface was right. If splitting forces every caller to redeploy, the interface was coupled to the deployment topology. That coupling is the problem. The service count is a distraction. The interface stability is everything.

"Although we cannot predict the future with any certainty, it is still possible to write code that is graceful and accommodating — rather than troublesome and resistant — in the face of change."

Graceful in the face of change. Not "scales to a million users." Not "uses the latest framework." Can you change your mind about a decision without rewriting the system? If yes, the architecture works. If no, the architecture is decoration with good slides.

Simplicity before generality

The principle that should govern every microservices platform decision:

"The best route to generality is through understanding known, specific examples and focusing on their essence to find an essential common solution. Simplicity through experience rather than generality through guesswork."

Build the specific service first. Make it work. Then, when you have a second specific service that resembles the first, find the common essence and extract it. This is the opposite of how most microservices platforms are built. The platform team designs a general-purpose service template with twenty configuration options, five deployment modes, and an abstraction layer that handles every possible use case — most of which never occur. That's not architecture. That's prophecy. Prophecy is usually wrong.

"A common problem in component frameworks, class libraries, foundation services, and other infrastructure code is that many are designed to be general purpose without reference to concrete applications. This leads to a dizzying array of options and possibilities that are often unused, misused, or just not useful."

The microservices platform designed before any services existed. The shared library that abstracts every database. The common logging framework with seventeen configuration levels. These are not solutions. They are options factories. They produce optionality, not functionality. Most of the options are never used. They exist because someone thought they might be needed. They are inventory. Inventory has cost. The cost is paid by every team that has to understand the platform before they can build the service.

"Favoring simplicity before generality acts as a tiebreaker between otherwise equally viable design alternatives. When there are two possible solutions, favor the one that is simpler and based on concrete need rather than the more intricate one that boasts of generality."

"The trick to achieving generality is, somewhat counterintuitively, to make the code specific enough to be fit for purpose."

Specific enough to work. General enough to adapt. Most microservices platforms miss both: too general to be useful for the specific case, too specific to adapt when the case changes. The middle is where Henney operates. It's harder. It's worth it.

The seven habits, distributed

Henney's most-watched talk — Seven Ineffective Coding Habits of Many Programmers — examines patterns acquired by imitation and never questioned. Each habit has a microservices equivalent. Each equivalent is common. Each is wrong.

1. Noisy code

"Comments should provide additional information that is not readily obtainable from the code itself. They should never parrot the code."

"A common fallacy is to assume authors of incomprehensible code will somehow be able to express themselves lucidly and clearly in comments."

Distributed: Swagger docs that repeat the endpoint name. Architecture decision records that say "we chose Kafka because it scales." Documentation that restates what the API already says. Noise is noise at any scale. A team that can't write a clear function won't write a clear service contract. The scale changes. The skill doesn't. If you can't modularize a monolith, you can't modularize microservices. You'll distribute the spaghetti.

2. Visual dishonesty

Code layout must reflect code structure. If indentation lies, the reader is confused. Distributed: the architecture diagram shows clean bounded contexts. The runtime call graph shows a complete mesh. The diagram lies. The system's structure is in the call graph, not the whiteboard. Visual dishonesty at the service level is more dangerous because the evidence is harder to see. You have to instrument the system to discover what the code would have told you directly.

3. Lego naming

"More words is not more meaning."

Distributed: CustomerProfileServiceManagerFactory. OrderProcessingOrchestrationController. EnterprisePaymentGatewayAdapterImpl. Lego naming at the service level catalogues rather than distinguishes. A service name should say what domain function it performs, not what design patterns it contains. The patterns are implementation. The domain function is the contract.

4. Under-abstraction

Primitive obsession scaled to services. Services that exchange raw JSON with no schema. Services that pass database IDs instead of domain references. "REST" APIs that are thin wrappers over CRUD. If your service API mirrors your database schema with HTTP verbs, you have not abstracted. You have exposed a tunnel into your storage. The coupling is total. Every caller is now coupled to your schema. Every schema change is a breaking change. The API didn't abstract the storage. It published it.

5. Unencapsulated state

Henney's analogy: wearing your underwear on the outside. Distributed: services that share a database. Services that read each other's tables. Services that depend on another service's internal state transitions rather than its published events. If Service A queries Service B's database directly, Service B has no encapsulation. Its state is public. Any schema change in B breaks A. The services are not independent. They are co-dependent with extra network hops and a false sense of modularity.

6. Getters and setters

"When it is not necessary to change, it is necessary not to change." — Lucius Cary

IDE-generated getters and setters become framework-generated CRUD endpoints. GET /customers/{id}, PUT /customers/{id}, DELETE /customers/{id}. Generated. Never questioned. The API exposes mutation where mutation should be a domain event (do you DELETE a customer or close their account?). The API exposes internal structure because the framework generated it from the entity. This is not an API. It is an ORM with a public URL and a false sense of RESTfulness.

7. Uncohesive tests

"For tests to drive development they must do more than just test that code performs its required functionality: they must clearly express that required functionality to the reader." — Nat Pryce & Steve Freeman

One test class per class becomes one integration test suite per service. Tests that verify internal implementation rather than external behavior. When you refactor a service's internals, the tests break — not because behavior changed, but because tests were coupled to implementation. The same anti-pattern, distributed. Test behavior. Not structure. At every scale. A test suite that prevents refactoring is not a safety net. It is a cage.

Code is inventory, not an asset

"Less code, more software. Less code = less bugs."

"Programmers write code: a formal plan of the software, expressing its intent in maximal detail. Software is the end product: in execution it is what the user perceives, interacts with and experiences. Sometimes this difference can be significant."

Code is the plan. Software is the product. Confusing them is why organizations measure productivity in lines written and celebrate growing codebases. A growing codebase is a growing liability. Every line must be tested, maintained, understood, migrated, deleted. The asset is the solved problem. The code is what it cost. A microservices migration that produces more total lines of code than the monolith it replaced is not an improvement. It is inventory growth. The problem didn't get bigger. The solution did.

"Duplicate code has a bad smell and violates the DRY principle... Contrary to benighted management belief, more is not better in this case. Every problem has an optimal code size: too short and the code is cryptic line noise; too long and you cannot see the wood for the trees."

Optimal size exists. Not zero. Not whatever ships. Somewhere in between. Finding it is the discipline. Most teams never look. They add code until the feature works, then stop. The codebase grows monotonically. Nobody's job is deletion. In a microservices world, the cost of excess code is amplified: every service carries its own surplus, and the surplus in each service compounds across the system. Ten services, each 20% larger than needed, is not 20% waste. It is 20% waste multiplied by ten different maintenance, testing, and deployment pipelines. The overhead is multiplicative. The benefit is imaginary.

The books behind the argument

Henney's microservices-relevant work spans two decades:

Pattern-Oriented Software Architecture, Volumes 4 & 5 (2007, with Frank Buschmann and Doug Schmidt). Volume 4: A Pattern Language for Distributed Computing. Volume 5: On Patterns and Pattern Languages. Read Volume 4 before you split another service. The patterns you need — distribution, communication, coordination — are already named and documented. You are discovering them the hard way, expensively, in production. The patterns are free. The downtime is not.

97 Things Every Programmer Should Know (2010, editor). Henney's six contributions are micro-essays in modularity:

  • "Comment Only What the Code Cannot Say." The code says what. The comment says why. At the service level: the API says what. The documentation says why the API exists. Don't document the endpoint. Document the domain decision that created it.
  • "Test for Required Behavior, Not Incidental Behavior." Test the contract, not the implementation. At the service level: test the API contract, not the internal implementation. If an internal refactor breaks the test suite, the tests were coupled. Decouple them.
  • "Test Precisely and Concretely." Vague tests are worse than no tests — they give false confidence. A service integration test that passes with a mock is testing the mock, not the service.
  • "Name the Date." Don't write getDate(). Write getExpiryDate(). Don't name a service data-processor. Name it invoice-generator. The extra word is not noise. It is the contract.
  • "Program with GUTs." Good Unit Tests. Test code is code. It deserves review, refactoring, the same standards. Service tests that nobody reads are not tests. They are rituals.
  • "Uncheck Your Exceptions." Checked exceptions create coupling. At the service level: checked exceptions are shared error types that force every caller to depend on every callee's internal error taxonomy. That coupling crosses service boundaries. It shouldn't.

97 Things Every Java Programmer Should Know (2020, co-editor with Trisha Gee). Same format, Java lens. The editorial voice is unchanged: practical, specific, allergic to dogma.

What Henney knew

The microservices movement made modularity a deployment concern. Henney's work — and the pattern literature he co-authored — makes modularity a knowledge concern. The deployment follows the knowledge boundary. Not the other way around.

If you split by knowledge — by what each module knows and hides — the deployment topology emerges naturally. Each knowledge boundary becomes a service boundary. Each hidden decision becomes an internal implementation. Each stable interface becomes an API contract. The services are modular because the knowledge is modular. The deployment is a consequence.

If you split by database table, by team size, or by "one service per aggregate," the knowledge boundaries don't match the service boundaries. Knowledge leaks across services. Coupling becomes the runtime reality, regardless of what the diagram shows. The services are not modular. They are fragments of a monolith, communicating over a network, with all the original coupling intact and new failure modes added.

Henney never wrote Building Microservices. He wrote the theory that explains why most of them fall over. Coupling is shared knowledge. Architecture is measured by cost of change. Modularity lives in the relationships, not the services. Simpler and more flexible is the only test that matters. Most microservices migrations fail that test. Henney's work explains why. It also explains how to pass it. The books are short. The ignorance is expensive.


References:

  • Kevlin Henney, Frank Buschmann, Doug Schmidt, Pattern-Oriented Software Architecture, Volumes 4 & 5, Wiley, 2007.
  • Kevlin Henney (ed.), 97 Things Every Programmer Should Know, O'Reilly, 2010.
  • Kevlin Henney and Trisha Gee (eds.), 97 Things Every Java Programmer Should Know, O'Reilly, 2020.
  • Seven Ineffective Coding Habits of Many Programmers, NDC/BuildStuff, 2014. InfoQ
  • The Architecture of Uncertainty, Agile Singapore, 2013.
  • Simplicity Before Generality, Use Before Reuse, Artima, 2012.
  • From Mechanism to Method: Generic Decoupling, Overload 60, 2004.
  • GOTO Conferences YouTube — Kevlin Henney

Infrastructure choice is engineering choice. The protocol, the format, the runtime — each is a decision that shapes everything built on top of it. DOT for pipelines, Temporal vs DBOS for durable execution, NATS for messaging. The choice determines the coupling, the scaling, the failure modes. The engineer who treats infrastructure as a commodity gets the failure modes of the default. The engineer who treats it as a design decision gets the failure modes they chose.

Modularity is not in the services. It is in the relationships between them. A system of twelve services in a complete call graph is not modular. It is a monolith with network latency.

Brooks on Software Design Series: why experts get it wrong

Novice mistakes are easy to spot. Expert mistakes are comprehensively, systematically wrong. And the organization is designed to prevent anyone from noticing.

designfred-brooksexpertiseparadigm-trapfeedback

Design is empirical. Two forces undermine empiricism. One is cognitive. The other is organizational. Both are invisible to the people inside them. Both are obvious to everyone outside. This is the definition of a structural problem.

Definitions

Paradigm trap. Deep expertise in one paradigm becomes a liability when the paradigm shifts. Intuition points systematically wrong. Your instincts become your enemy. Slowly. Without telling you. You feel smarter than ever.

Novices make technical mistakes. Wrong data structure. Missed edge case. Tests catch these. The system breaks visibly. The junior engineer learns something. Progress occurs. The mistake is the lesson.

Experts make a different mistake. Designs that are comprehensively, systematically wrong. Internally consistent. Well-executed. Every part fits. And it solves the wrong problem. Or uses assumptions from a previous era. Or optimizes for a constraint that no longer exists. The expert is the last to know because the expert validates the design against the expert's own assumptions. That is not a review. That is a mirror. Mirrors don't find bugs.

"The expert's very expertise becomes a liability when the paradigm shifts. The habits that served him well in one era mislead him systematically in the next. He is not making small errors; he is solving yesterday's problem with today's tools."

Term from Thomas Kuhn, The Structure of Scientific Revolutions (1962). Old-paradigm practitioners are last to see the new one. The people who built mainframes didn't invent PCs. The monolith experts didn't lead microservices. The relational purists said NoSQL would never work. They were right about some things and wrong about others. The paradigm shifted anyway. It always does.

Paradigm shifts: mainframes → minicomputers → microservices → probabilistic systems. At each step, the previous generation's experts produced beautiful, coherent designs for the wrong problem. A novice's bug fails a test. An expert's paradigm error passes all tests — the tests were written within the same paradigm. The test suite is also wrong. Nobody wrote a test for "are we solving the right problem?" Nobody ever does. It's not in the test plan template.

Defense: empiricism. Test designs against reality. Prototype. Watch users. Iterate on evidence. Ask yourself: when did you last change a fundamental decision because of something a user did? If the answer is "never," you are not an expert. You are a paradigm with a pulse.

Divorce of design. Designers, builders, and users become separated. Each handoff loses knowledge, accountability, and feedback speed. The designer designs in a vacuum. The builder builds from a document. The user suffers both. Nobody connects the dots because the dots are in different departments that report to different VPs who are in different meetings.

"The designer who does not build, and the builder who does not use, are both crippled."

Epistemology, not career advice. Don't build → don't know if design works. Don't use → don't know what "works" means. Each handoff is lossy compression. Information is lost. Accountability is diffused. Nobody is wrong. Everyone is slightly less right. The sum of partial correctness is not correctness. It is a bug report nobody fully owns.

Wright brothers: designed, built, flew. Bad decision showed up that afternoon. Loop: hours. Modern software: months. Designers write specs. Implementers write code. Users file bugs. Each step loses information. The bug report is a shadow of the experience. The code is a shadow of the spec. The spec is a shadow of the intent. By the time the user suffers, the original decision has been through three lossy encodings and nobody can trace it to its source. Nobody is accountable. The process worked. The product didn't.

The architect who never codes designs abstractions elegant on paper and unbuildable. The developer who never meets users builds features technically impressive and functionally useless. Each is doing their job as defined. The definition is wrong. The job is making something that works for the user. Everything else is overhead. Most of what we call process is overhead that has been institutionalized into job descriptions.

Fix: shorten the loops. Designers build. Builders use. Users in the room. If your users aren't in the room, you're not designing for them. You're designing for your idea of them. Your idea is wrong. The structures that prevent this benefit the people who could change them. That is why they persist. That is why every part of this series sounds obvious and almost no organization does any of it.

Connects to Part 3. The designer needs protection — and contact with reality. Protected without feedback is marooned. Marooned designers produce marooned designs. The organization calls it "vision." It is isolation. Isolation produces coherence with the wrong world. The design is beautiful. The user is baffled. Everyone did their job.


← Part 5 · Part 1 · Part 2 · Part 3 · Part 4 · Part 7 →

Fred Brooks, The Mythical Man-Month (1975, Anniversary Ed. 1995), No Silver Bullet (1986), The Design of Design (2010). Brooks & Blaauw, Computer Architecture: Concepts and Evolution (1997).

Brooks's principles apply beyond software. Conceptual integrity, the one-mind rule, the empiricist method — these are engineering principles that hold across any designed system. The building, the organization, the codebase, the protocol. The medium changes. The principles don't. That is the definition of engineering: principles that hold across domains.

The expert's greatest vulnerability is their own expertise. What worked in the old paradigm is precisely what fails in the new one. The expert is the last to see the shift because they are the most invested in the old way.

Brooks on Software Design Series: build, test, iterate

If we can't think our way to a correct design, what do we do? Build, test, iterate. The scientific method, applied to the glorious mess of making software.

designfred-brooksempiricismprototypingconstraints

The rational model is wrong. What's the alternative? Science. Not computer science — actual science. Hypothesize, test, learn, repeat. You are not designing a system. You are running experiments on your own ignorance. The experiments will hurt. That's how you know they're working.

"I am a hard-core empiricist, in design as in science. I don't believe we can think our way to a correct design; we must build, test, and iterate."

Definitions

Empiricist method. You cannot think your way to a correct design. Build, test, learn, iterate. If this sounds obvious, ask why your team spent three months debating an architecture nobody has prototyped. The debate felt productive. It was not. It was comforting. Comfort is not progress.

Not the Brooks of The Mythical Man-Month. Younger Brooks believed planning — "plan to throw one away" meant doing the rational process twice. Older Brooks: the first plan was never going to be right. No analysis would have fixed it. The only path runs through being wrong. Intelligence is knowing the design is wrong. Wisdom is shipping anyway to find out why. Seniority is having done this enough times to stop arguing about it.

Contrast Wirth's stepwise refinement (1971): decompose, refine until trivial. Works when you already understand the problem. Brooks: you never do. Decomposition emerges through building and testing. Wirth's method: well-understood problems. Brooks's method: everything else. Which is most of what you get paid for. The easy problems were automated decades ago.

Six steps: study domain → design (knowing it's wrong) → prototype → test with real users → iterate → build incrementally. Notice "argue about it in Slack" is not a step. Notice "write a design doc and never revisit it" is not a step. Notice "get sign-off from seven stakeholders" is not a step. The steps are hard. That's why they're skipped.

Prototype. A concrete version built to be tested and discarded. Not a draft. Not "the MVP we'll refactor later." A question posed to reality. Reality answers. Reality is usually right. Reality is also usually impolite about it.

"The prototype is the pivot of the design process. It makes ideas concrete and thereby falsifiable. A prototype that fails teaches more than a specification that pleases."

Specifications are unfalsifiable. Nobody ever looked at a spec and said "this won't work." They looked at a spec and said "looks good" — which is worse. "Looks good" means "I haven't found the problem yet." Only running code can be wrong in a way that teaches. Only a crash tells you where the bridge was weak. Specifications don't crash. That's the problem.

Formal methods. Proving programs correct by deduction. Works for small modules. Cannot scale. The mathematicians disagree. The mathematicians don't ship software on deadlines with changing requirements.

"Formal methods — proving programs correct — represent rationalism's last stand in software. They work in principle for small, well-specified modules. They cannot scale to large, complex, evolving systems. No other design discipline even attempts formal correctness proofs. Architects do not prove buildings will stand; they build them and test them."

The rationalist dream — correct by construction — survives only in CS departments and grant proposals. Every other discipline abandoned it centuries ago. Bridge builders test. Aircraft engineers test. The people who build things that kill you if they fail? They test relentlessly. The people who build things that lose your data? They debate formal verification on Hacker News.

No Silver Bullet (1986): no breakthrough eliminates essential difficulty. Computer Architecture (1997): even ISAs evolved through trial and error. Hamming: "The purpose of computing is insight, not numbers." Designing is insight, not specifications. The spec is the fossil. The prototype is the living thing. Fossils are evidence. They are not alive.

Constraints

Constraints as friends. No constraints = no criteria for excellence. Constraints make the problem solvable. "Build anything" is not a brief. It is a cry for help. It is also why your last "greenfield" project was harder than the legacy one.

Infinite possibility paralyzes. Clear constraints — budget, schedule, weight, power — create a defined field. Creativity: elegant solutions within boundaries. Hoare: "Premature optimization is the root of all evil." Brooks goes further. Constraints are not deferred evils. They are the conditions that make design possible. Without walls, you're not in a room. You're in a void. People in voids don't design. They drift.

"When you specify something to be designed, tell what properties you need, not how they are to be achieved."

Clients confuse requirements with implementation. "Use React" is not a requirement. "Renders at 60fps" is. The how is the designer's problem. The what is the client's. Both need to learn which is which. This learning takes years. It cannot be shortcut by a requirements-gathering workshop.

"The hardest part of design is deciding what to design. The chief service of a designer is helping clients discover what they really want."

Every hour clarifying saves ten building the wrong solution. This ratio is remarkably stable across industries, technologies, and decades. It is either a law of nature or evidence that we are all, collectively, terrible at knowing what we want until we see it. Either way, budget for clarification. It's cheaper than rework. Nobody budgets for it.

Cautionary tale: helicopter project added "fly across the Atlantic" as a final requirement. Contradicted every constraint. But it was documented. The rational model treats all documented requirements as valid. No defense against absurdity. The helicopter was never built. The requirement lived forever in the document. Some say it still flies there, crossing the Atlantic on PDF pages, untroubled by physics.

User models

User model. Write down who the user is, what they know, what they need. It will be wrong. Wrong and precise beats vague. "The user is a domain expert who uses the command line daily" is wrong and useful. "The user wants a good experience" is correct and worthless. One you can test against. The other is a fortune cookie.

"Better a precise model, even if wrong, than a vague one. A precise model exposes its assumptions and invites correction; a vague one is unfalsifiable and thus unhelpful."

Write nothing: everyone fills in their own model, nobody disagrees. Explicit → testable → correctable. Empiricism applied to the most important unknown. UX designers call these personas. Engineers call them "that thing we should have written down six months ago." Both are right. The persona would have prevented the rewrite.


← Part 4 · Part 1 · Part 2 · Part 3 · Part 6 → · Part 7

Fred Brooks, The Mythical Man-Month (1975, Anniversary Ed. 1995), No Silver Bullet (1986), The Design of Design (2010). Brooks & Blaauw, Computer Architecture: Concepts and Evolution (1997).

Brooks's principles apply beyond software. Conceptual integrity, the one-mind rule, the empiricist method — these are engineering principles that hold across any designed system. The building, the organization, the codebase, the protocol. The medium changes. The principles don't. That is the definition of engineering: principles that hold across domains.

The empiricist method is not about not planning. It is about not trusting the plan. The plan is a hypothesis. The prototype is the experiment. The experiment either confirms the hypothesis or teaches something the plan missed.

Brooks on Software Design Series: protect the designer

If conceptual integrity requires one mind, you must protect that mind from the organization. Real authority, organizational backing. Or: how to keep your best designer from quitting.

designfred-brookssystem-360interface-designarchitecture

One mind must control the design. How do you protect that mind? The organization wants features. The designer wants coherence. One of them has to win. The organization has more people. The designer needs better defenses.

Organizations generate feature requests, compatibility demands, and stakeholder preferences. Each is reasonable alone. Together, they destroy coherence. The designer who says no needs protection, or the no won't stick. "No" costs political capital. "Yes" gets you promoted. Guess which one happens more.

Definitions

Interface integrity. The interface is the system for the user. It must have conceptual integrity above all else. The backend can be a disaster. The user should never know. This is the entire job of API design.

"For the user interface, conceptual integrity is even more essential. The interface is the system for the user. If the interface has multiple personalities, the user must learn each one, must decide which to use when, and will be confused by their inconsistencies."

You can distribute the backend across a hundred microservices on five clouds. You cannot distribute the user's mental model. Mental models don't shard. They don't load balance. They don't fail over. They just break.

The interface owner needs veto power. Real veto — not advisory. Advisory means the VP overrides the designer. VP wins. User loses. Feature ships. Nobody uses it. Everyone is confused. The VP moves to a new role. The designer inherits the mess. This is not hypothetical. This is Tuesday.

Architecture/implementation separation. Small team defines what. Large team builds how. Roles distinct. Staffed differently. The architecture team thinks. The implementation team does. Both are essential. Only one gets protected.

"The architecture must be separated from implementation. This was the key organizational insight of System/360: a small architecture team defines what the machine is; a large implementation team builds it. The architecture team must be protected; the implementation team must be coordinated."

Architecture team: protection from external pressure. Implementation team: coordination across contributors. Conflate them: nobody does either well. This is your startup's "everyone is full-stack" model. It works until it doesn't. Then you hire architects and call it "maturing." Then the architects complain they have no authority. The cycle is predictable. Nobody reads Brooks. The cycle continues.

The System/360 pattern

"On System/360, a small team — Brooks, Amdahl, Blaauw — controlled the architecture. We had the authority to say no. More importantly, we were protected from the organizational forces that dilute design: feature requests from field sales, compatibility demands, performance optimizations that compromise clean abstraction."

Two things mattered. Authority to say no. Protection to exercise it. Field sales wanted features for customers. Engineering wanted optimizations. Customers demanded compatibility. Each reasonable alone. Together: destroyed coherence. The team had power — and organizational backing. Without backing, power is just a loudly stated opinion that gets overruled in the next steering committee.

David Parnas reached the same conclusion. Information hiding (1972): modules conceal design decisions from each other. Cannot do this with a committee. Hiding requires one mind to decide what to expose and bury. A committee exposes everything and hides nothing, which is also how it makes decisions. The meeting minutes are public. The reasoning is not.

Every project needs a design owner. One person. Reviews every interface, abstraction, user-visible decision. Authority to say no without escalation. Hard role: hold the entire system in your head, have taste, say no repeatedly — and be trusted. These people are underpaid relative to their value and over-stressed relative to their support. They know this. They stay anyway. That is the only reason your system still works.

The industry sort of does this. Architects, tech leads, staff engineers. Rarely with enough separation to say no to the VP. We don't train, hire, or protect for this role. Then we wonder why systems feel like patchwork quilts. We built the quilter's guild and asked why nobody weaves.

"Plan to throw one away; you will, anyhow." — The Mythical Man-Month, 1975

Accept the first version will be wrong. If you're not embarrassed by version one, you shipped too late. Bridges to Part 4.

"The building of a design, indeed, is the forcing of the will of one upon the stuff of the world."

Design is not consensus. It is imposition. Coherence forced onto a medium with no opinion. An act of authority. Does your organization have the nerve? Most don't. Most call a meeting. The meeting schedules a follow-up. Nothing is forced. Nothing coheres.


← Part 2 · Part 1 · Part 4 → · Part 5 · Part 6 · Part 7

Fred Brooks, The Mythical Man-Month (1975, Anniversary Ed. 1995), No Silver Bullet (1986), The Design of Design (2010). Brooks & Blaauw, Computer Architecture: Concepts and Evolution (1997).

Brooks's principles apply beyond software. Conceptual integrity, the one-mind rule, the empiricist method — these are engineering principles that hold across any designed system. The building, the organization, the codebase, the protocol. The medium changes. The principles don't. That is the definition of engineering: principles that hold across domains.

Protecting the designer is not about giving them autonomy. It is about giving them authority. Autonomy without authority is frustration. Authority without protection is theater.

Brooks on Software Design Series: great designers, not great processes

Great designs come from great designers, not great processes. Organizations must grow, protect, and retain design talent. Your agile coach can't save you. Only taste can.

designfred-brooksgreat-designersmentorship

Parts 1-6: conceptual integrity, one mind, protection, rational model critique, empiricism, forces against it. The dominoes all fall one way. The conclusion is uncomfortable. The industry has spent fifty years avoiding it.

Conclusion: great designs come from great designers, not from great processes. Your Jira workflow has never designed anything. Your RFC template has never had an idea. Your retrospective has never produced insight — it produced action items. Action items are not designs. They are evidence that a meeting occurred.

This runs through all four Brooks books. The Mythical Man-Month (1975): the chief architect. No Silver Bullet (1986): tools alone cannot produce better design. Computer Architecture (1997, with Blaauw): every great machine traces to one or two minds. The Design of Design (2010): the organizing thesis. Four books, one argument. The industry read them and built SAFe anyway. Brooks is patient. Brooks is dead. The argument remains.

Definitions

Process vs. talent. Process prevents bad design from shipping. It cannot produce good design. Process is a seatbelt. It will not drive the car. It will not choose the destination. It will only prevent some of the damage when the driver — who is still necessary — makes a mistake.

"Great designs come from great designers. Process can make a good design better; it cannot make a bad designer good."

Modern organizations invest in process: code review, design review, RFCs, ADRs, retrospectives. Supporting structures, not substitutes. Process raises the floor. Talent raises the ceiling. Conflating them is the central mistake. You cannot review your way to brilliance. You can only review your way to adequacy. Adequacy at scale is still adequacy. The world has enough adequate software. Nobody remembers who built it.

Knuth took a decade for TeX — one mind, one vision, one beautifully typeset result. Alan Kay: best software is "a single person's vision carried through." Hoare (1980 Turing lecture): "the most dangerous error is believing better tools can replace individual insight." Licklider (1960): couple human intuition with machine power. The generation agreed. Process supports. Talent creates. This is not controversial among people who have built great things. It is only controversial among people who manage them and wish the management were sufficient.

Process sequencing. Designer designs first. Process applies second. Order matters. Skip this and you get compromise dressed in review comments, which is still compromise.

"The trick is to hold process off long enough to permit great design to occur, so that the lesser issues can be debated once the great design is on the table — rather than smothering it in the cradle."

Committee before design = compromise. Design before committee = review finds weaknesses without diluting vision. Most organizations reverse it. Committee first, compromise second, individual execution third. Integrity lost at step one. The rest is expensive theater with good catering.

Dual ladder. Technical and managerial paths: parallel, equal compensation, equal respect. Every company claims to have this. Almost none do. You can tell by asking one question: does your Distinguished Engineer report to a VP who controls their compensation? If yes, the ladder is a label. Labels are cheap. That's why everyone has one.

"The dual ladder is everywhere espoused and nowhere practiced. The managerial ladder remains the path to power, prestige, and pay. The technical ladder is too often a consolation prize."

Make it real: equal pay (DE = VP), visible strategic voice, budget protection, staff with exemplars. Otherwise it's a parking lot for brilliant people you don't want to lose but don't want to empower. They know. They're brilliant.

Growing designers

Recruit for design sense. Look at what the candidate built. Past performance predicts. Whiteboard interviews predict nothing except whiteboard interview performance, which is a skill nobody needs after the interview.

"Hire for demonstrated design ability, not for interview prowess. Look at what the candidate has built. Does it show conceptual integrity? Do the abstractions make sense?"

Mentor. Taste transmits through apprenticeship. Not docs. Not talks. Not "lunch and learns." Through working next to someone who has it and watching how they think.

"Design judgment is not taught; it is caught. The master-apprentice relationship is how taste transmits. A junior absorbs judgment: what to optimize for, what to ignore, when to fight."

Rotate. Breadth builds pattern recognition. One domain, one technology, one problem space = one pattern. You cannot synthesize from one example any more than you can learn a language from one word.

"The designer who has worked in only one domain has a narrow base. Breadth builds the repertoire from which great designs are synthesized."

Protect from managing. The reward for great design is promotion to management. This is like rewarding a great chef by making them run the restaurant. Different skills. Different outcomes. The chef stops cooking. The food gets worse. Everyone notices except the person who promoted the chef.

"The proper office of the manager is to protect his great designers from managing."

Highest-value activity is designing. Every hour in meetings is lost design work. Count your best designer's meetings. More than five per week? You're burning your best asset for scheduling convenience. The calendar is a furnace. Your talent is the fuel.

Esthetics, exemplars, cases

Esthetics. "Clean" and "elegant" are not metaphors. They denote fitness, coherence, economy — judged by those with taste. Taste is real. Taste is testable. Taste is not subjective in the way that matters. You can be wrong about elegance. People wrong about elegance are usually wrong about estimates too, for the same reason: they can't see the shape of the thing.

"We speak of 'clean' machines, 'elegant' languages, 'beautiful' proofs. These are not mere metaphors. They denote a real property: the fitness, coherence, and economy of a design."

Develop taste: study styles, practice another's, revise for consistency, hire for demonstrated taste. There is no shortcut. Taste is accumulated judgment. Judgment is accumulated mistakes. Mistakes are accumulated by doing. Go do. Make mistakes. Learn to see them before you make them. That's taste.

Exemplar gap. Software designers don't study exemplars like architects study buildings. Architects visit buildings. Composers study scores. Most software designers have not read the source code of the systems they admire. They use them. They don't study them. That's like becoming a novelist by only reading book reviews. You'll learn what's popular. You won't learn how sentences work.

"Architects study buildings; composers study scores; writers study books; painters visit museums. Software designers too rarely study existing designs, and when they do, it is to learn how to use the system, not why it was designed as it was."

Great designs contain lessons not conveyable in principles — only in the decisions that produced a result. Read TeX's source. Read the Unix kernel. Read anything by people who thought harder than you. It's all there, waiting. Almost nobody does it. The people who do become the designers everyone else envies.

Case studies. System/360, OS/360, beach house, kitchen, book design. Brooks uses his own work. Use yours. Study your wins and your disasters. Both have more to teach than any methodology book.

"Why an 8-bit byte? Why 32-bit words? Why separate architecture from implementation? Each decision made by a small group, debated intensely, then locked. This discipline — decide, commit, don't revisit — ships coherent architecture."

Second-system effect. The most dangerous system is the second. Version one succeeds. Version two gets every feature cut from version one. Version two sinks. Everyone is surprised. Nobody should be. This has been documented since 1975. It still happens. It will happen on your next project unless you actively prevent it.

"The second is the most dangerous system a designer ever builds. Having succeeded with the first, he loads the second with every feature he omitted. The result is a bloated, over-budget, late disaster." — The Mythical Man-Month, 1975

OS/360 was the example. JCL was the scar: "the worst computer language ever devised, a triumph of committee design over conceptual integrity." Brooks wrote this about his own project. That is intellectual honesty. Most of us blame the tools, the timeline, the market. He blamed himself and wrote four books. The least we can do is read one of them.

"I have designed in five media: computer architecture, software, houses, books, and organizations. The principles of design are independent of the medium."

Same principles govern a computer architecture and a kitchen renovation. If your design principles can't survive contact with a kitchen, they weren't principles. They were platform-specific habits dressed up as wisdom. The test of a principle is whether it holds when the medium changes. Brooks tested his across five media. How many have you tested yours across?

Does your organization take these claims seriously? Design authority? Empirical discovery? Growing designers? Most answer no. Brooks spent six decades explaining the mistake. We spent six decades nodding and building process instead. The book is on the shelf. It's short. You have time. The question is whether you have the nerve.


← Part 6 · Part 1 · Part 2 · Part 3 · Part 4 · Part 5

Fred Brooks, The Mythical Man-Month (1975, Anniversary Ed. 1995), No Silver Bullet (1986), The Design of Design (2010). Brooks & Blaauw, Computer Architecture: Concepts and Evolution (1997).

Brooks's principles apply beyond software. Conceptual integrity, the one-mind rule, the empiricist method — these are engineering principles that hold across any designed system. The building, the organization, the codebase, the protocol. The medium changes. The principles don't. That is the definition of engineering: principles that hold across domains.

Process raises the floor. Talent raises the ceiling. The organization that invests only in process gets consistent mediocrity. The organization that invests only in talent gets inconsistent brilliance. The organization that invests in both gets both.

Brooks on Software Design Series: the waterfall is wrong

The rational model says gather requirements, design, implement, test, ship. Brooks says it is wrong and harmful. You don't know the goal at the start. Neither does your client. The Gantt chart is a lie.

designfred-brookswaterfallrational-modelco-evolution

Parts 1-3: who designs (one mind, protected). Parts 4-5: how design works. Spoiler: not the way your project plan says. The project plan was wrong before it was printed.

The dominant model is wrong. Not slightly. "Earth is flat" wrong.

Definitions

Rational model (waterfall). Gather requirements. Design. Implement. Test. Ship. Each phase finishes before the next. Proceeds logically from premises to conclusion. Makes sense on a Gantt chart. Has never once worked in reality. Yet we keep using it because it makes managers feel better.

Clean. Orderly. Wrong.

"The Waterfall Model is wrong and harmful; we must outgrow it. What is wrong is that it is an essentially rational model, and for wicked problems, the rational model is simply the wrong model."

Not "sometimes inappropriate." Wrong and harmful. It demands decisions at maximum ignorance — the beginning — and forbids revisiting them. It's like ordering dessert before you've seen the menu, eaten the meal, or confirmed you're hungry. Then acting surprised when nobody wants the tiramisu. Then blaming the tiramisu.

The model assumes the designer knows the goal at the start. False. Laughably false. Anyone who has built anything real knows this. Anyone who hasn't nods along with the Gantt chart.

Herbert Simon: design as systematic search. Goals, utility functions, constraints. Find the optimum. Theory is beautiful. Simon won a Nobel Prize. His theory still doesn't survive contact with a real client who changes their mind after seeing the first prototype. The client is not irrational. The model is wrong about when knowledge arrives.

Practice: nobody knows the goal. Not the designer. Not the client. "That's what I asked for, but that's not what I want." Every designer has heard this. It means the process is working, not failing. A client cannot articulate needs until they see something. They can spot what's wrong instantly. They cannot describe what's right before anything exists. This is not a communication failure. This is how cognition works. Seeing is knowing. Speculating is guessing.

Co-evolution. Requirements and design change each other. Designing reveals new requirements. New requirements change the design. Cycle continues until both stabilize. Or until the budget runs out. Whichever comes first. Usually the budget.

"Requirements and design co-evolve. The act of designing changes the designer's understanding of the problem. As the design emerges, the requirements change. This is not failure; it is discovery."

Requirements are not extracted like ore. They are produced through designing. Each iteration teaches. The process ends when further iteration yields diminishing returns. In practice, it ends when the PM says "we need to ship." This is also a form of diminishing returns. Just not the one Brooks had in mind.

Peter Naur, same era. Theory building (1985): a program is not its code. It is the theory its builders hold of the problem. Cannot be extracted upfront. Built through designing. When the last person who understands the system leaves, the theory leaves with them. The code remains. Nobody knows why it works. This is called "legacy." It is also called "most production systems."

Brooks and Naur: the real product is understanding. Understanding emerges through the work. Documentation is not understanding. Documentation is a fossil of understanding that was alive six months ago. The fossil is useful. It is not the animal.


← Part 3 · Part 1 · Part 2 · Part 5 → · Part 6 · Part 7

Fred Brooks, The Mythical Man-Month (1975, Anniversary Ed. 1995), No Silver Bullet (1986), The Design of Design (2010). Brooks & Blaauw, Computer Architecture: Concepts and Evolution (1997).

Brooks's principles apply beyond software. Conceptual integrity, the one-mind rule, the empiricist method — these are engineering principles that hold across any designed system. The building, the organization, the codebase, the protocol. The medium changes. The principles don't. That is the definition of engineering: principles that hold across domains.

The waterfall is not wrong because it is sequential. It is wrong because it assumes knowledge arrives before building. Knowledge arrives through building. The sequence is backwards.

Brooks on Software Design Series: one mind

Conceptual integrity requires one mind — or at most a resonant pair. Committees produce compromises, not coherence. Three people is already a negotiation.

designfred-brooksone-mindcollaborationbrooks-law

Conceptual integrity requires one mind. How do you achieve that in an organization of many? Short answer: you don't. You fight a rearguard action and hope for the best.

The design must proceed from one mind, or from a very small number of agreeing resonant minds. Most organizations read this, nod, and schedule a cross-functional alignment workshop. The workshop produces a shared document. The document has no owner. Nothing improves. This is the industry in microcosm.

Definitions

One-mind rule. The conceptual design — core abstractions, primitives, relationships, user mental model — must be controlled by one person. At most, two in genuine resonance.

"The Design of Design sharpens the earlier contention: the design must represent the vision of one designer or, at most, a pair."

In 1975, Brooks allowed implementation by teams if architecture belonged to one mind. By 2010: architecture itself can have at most two authors. At this rate, by 2045 he'd argue for half a designer. The trend line is clear. The industry is moving the opposite direction.

Resonant pair. Two designers who share a mental model so completely either can speak for the architecture. Finishing each other's thoughts. Same taste. Same instincts. Same willingness to say no to the same things.

"Two people can serve as one mind only if they are in genuine resonance — finishing each other's thoughts, sharing a mental model so deeply that each knows what the other would decide. Three cannot."

Brooks found this with Gerrit Blaauw on System/360. Rare. Most pairs are just two people who've learned which topics to avoid. Resonance is not collaboration. Resonance is one mind in two bodies. If you have to explain your decisions to your partner, you are not a resonant pair. You are coworkers.

Committee design. Multiple stakeholders add requirements. Result: accommodates everyone, satisfies no one. The platypus is nature's committee design. It works. Nobody would design it from scratch.

"Design by committee produces designs that offend no one and satisfy no one. Each member's wish list is accommodated, each objection smoothed over, until the result is a feature-laden compromise lacking any coherent vision."

Structural. Each new mind adds assumptions. Reconciliation produces compromises. Each compromise chips at integrity. The committee made it safer, not better. Safety is the enemy of coherence. Never confuse "nobody objected" with "this is good." Nobody objects to mediocre food either. That's why most restaurants are forgettable.

Teams help with requirements (more edge cases), design space exploration (brainstorming), and implementation (parallel work). The conceptual design must belong to one person. Teams exist to execute, not to design. Meetings exist for reasons less clear.

Costs

Learning cost. Transferring a vision to n people takes n × l effort. For ten people, more time teaching than designing. Onboarding takes six months and nobody questions it because everyone has forgotten it could be otherwise.

Communication cost. n people = n(n−1)/2 paths. Five: 10. Ten: 45. One hundred: 4,950. Overhead grows quadratically. Throughput does not. Mathematics is cruel and indifferent to your standup cadence.

Change control cost. More contributors = harder changes. The design calcifies. A living vision becomes a frozen document nobody fully owns. Then someone says "we should refactor" and the cycle begins again, with a larger committee this time.

Brooks's Law. Adding people to a late project makes it later. Same math applies to design. Adding designers dilutes. Your manager's solution to a late project is more people. Your manager's solution is wrong. Your manager is applying a linear fix to a quadratic problem. Mathematics will win.

"Adding manpower to a late software project makes it later." — The Mythical Man-Month, 1975

Essential complexity. Inherent in the problem. Cannot be eliminated. Like death, taxes, and npm dependencies.

Accidental complexity. Imposed by tools and methods. Can be reduced. This is what you spend most of your time fighting. The rest is meetings about complexity reduction strategies you'll never implement.

From No Silver Bullet (1986). The rational model treats all complexity as accidental. Brooks: essential complexity remains. Conceptual integrity manages it — one voice in the design, a hundred hands building. Without the one voice, the hundred hands build a hundred different things and call it a microservices architecture.

The rule is structural, not preferential. A design is interdependent decisions. Different people make different decisions — constraints conflict — system acquires multiple personalities — user suffers. "Design by community" is incoherent. Output is a negotiated settlement. Settlements govern societies. They cannot produce coherent software. The EU is not an API. If it were, it would have seventeen conflicting ways to authenticate.

Conway's Law (1968): organizations produce designs that copy their communication structures. Read with Brooks: coherent design needs coherent design organization. One mind. A committee's communication graph is complete. Its output will be, too. This is why your microservices map exactly to your org chart. It's not supposed to. It's supposed to map to the problem. The problem doesn't care about your reporting lines.

Every project needs a design owner. Reviews every interface. Says no without escalation. Job: conceptual integrity. If this person doesn't exist on your project, they are not you. Find them. Or become them. Either way, you're currently in trouble you haven't noticed yet.


← Part 1 · Part 3 → · Part 4 · Part 5 · Part 6 · Part 7

Fred Brooks, The Mythical Man-Month (1975, Anniversary Ed. 1995), No Silver Bullet (1986), The Design of Design (2010). Brooks & Blaauw, Computer Architecture: Concepts and Evolution (1997).

Brooks's principles apply beyond software. Conceptual integrity, the one-mind rule, the empiricist method — these are engineering principles that hold across any designed system. The building, the organization, the codebase, the protocol. The medium changes. The principles don't. That is the definition of engineering: principles that hold across domains.

The one-mind rule is not about ego. It is about attention. One mind can hold the entire design. Two minds can hold it if they are in resonance. Three minds produce a committee. Committees produce compromises.

Brooks on Software Design Series: conceptual integrity

The system feels like one mind designed it. The most important property of any designed thing.

designfred-brooksconceptual-integrity

In 2010, Fred Brooks published The Design of Design. It is his best book and his least read.

The argument: conceptual integrity is the most important property of any designed system. Everything else serves it.

"I will contend that conceptual integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas." — The Mythical Man-Month, 1975

The tradeoff: omit useful features to preserve coherence. Say no to good ideas that don't fit.

Definitions

Conceptual integrity. The system feels like one mind designed it. The user forms one mental model. They predict behavior in new situations.

Orthogonality. One way to do each thing. No overlapping concepts. Non-redundancy.

Propriety. Nothing unnecessary.

"The essential skill of the designer is saying no — repeatedly, to smart people with good arguments — and having the authority to make it stick. Every added feature is a subtraction from all existing features, for it adds complexity without corresponding benefit."

Feature cost is multiplicative. Each new thing makes every existing thing harder to find, learn, and use.

Generality. No arbitrary limits. Primitives compose to handle unanticipated cases. Generality without orthogonality is a kitchen sink. Orthogonality without generality is a toy.

Brooks was not alone. Knuth: programs should be readable by humans. Dijkstra: elegance prevents bugs — transparent structure hides nothing. Kernighan and Plauger on clarity vs. muddle in The Elements of Programming Style (1974). Wirth: Pascal as deliberate integrity — one mind, one language. Lampson: the Alto's GUI worked because one designer decided what to leave out.

Brooks's contribution: the structural precondition. One mind. In Computer Architecture (1997, with Blaauw), he showed this governed even ISAs, where orthogonality of operations and addressing modes was the explicit goal.

Proof

"Reims Cathedral has conceptual integrity. Built over eight generations of architects, each stuck to the original plan. The result is a unified work, coherent in every detail. Most cathedrals are not like this; their conflicting concepts produce architectural chaos."

Reims worked because the original architect's plan had authority that outlived him. Two centuries of successors submitted. It should not exist. That it does is the evidence.

Most cathedrals: Gothic nave, Renaissance facade, Baroque chapel on Romanesque transept. Each generation had a vision. Nobody had authority. Chaos.

Software systems are cathedrals built over decades. Unix: Thompson and Ritchie, one resonant pair. Lisp: McCarthy, one mind. Go: Thompson, Pike, Griesemer. The Macintosh: Jobs. These are Reims. Thompson: "One of my most productive days was throwing away 1,000 lines of code."


Part 2 → · Part 3 · Part 4 · Part 5 · Part 6 · Part 7

Fred Brooks, The Mythical Man-Month (1975, Anniversary Ed. 1995), No Silver Bullet (1986), The Design of Design (2010). Brooks & Blaauw, Computer Architecture: Concepts and Evolution (1997).

Brooks's principles apply beyond software. Conceptual integrity, the one-mind rule, the empiricist method — these are engineering principles that hold across any designed system. The building, the organization, the codebase, the protocol. The medium changes. The principles don't. That is the definition of engineering: principles that hold across domains.

Conceptual integrity is not about consistency. It is about coherence. A system can be consistently bad. A coherent system feels like one mind designed it, even if many hands built it.

Temporal vs DBOS for Go: two paths to durable execution

Temporal gives you a battle-tested orchestration platform with event-sourced replay. DBOS gives you durable workflows in a single Postgres-backed binary. Which one fits your Go stack?

gotemporaldbosdurable-executionworkflowspostgres

Durable execution — the guarantee that a workflow runs to completion even if the process crashes, the machine dies, or the network partitions — is becoming table stakes for backend systems. Two projects in the Go ecosystem take fundamentally different paths to the same destination: Temporal and DBOS.

Temporal is the incumbent. Separate infrastructure, event-sourced replay, a mature SDK that replaces Go's concurrency primitives with deterministic equivalents. DBOS is the challenger. PostgreSQL-native, checkpoint-based, a thin layer over your existing database that gives you durable workflows, queues, and scheduling without deploying anything new.

Choosing between them is not about which one is "better." It is about which architecture matches your operational constraints.

Temporal: the platform

Temporal is a distributed orchestration engine. You deploy a Temporal server (or use Temporal Cloud), and your Go workers connect to it. Workflows are deterministic functions that use Temporal's SDK replacements for Go primitives: workflow.Sleep() instead of time.Sleep(), workflow.Go() instead of go, workflow.Channel() instead of chan, workflow.Selector() instead of select.

The determinism constraint is the defining tradeoff. Because Temporal replays workflow code against event history to rebuild state after a crash, your workflow code must produce the same sequence of decisions given the same history. This means no time.Now(), no random numbers, no direct database calls inside a workflow. Non-deterministic work goes into Activities — functions that Temporal calls outside the replay sandbox, with configurable retries and timeouts.

The Go SDK (v1.44.0 as of mid-2026) is mature. Here is a user onboarding workflow — charge a customer, provision services, send a welcome email — with configurable retries and timeouts:

// Workflow: deterministic coordination logic
func OnboardingWorkflow(ctx workflow.Context, input OnboardingInput) error {
    ao := workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:    time.Second,
            BackoffCoefficient: 2.0,
            MaximumAttempts:    3,
        },
    }
    ctx = workflow.WithActivityOptions(ctx, ao)

    // Step 1: Charge payment
    var chargeID string
    if err := workflow.ExecuteActivity(ctx, ChargeCustomer, input.CustomerID, input.Amount).Get(ctx, &chargeID); err != nil {
        return fmt.Errorf("charge failed: %w", err)
    }

    // Step 2: Provision resources
    var resourceIDs []string
    if err := workflow.ExecuteActivity(ctx, ProvisionResources, input.Plan, input.Region).Get(ctx, &resourceIDs); err != nil {
        // Compensation: refund if provisioning fails
        if refundErr := workflow.ExecuteActivity(ctx, RefundCharge, chargeID).Get(ctx, nil); refundErr != nil {
            return fmt.Errorf("provision failed, refund also failed: %w", refundErr)
        }
        return fmt.Errorf("provision failed, refunded: %w", err)
    }

    // Step 3: Send welcome email (fire-and-forget with short timeout)
    emailCtx := workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
        StartToCloseTimeout: 10 * time.Second,
        RetryPolicy:         &temporal.RetryPolicy{MaximumAttempts: 5},
    })
    if err := workflow.ExecuteActivity(emailCtx, SendWelcomeEmail, input.Email, resourceIDs).Get(ctx, nil); err != nil {
        workflow.GetLogger(ctx).Warn("welcome email failed, not fatal", "error", err)
    }

    return nil
}

// Activity: non-deterministic code — safe to call external APIs
func ChargeCustomer(ctx context.Context, customerID string, amount int) (string, error) {
    return billingClient.CreateCharge(ctx, customerID, amount)
}

Each ExecuteActivity call checkpoints progress in Temporal's event history. If the worker crashes after ChargeCustomer succeeds but before ProvisionResources starts, the workflow resumes from the charge — it does not double-charge. The compensation logic (refund on provision failure) is explicit in the workflow code, not buried in infrastructure config.

Recent SDK additions include Standalone Activities (durable job processing without a parent workflow), Worker Versioning for safe deploys of workflow code changes, and Nexus for cross-service orchestration. The tooling — workflowcheck for static analysis of determinism violations, replay tests for verifying workflow code against historical event histories — helps catch issues before they reach production.

Running Temporal means running a Temporal server. This is a non-trivial operational commitment. The server requires a database (MySQL or PostgreSQL), an Elasticsearch instance for visibility, and a multi-service deployment for production. Temporal Cloud exists, but it is a paid service with its own pricing model. For teams already running significant infrastructure, this is manageable. For small teams or single-service deployments, it is overhead that must be justified.

DBOS: the embedded alternative

DBOS takes the opposite approach. There is no external server to deploy. You add dbos-transact-golang to your Go module, point it at a PostgreSQL database, and you have durable workflows. All state — inputs, outputs, step progress, sleep timers, queue positions, notifications — is checkpointed in Postgres. If your process crashes, on restart all workflows automatically resume from the last completed step.

The programming model is simpler than Temporal's because there is no replay. Workflows are regular Go functions. Non-deterministic operations go into steps via dbos.RunAsStep(). Here is the same user onboarding workflow in DBOS:

func OnboardingWorkflow(ctx dbos.DBOSContext, input OnboardingInput) error {
    // Step 1: Charge payment (wrapped — output is checkpointed)
    chargeID, err := dbos.RunAsStep(ctx, func(ctx context.Context) (string, error) {
        return billingClient.CreateCharge(ctx, input.CustomerID, input.Amount)
    }, dbos.WithStepName("charge"), dbos.WithStepMaxRetries(3))
    if err != nil {
        return fmt.Errorf("charge failed: %w", err)
    }

    // Step 2: Provision resources
    resourceIDs, err := dbos.RunAsStep(ctx, func(ctx context.Context) ([]string, error) {
        return provisioningClient.CreateResources(ctx, input.Plan, input.Region)
    }, dbos.WithStepName("provision"), dbos.WithStepMaxRetries(3))
    if err != nil {
        // Compensation: refund
        dbos.RunAsStep(ctx, func(ctx context.Context) (any, error) {
            return nil, billingClient.Refund(ctx, chargeID)
        }, dbos.WithStepMaxRetries(5))
        return fmt.Errorf("provision failed, refunded: %w", err)
    }

    // Step 3: Send welcome email (fire-and-forget, best-effort)
    dbos.RunAsStep(ctx, func(ctx context.Context) (any, error) {
        return nil, emailClient.SendWelcome(ctx, input.Email, resourceIDs)
    }, dbos.WithStepName("email"), dbos.WithStepMaxRetries(5))

    return nil
}

Each RunAsStep checkpoint its return value to Postgres. If the process crashes after charging but before provisioning, the workflow resumes from the next step — the charge is never duplicated. The compensation path is explicit in application code. No separate activity registration, no SDK replacement for time.Now(), no sandbox. Just Go functions with step-wrapped I/O and Postgres as the durability layer.

This simplicity extends across the feature set. Durable queues give you concurrency control and rate limiting without a message broker:

queue := dbos.NewWorkflowQueue(ctx, "task_queue",
    dbos.WithWorkerConcurrency(5),
    dbos.WithRateLimiter(&dbos.RateLimiter{
        Limit: 100, Period: 60 * time.Second,
    }))

Durable sleep persists wake-up time to Postgres — a workflow can dbos.Sleep(ctx, 48*time.Hour) and survive process restarts across those two days. Cron scheduling is a single option on workflow registration. Send/Recv provides durable notifications between workflows with exactly-once semantics. These are not add-ons. They are part of the same library.

The cost is scale. DBOS is bounded by a single Postgres instance (or cluster). Temporal scales horizontally across workers and partitions. For most applications this distinction is theoretical — a well-tuned Postgres instance handles millions of workflow steps — but for the largest deployments, Temporal's distributed architecture is the right call.

Setup side by side

The difference in operational commitment is visible even at the setup stage.

Temporal requires worker registration, activity registration, and a running Temporal server. A minimal worker binary looks like this:

func main() {
    c, _ := temporalclient.NewClient(temporalclient.Options{})
    defer c.Close()

    w := worker.New(c, "onboarding-queue", worker.Options{})
    w.RegisterWorkflow(OnboardingWorkflow)
    w.RegisterActivity(ChargeCustomer)
    w.RegisterActivity(ProvisionResources)
    w.RegisterActivity(RefundCharge)
    w.RegisterActivity(SendWelcomeEmail)

    if err := w.Run(worker.InterruptCh()); err != nil {
        log.Fatal(err)
    }
}

Workflows, activities, and the worker are three distinct concerns. You register each activity by name. Temporal's type system enforces the separation — workflow.Context for workflows, context.Context for activities — which prevents you from accidentally calling non-deterministic code inside a workflow.

DBOS collapses this into a single lifecycle block:

func main() {
    ctx, _ := dbos.NewDBOSContext(context.Background(), dbos.Config{
        AppName:     "onboarding",
        DatabaseURL: os.Getenv("DBOS_SYSTEM_DATABASE_URL"),
    })

    dbos.RegisterWorkflow(ctx, OnboardingWorkflow)
    queue := dbos.NewWorkflowQueue(ctx, "onboarding-queue",
        dbos.WithWorkerConcurrency(10))

    if err := dbos.Launch(ctx); err != nil {
        log.Fatal(err)
    }
    defer dbos.Shutdown(ctx, 30*time.Second)
}

No separate activity registration. No external server URL. The database connection string is the only infrastructure dependency. The Launch call starts the worker loop, and Shutdown drains in-flight work gracefully.

The difference is not just lines of code. It is the number of things you need to have running before the code works. Temporal needs a server, a database, and your worker. DBOS needs a database and your binary. For a team evaluating durable execution for the first time, that gap determines whether they ship or shelve.

The determinism constraint, concretely

The sharpest practical difference is how each system handles non-deterministic Go code inside a workflow. Consider a workflow that needs the current time:

Temporal — this fails:

// INSIDE A WORKFLOW — WRONG
now := time.Now() // Non-deterministic! Replay will see a different value.

Temporal enforces this at multiple levels. The SDK documentation lists the allowed API surface inside workflows. The workflowcheck linter catches violations at build time. The replay test framework re-executes your workflow against recorded event histories and fails if the code path diverges. You must use workflow.Now() instead, which returns the timestamp from the event history rather than the system clock:

// CORRECT inside a Temporal workflow
now := workflow.Now(ctx)

DBOS — this works fine:

// Inside a DBOS workflow — fine, no replay ambiguity
now := time.Now()

Because DBOS does not replay workflow code — it checkpoints step outputs and resumes from the last checkpoint — time.Now() inside a workflow is harmless. The workflow runs once forward. If it crashes after step 3, it resumes by re-executing the workflow function from the top, but dbos.RunAsStep returns the cached output for steps 1–3 without re-executing their bodies. The non-deterministic code inside the step closure is never re-run.

The tradeoff is clear. Temporal guarantees that your workflow code produces the same decisions given the same history — a strong correctness property, paid for with a constrained programming model. DBOS guarantees that completed steps are never re-executed — a weaker property, but one that lets you write regular Go. If your workflows are straightforward chains of API calls with compensation on failure, DBOS's model is sufficient and less constraining. If your workflows contain complex branching, signals, or state machines where replay divergence would mean incorrect business outcomes, Temporal's enforcement is worth the ceremony.

Four stories

Abstract comparisons only go so far. Here are four concrete scenarios — each drawn from real architectural choices — that show how the tradeoffs play out.

Story 1: The two-person startup

Maya and Carlos are building a billing platform. Their stack is Go, Postgres, and a few Lambda functions. They need order processing to be durable — a customer signs up, a Stripe charge must succeed, a provisioning call to a third-party API must complete, and an invoice must be generated. If any step fails mid-flight, the whole thing must recover without double-charging.

They evaluate Temporal first. The programming model looks great. Then they read the deployment guide: Temporal server, MySQL or PostgreSQL, Elasticsearch for visibility, multi-service setup for production. They have two people and a managed Postgres instance. Running Temporal themselves is a non-starter, and Temporal Cloud adds a line item they cannot justify at this stage.

They try DBOS. go get github.com/dbos-inc/dbos-transact-golang, point it at their existing Postgres instance, and they have durable workflows by the end of the day. The order processing saga — three steps with compensation on failure — is 40 lines of Go. They ship within the week.

What happened: DBOS won because the operational cost of Temporal exceeded the value of its additional capabilities. For a small team, "just a library" is the right abstraction. The constraint they accepted — bounded by a single Postgres instance — is irrelevant at their scale. If they grow to need Temporal later, the workflow logic ports across because the concepts (steps, compensation, idempotency) are the same.

Story 2: The enterprise polyglot migration

FinServCo is migrating a monolith to microservices. They have teams writing Go, Java, and Python. Their core business process — account opening — spans 14 steps across 6 services: identity verification (Go), fraud check (Java), credit check (Python), account creation (Go), card issuance (Java), and welcome kit dispatch (Python). The saga must handle partial failures with compensating transactions at each step. It must be visible in a single dashboard. It must survive any individual service going down.

They prototype the saga in DBOS on a single Go service. It works. Then they realize: every non-Go step needs an HTTP wrapper, every service needs to expose a compensation endpoint, and the DBOS workflow becomes a fragile orchestration hub that must coordinate everything over the network. The simplicity they gained by embedding the workflow engine they lose in the integration layer.

They switch to Temporal. Each team writes their step in their own language as a Temporal Activity. The Go service hosts the workflow, but the activities are polyglot — Temporal's SDK handles the cross-service communication. The Temporal UI gives them a single pane of glass across all 14 steps, with retry status, input/output inspection, and stack traces on failure. When the credit check service goes down during a deploy, in-flight account openings pause and resume automatically.

What happened: Temporal won because polyglot orchestration and visibility mattered more than operational simplicity. DBOS would have worked technically, but the integration tax of wrapping every non-Go service as an HTTP endpoint would have erased the simplicity advantage. Temporal's architecture — separate server, polyglot workers, unified visibility — matched the organizational structure of the problem.

Story 3: The team already running Temporal

PlatformCo has run Temporal in production for two years. They have 40 workflow types, a dedicated infrastructure team managing the Temporal cluster, and engineers who know the SDK cold. But they are tired.

Tired of the upgrade cycles. Tired of the Elasticsearch index falling over during traffic spikes. Tired of explaining to new hires why time.Now() inside a workflow is a compile error. Tired of the determinism debugging sessions where someone accidentally closed over a map and the replay diverged three weeks later.

They start a new service — a simple notification pipeline (ingest event → enrich with user data → route to channel → record delivery). Four steps, no branching, no signals, no complex state machines. It does not need Temporal's full power. They build it in DBOS. The Temporal cluster stays for the complex 14-step account provisioning sagas. The new service ships faster, debugs easier, and has zero new infrastructure.

What happened: They did not migrate off Temporal. They stopped using it for problems that did not need it. The sweet spot for DBOS is the 80% of workflows that are linear chains of API calls with compensation. The sweet spot for Temporal is the 20% that involve branching, signals, child workflows, or cross-service coordination. The platforms coexist.

Story 4: The 72-hour approval workflow

RegTech builds a compliance review system. A case is submitted. A human must approve or reject within 72 hours. If no response, the case auto-escalates. During those 72 hours, the workflow must survive process restarts, deploys, and database failovers. After approval, the case moves to archival. After rejection, it moves to remediation.

They build it in DBOS first:

func ComplianceReview(ctx dbos.DBOSContext, caseID string) error {
    // Assign reviewer and notify
    reviewer, _ := dbos.RunAsStep(ctx, func(ctx context.Context) (string, error) {
        return assignReviewer(ctx, caseID)
    }, dbos.WithStepName("assign"))
    dbos.RunAsStep(ctx, func(ctx context.Context) (any, error) {
        return nil, notifyReviewer(ctx, reviewer, caseID)
    }, dbos.WithStepName("notify"))

    // Wait for human approval — 72h timeout, survives everything
    result, err := dbos.Recv[string](ctx, "approval."+caseID, 72*time.Hour)
    if err != nil {
        dbos.RunAsStep(ctx, func(ctx context.Context) (any, error) {
            return nil, escalate(ctx, caseID)
        })
        return nil
    }

    if result == "approved" {
        dbos.RunAsStep(ctx, func(ctx context.Context) (any, error) {
            return nil, archive(ctx, caseID)
        })
    } else {
        dbos.RunAsStep(ctx, func(ctx context.Context) (any, error) {
            return nil, remediate(ctx, caseID)
        })
    }
    return nil
}

The dbos.Recv call pauses the workflow for up to 72 hours. The wake-up time is checkpointed in Postgres. If the server restarts during those three days, the workflow resumes with the timer intact. When the reviewer's HTTP endpoint calls dbos.Send(ctx, workflowID, "approved", "approval."+caseID), the workflow wakes up and continues. No external timer service, no scheduled-job table, no cron polling.

They could build this in Temporal too — with signals and timers. The Temporal version would be equally durable and more scalable. But they would need to run a Temporal cluster for a workflow that fires a few hundred times a day and sleeps for three days each time. The infrastructure-to-business-logic ratio is backward.

What happened: DBOS won specifically because of the sleep pattern. dbos.Sleep and dbos.Recv turn "wait for something, survive anything" into a single function call persisted in Postgres. Temporal does this too, but the operational overhead is harder to amortize when the workflow is mostly waiting.

Head-to-head

Property Temporal DBOS
Architecture External server + workers Embedded library + Postgres
Persistence model Event-sourced, replay-based Checkpoint-based, step-level
Determinism guarantee Enforced by SDK + static analysis Convention, not enforced
Go primitives Must use SDK replacements Regular Go, wrap I/O in steps
Deployment Temporal server + DB + ES Your binary + Postgres
Scaling Horizontal (workers + partitions) Vertical (Postgres) + horizontal workers
Queues Task queues built in NewWorkflowQueue
Scheduling Via client API / external cron Built-in cron expressions
Sleep workflow.Sleep() dbos.Sleep()
Resumability Automatic via event replay Automatic via checkpoint
Maturity Battle-tested since 2020 GA since ~2024
Go SDK go.temporal.io/sdk v1.44 dbos-transact-golang v0.18
License MIT MIT

When to choose Temporal

You already run Temporal. If your organization has a Temporal deployment, the operational cost is sunk. Adding new workflows in Go is the path of least resistance.

You need horizontal scale. Temporal's worker pools and partitioned task queues handle throughput that would overwhelm a single Postgres instance. If you are building the control plane for a high-volume system, Temporal's architecture is designed for that.

You want enforced determinism. Temporal's SDK catches non-deterministic patterns at development time. The replay model means your workflow code is tested against real event histories. If correctness under failure is your primary concern, Temporal's constraints are features, not bugs.

You are multi-language. Temporal has mature SDKs for Go, Java, TypeScript, Python, and .NET. If your workflows span services in different languages, Temporal's polyglot support matters.

When to choose DBOS

You want zero new infrastructure. If you already run Postgres, you already run DBOS. There is no server to deploy, no new database to provision, no additional service to monitor. The operational simplicity is the primary value proposition.

Your team is small. Three developers with a Postgres database should not be running a Temporal cluster. DBOS turns durable execution from an infrastructure problem into a library problem. This is the right abstraction level for most teams.

You want deadline-driven sleep. dbos.Sleep(ctx, 48*time.Hour) works across process restarts and redeploys. Temporal's workflow.Sleep() does the same, but the ergonomics of colocating your sleep logic with your workflow code in a single binary are simpler with DBOS.

You are building a Go-native service. DBOS feels like Go. No two-function split between workflow and activity. No SDK replacements for time.Now(). Just regular Go functions with step-wrapped I/O. The learning curve is a fraction of Temporal's.

The deeper split

The choice between Temporal and DBOS reflects a broader tension in backend infrastructure: platform vs library. Temporal is a platform — separate, powerful, operationally significant, and extremely capable. DBOS is a library — embedded, simpler, and limited by the constraints of the database it sits on.

Neither is wrong. The platform approach wins when the problem is large enough to justify the operational cost. The library approach wins when the problem is important but the infrastructure overhead is not. Durable execution is too valuable to skip entirely. The question is whether you want to run a platform for it, or just import a library.


References:

Infrastructure choice is engineering choice. The protocol, the format, the runtime — each is a decision that shapes everything built on top of it. DOT for pipelines, Temporal vs DBOS for durable execution, NATS for messaging. The choice determines the coupling, the scaling, the failure modes. The engineer who treats infrastructure as a commodity gets the failure modes of the default. The engineer who treats it as a design decision gets the failure modes they chose.

Durable execution is not about surviving crashes. It is about making crashes irrelevant. The system that resumes from the last checkpoint does not care that it crashed. The crash is a footnote. The resume is the story.

Why DOT files work for dark factories

Two independent dark factory tools — Kilroy and Mammoth — both chose Graphviz DOT as their workflow format. The reasons are practical, not theoretical: DOT is tiny, readable, visualizable, diffable, and editable by both humans and agents.

dark-factorydotgraphvizagentspipelinesworkflow

Two dark factory tools landed on the same format independently: Graphviz DOT. Kilroy uses it. Mammoth uses it. Neither project seems to have coordinated on the choice. The convergence is worth understanding.

But first, what these tools actually do.

Kilroy: the heavy factory

Kilroy is the heavier of the two — opinionated, factory-scale, closer to the full dark factory vision. Its workflow starts with English requirements: you describe what you want, and Kilroy converts that prose into a Graphviz DOT pipeline. It then validates the graph structure, and runs each node with coding agents operating in isolated Git worktrees. This is not a thin wrapper around an LLM call — it is a production mechanism.

The isolation matters. Each agent invocation runs in its own Git worktree, so agents cannot interfere with each other's file state. This is exactly the kind of engineering discipline the task automation factory paper calls for: execution that is replayable, inspectable, and isolated.

Kilroy records everything. Typed run events and artifacts go into CXDB — a structured event store for run history. Git branches hold the code history. The split is deliberate: Git tracks the code, CXDB tracks the run. You can resume a run from logs, from CXDB, or from a run branch. If something fails, you are not starting from scratch. You are rewinding to a known state and continuing.

This is the factory model made concrete: specification (English requirements) → pipeline (DOT) → isolated execution (Git worktrees) → event recording (CXDB) → resumability (logs, CXDB, run branch). Each stage is auditable. Each stage is replayable. Each stage leaves a record.

Mammoth: the clean runner

Mammoth takes a different approach — lighter, more direct, but equally committed to DOT as the workflow format. It runs DOT-based LLM agent pipelines, validates graphs, supports checkpointing, and includes an HTTP server mode. Its architecture spans DOT parsing, run persistence, a web UI, a TUI, an MCP server, and a CLI — all built around an external tracker execution library.

The web UI is where Mammoth differentiates itself. It includes a spec builder, a DOT editor, and a pipeline runner — all in one interface. You can build the specification, edit the pipeline graph, and execute it without leaving the browser. This lowers the barrier to entry. Kilroy expects you to operate a factory. Mammoth gives you a control panel.

The checkpointing support means Mammoth pipelines can pause, save state, and resume — not as a side effect of Git branches and event stores, but as a first-class feature of the runner. This is a different philosophy from Kilroy's "logs + CXDB + run branch" model, but it solves the same problem: long-running agent pipelines need to survive failure.

Why DOT?

Two independent projects, same format choice. The reasons are practical, not theoretical. Here is why DOT wins for dark factory workflows:

DOT is tiny. The entire DOT language fits in a README. Nodes, edges, attributes, subgraphs. That is basically it. You can learn the syntax in ten minutes. For a format that both humans and agents need to read and write, this minimal surface area is a feature. Every additional syntax element is something an agent can hallucinate.

DOT is readable. A DOT file is a list of relationships. A -> B [label="depends on"]. You can read it without tooling. You can reason about the graph structure by looking at the text. This matters when you are debugging a pipeline at 2am and do not want to fire up a graph visualizer to understand the control flow.

DOT is visualizable. Run it through Graphviz and you get a rendered graph. The same file that drives execution also produces documentation. No translation step, no sync problem — the pipeline definition is the diagram. For teams adopting dark factories, this collapses two artifacts (workflow spec + architecture diagram) into one.

DOT already means graph. The semantics align with the problem. A dark factory pipeline is a directed graph of tasks with dependencies. DOT is a language for describing directed graphs with attributes. The impedance mismatch is zero. You are not forcing a workflow concept into a format designed for something else — DOT was built for exactly this kind of structure.

DOT works well in Git diffs. Each edge is typically one line. Adding a node, reordering dependencies, inserting a new pipeline stage — these produce clean, readable diffs. Compare this to JSON or YAML, where a single added node can cascade through indentation changes across the entire file. DOT diffs tell you what changed at a glance.

DOT lets humans and agents edit the same workflow. This is the killer feature for dark factories. An agent can generate a DOT pipeline from a specification. A human can review it, edit it, add an edge, remove a node — all in the same format, with the same tooling, in the same file. There is no round-trip through a GUI. There is no intermediate representation that only one side understands. The DOT file is the shared artifact.

The format is the interface

Dark factories need a format that sits at the boundary between specification and execution. The format is what the human reviews. It is what the agent generates. It is what the runner executes. It is what goes into version control. It is what you look at when something fails.

Most workflow formats are optimized for one of these audiences. YAML is optimized for machines (and debatably so). GUI-based workflow builders are optimized for humans (at the cost of diffability and agent-writability). Custom DSLs are optimized for a specific tool (at the cost of portability and learnability).

DOT is unusual because it is not optimized for any particular audience — it is just small enough, old enough, and general enough that it works for all of them. That is the definition of a good interface: it gets out of the way.

The convergence of Kilroy and Mammoth on DOT is not a coincidence. It is a signal. When two independent dark factory implementations both reach for the same format, the format itself is part of the design space. If you are building tooling in this area, the question is not "should I use DOT?" It is "is there a reason not to?"


References:

  • Kilroy — English-to-DOT pipeline generation, Git worktree isolation, CXDB event recording
  • Mammoth — DOT-based pipeline runner with web UI, checkpointing, MCP server
  • Graphviz DOT language

Infrastructure choice is engineering choice. The protocol, the format, the runtime — each is a decision that shapes everything built on top of it. DOT for pipelines, Temporal vs DBOS for durable execution, NATS for messaging. The choice determines the coupling, the scaling, the failure modes. The engineer who treats infrastructure as a commodity gets the failure modes of the default. The engineer who treats it as a design decision gets the failure modes they chose.

DOT is not a file format. It is a contract between humans and agents. The human reads the graph. The agent executes the graph. The DOT file is the shared artifact. The artifact is the interface.

Software dark factories: specs in, software out

The dark factory model — where humans write specs and AI agents handle everything else — is not a thought experiment. StrongDM is already running one. Here's what that means for how we build software.

aiagentssoftware-engineeringdark-factoryautomation

In the 1980s, the Japanese robotics company FANUC built a factory where robots manufactured other robots. No human workers. No lights — because nobody was there to need them. The machines just ran.

That image — a dark, humming factory floor producing goods around the clock with zero people in the loop — has haunted manufacturing ever since. Now it has arrived in software.

The term "dark factory" was adapted to software development by Dan Shapiro in January 2026, who laid out a five-level framework for AI-assisted coding. Level 0 is hand-written code. Level 5 is: specs go in, software comes out. Three weeks after Shapiro's post, StrongDM revealed they had been running a dark factory internally since mid-2025. This is not a thought experiment. It is running in production.

The five levels to lights-out

Shapiro's framework maps cleanly to the self-driving car levels. It is worth walking through, because each step describes a working mode that exists today.

Level 0 — Manual. Hand-written code. AI is absent or an afterthought. This is fading fast even among skeptics.

Level 1 — Task delegation. AI handles discrete, on-command tasks: generate unit tests, scaffold a component, write a docstring. The human drives; AI is a tool.

Level 2 — Pair programming. Real-time collaboration between developer and AI. The human guides direction, AI generates code. Shapiro estimates roughly 90% of "AI-native" developers operate here. This is the current default.

Level 3 — Code review. The relationship inverts. AI authors the code; humans review diffs and approve PRs. The developer becomes a manager, not a maker. This is where the psychological shift happens — you stop thinking "I write code" and start thinking "I specify behavior, inspect output, and approve."

Level 4 — Spec-driven development. Humans write detailed specifications — behaviors, acceptance criteria, edge cases — and hand them to AI agents. Hours later, humans check outputs against specs and tests. The developer becomes a product manager. The unit of work is no longer a pull request; it is a specification document.

Level 5 — The dark factory. Specs go in. Software comes out. AI agents write the code, other AI agents review it, still others test it. Agents iterate on failures autonomously. The human role is exclusively defining what to build and why. The how is fully automated.

StrongDM is already doing it

The jump from theory to practice came fast. Three weeks after Shapiro's post, StrongDM — an infrastructure access company — went public with an internal dark factory they had been running since mid-2025. The details are striking:

  • Team size: three engineers.
  • Rules: "Code must not be written by humans" and "Code must not be reviewed by humans."
  • Process: Engineers write prose specifications covering edge cases, error handling, and acceptance criteria. AI agents generate the code. Other AI agents review it. Still others test it. Agents iterate on failures autonomously. Humans touch only the specification and validation layers.
  • Spend benchmark: If you aren't spending at least $1,000 per engineer per day on AI compute, "you have room for improvement."

Simon Willison, co-creator of Django, visited the StrongDM team and described their approach as "very convincing." His takeaway: the critical investment was not in better AI models but in better specifications and test coverage. The quality of the output was a direct function of the quality of the input.

This is the inversion that matters. In a traditional team, you hire for coding skill and hope the specification thinking comes along with it. In a dark factory, specification thinking is the job. Coding is a downstream implementation detail handled by machines.

The bottleneck moves upstream

For decades, the primary constraint on software output was typing speed — or more precisely, the rate at which a human could translate intent into code, handle edge cases, write tests, and iterate through review. Dark factories move the bottleneck from implementation to specification.

This has consequences:

Vague requirements become instantly visible. When a human team receives an underspecified ticket, they fill in the gaps with judgment, experience, and hallway conversations. When an AI agent receives an underspecified spec, it produces exactly what you asked for — and you discover at validation time that what you asked for was wrong. The feedback loop is shorter and more brutal. Ambiguity that would have been absorbed by a senior engineer's intuition now produces a broken build.

Specification becomes a first-class engineering discipline. Writing a spec that an AI can execute against is not the same as writing a Jira ticket. It requires defining behaviors, acceptance criteria, edge cases, and error handling with enough precision that a machine — with no context, no judgment, no hallway conversations — can produce working software. This is a skill. It can be learned. And in a dark factory world, it is the skill that determines output quality.

Senior expertise concentrates differently. Architecture, system design, security, UX — the knowledge that feeds into specifications — becomes more valuable, not less. What becomes obsolete is hand-coding CRUD endpoints or writing boilerplate authentication flows. The senior engineer stops being a high-throughput typist and becomes a high-precision spec writer and validator.

The agency model flips

For software agencies, the dark factory model rewrites the business equation. The traditional model sells developer hours. More work means more billable hours — a linear relationship between output and headcount. In a dark factory, the constraint is not hours but clarity of thought. A small team running a dark factory pipeline can match the output of a much larger traditional team because the typing is free.

The new agency pitch becomes: you are not paying us to write code. You are paying us to define exactly what should be built, validate that it was built correctly, and own the outcome. The value is in the specification craftsmanship and the validation rigor, not in the keystrokes. This is a better business — higher margins, faster delivery, cleaner differentiation — but it requires agencies to sell something they are not used to selling: their thinking, not their typing.

The risks are real

None of this is free of problems.

Quality at scale is unproven. StrongDM's experiment is one team, one domain, one set of constraints. Multiple analyses show AI-generated code carries higher defect rates than human-written code. Dark factories demand extraordinarily robust automated testing and validation to compensate for the absent human review layer. CodeRabbit and similar companies are building tools to address this gap, but the tooling is young.

Specification debt replaces technical debt. Flawed specs produce flawed outputs faster and more confidently than human teams would. The code "works" according to its tests, but the tests were generated from flawed specs. You can end up with a system that passes every automated check and is still wrong — at scale, at speed. Debugging why requires going back to the spec, which was written by a human who may or may not still be on the project.

Trust and compliance are open questions. For enterprise buyers concerned with security, compliance, and maintainability, "no human ever reviewed this code" is not a selling point. The dark factory model will need to build trust — through audit trails, verification tooling, and proven track records — before it is acceptable in regulated environments.

The name might be too good. "Dark factory" is visceral, memorable, and slightly ominous. That is part of its power. It is also part of the risk — the term invites reaction before understanding. Expect pushback from people who hear "no humans" and think "no accountability."

What happens next

The trajectory is clear even if the timeline is not. AI models improve with each release. Agent tooling matures rapidly. The economics are compelling: StrongDM's three-person team plus high compute costs still dramatically undercuts the cost of a traditional team building the same output. And a large percentage of professional software development applies well-understood patterns to business problems — CRUD, auth, API plumbing, dashboard construction. This kind of work is factory-shaped whether we like the metaphor or not.

The open questions are about trust, quality, and adoption curves. Most organizations will spend years at Levels 3 and 4 before approaching true dark factory operations. The tooling needs to catch up. The specification craft needs to develop as a discipline. And the industry needs to figure out what "accountable" means when no human touched the code.

But the direction of travel is set. The dark factory is not science fiction. It is running right now, with real users, on real infrastructure. The only question is how fast it spreads.


References:

  • Dan Shapiro, The Five Levels: from spicy autocomplete to the software factory (January 2026)
  • The "software factory" term itself predates the AI era: Robert W. Bemer's Machine-controlled production environment at the 1968 NATO Software Engineering Conference (Garmisch), with his Checklist for planning software system production dated August 1966 in the same report — one of the earliest explicit formulations of the concept. See The Factory Is Not Dead for the full history and qualification.
  • Dark Factory, What is dark factory software development?
  • Simon Willison's commentary on the StrongDM dark factory approach (referenced in the Dark Factory post above)
  • FANUC's lights-out factory — the manufacturing origin of the "dark factory" term

Systems design is the core engineering discipline. Every system — whether a dark factory, an agent governance framework, or a software architecture — involves the same set of decisions: what are the components? what are their interfaces? what changes do we hide? what stays stable? The engineer who can answer these questions can design any system. The domain provides the constraints. The principles provide the method.

A dark factory is not a tool. It is a production function. The inputs are specifications. The outputs are software. The factory is the mechanism. The mechanism determines the economics.

Task automation economics: why an agent run is not automation

A new paper argues that agentic AI execution is not automation — and that the real economic unit is the verified automation asset, not the agent run.

agentsautomationeconomicsdata-engineeringsoftware-engineering

A successful agent run feels like progress. The model understood the task, called the right tools, produced the output. Ship it.

But a new paper by Mohamed A. Fouad (On Task Automation Economics) argues that this feeling is a category error. An agent run is an event. Automation is capacity. Confusing the two is the central mistake teams make when adopting agentic AI for recurring work.

The paper is compact — 9 pages — but the argument is sharp and worth engaging with. Here is my reading of it.

The category error

The paper opens with a clean distinction:

Agentic AI execution is not automation. Automation begins when recurring work becomes verified software capacity.

The problem is not that agents lack capability. They can perform knowledge-based tasks — ingest files, map schemas, clean records, transform tables. The problem is that a successful execution leaves behind no reusable organizational capacity. The next time the same task comes up, you run the agent again. You purchase the output again. Nothing accumulates.

This is the category error: treating an event as an asset. An agent run proves feasibility. It does not create capacity. The distinction matters because repeated execution has weak economics — each run repurchases output — while verified assets accumulate value across reuse.

Task automation economics

The paper names the decision problem task automation economics: when should recurring work stop being bought as separate agent runs and become governed software capacity?

This extends Barry Boehm's software engineering economics framework from software products to agent-mediated task assets. The economic unit is not the agent, the prompt, or the run. It is the verified automation asset — a released object with seven reviewable parts:

  1. Specification — what the task does, explicitly
  2. Governance template — what rules and evidence apply
  3. Pipeline — the executable mechanism
  4. Criteria — how acceptance is judged
  5. Evidence — records that the criteria were met
  6. Release snapshot — the versioned, reviewed state
  7. Replacement rule — when and how the asset should be retired or replaced

The economics become favorable when reuse value exceeds lifecycle cost. Value comes from accumulated capability. Cost comes from specification, engineering, verification, audit records, and replacement. This is a familiar tradeoff — it is the same logic behind technical debt in ML systems (Sculley et al., 2015). The difference is that with agentic AI, the run-level capability is so easy to achieve that teams never graduate to the asset level. They stay at "let me just run the agent again" indefinitely.

The task automation factory

If task automation economics names the why, the task automation factory names the how. The paper defines it as a production mechanism with five stages:

  1. Select candidate demand — identify recurring work
  2. Specify rules and evidence — make acceptance criteria explicit
  3. Engineer tools and pipelines — build the executable mechanism
  4. Verify with tests and audit records — prove the asset works
  5. Release as a versioned asset — make it reusable

Each stage has a failure mode. If you skip specification, automation guesses. If you skip engineering, it remains repeated execution. If you skip verification, it cannot be trusted. If you skip release, it cannot be reused. If you skip replacement, it decays.

This connects directly to the dark factory concept — and the paper explicitly references Dan Shapiro's five levels and the broader dark factory discourse. But Fouad adds an important constraint: a task automation factory is dark only as a production metaphor. It must not make data, pipeline, or workflow accountability invisible. Rules, evidence, audit records, and replacement paths must remain visible. This is not lights-out as a trust model. It is lights-out as an execution model, with governance kept fully lit.

The evaluation checklist

The paper provides a concrete test for whether a task has graduated from execution to automation. Six questions a reviewer should be able to answer:

Criterion Reviewer question
Explicit task Is the recurring work described clearly enough to engineer?
Defined evidence Is acceptance tied to rules and records?
Replayable pipeline Can the mechanism be rerun and inspected?
Reconstructible acceptance Can a reviewer explain why the asset passed?
Known release Is reuse tied to a reviewed release state?
Replacement path Is there a trigger for repair or retirement?

This checklist is deliberately narrow. It does not test whether every AI risk has been addressed. It tests whether a recurring task has become a reviewable asset. A high-scoring agent run is insufficient if the team cannot link it to a rule, an evidence record, a verification criterion, a release snapshot, and a replacement path. Conversely, a modest deterministic tool may be more valuable than a sophisticated agent if it replaces repeated execution with governed capacity.

Why data engineering

The paper grounds the argument in data engineering workflows: ingestion scripts, schema mappings, cleaning rules, transformation jobs, validation checks, orchestration DAGs, and backfill procedures. These tasks recur constantly. They require interpretation, system context, and operational judgment. They also require evidence preservation — sources, transformations, checks, and lineage must remain inspectable.

Data engineering is a strong choice of domain because the gap between run and asset is so visible there. A one-off script that cleans a table is useful once and decays when the schema changes. The team runs it again with adjustments. And again. Each run works. But nothing accumulates. The task automation factory turns that repeated demand into a verified, reusable, replaceable asset — and the economics shift from repurchasing output to accumulating capacity.

What I take from this

Three things stand out.

First, the paper is making an argument about organizational capability formation, not about agent performance. The better agents become at execution, the easier it becomes to mistake performance for capacity. This is a real risk. Teams that optimize for run success rates are optimizing the wrong variable. The variable that matters is the conversion rate from repeated runs to released assets.

Second, the verified automation asset is a useful concept independent of the tooling. Even if you never touch axnrun (the open-source runtime the paper uses for grounding), the seven-part structure — specification, governance, pipeline, criteria, evidence, release, replacement — gives you a checklist for auditing whether your team is accumulating capacity or just accumulating runs.

Third, the paper is short and reads more like a position paper than an empirical study. That is a feature, not a bug. The claim is limited: not every task should be automated. The argument applies to recurring workflow demand where evidence and review matter. The evaluation criteria are offered as a practical test, not a formal framework. And the paper is honest about what is not yet demonstrated — conversion rates, time-to-asset, reuse counts, audit reconstruction success. Those measurements are left as future work.

The core insight — that a run is an event and automation is capacity, and confusing the two is a category error — is worth sitting with. Especially if your team is doing a lot of successful agent runs and wondering why it does not feel like progress is accumulating.


Reference: Mohamed A. Fouad. On Task Automation Economics. arXiv:submit/7796173, July 2026. (Talk page with slides)

Systems design is the core engineering discipline. Every system — whether a dark factory, an agent governance framework, or a software architecture — involves the same set of decisions: what are the components? what are their interfaces? what changes do we hide? what stays stable? The engineer who can answer these questions can design any system. The domain provides the constraints. The principles provide the method.

An agent run is not automation. It is an event. Automation is capacity. The difference is that an event proves something can be done. Capacity means it will be done, repeatedly, without further human intervention.

The economics of the dark factory: what happens when code is free

When implementation cost approaches zero, the economics of software production invert. The scarce resource is no longer coding — it's specification, validation, and trust.

dark-factoryeconomicssoftware-engineeringagentsbusiness

Software has always had unusually favorable marginal economics. Once written, a piece of software can be copied and distributed at near-zero marginal cost. This is why software businesses scale differently from physical-goods businesses.

But there was always a catch: the first copy was expensive. Writing the software required skilled labor, and that labor was the dominant cost in software production. The marginal cost of distribution was zero, but the fixed cost of creation was high.

Dark factories change this. When AI agents write the code, the fixed cost of creation drops dramatically. Not to zero — specification and validation remain — but to a fraction of what it was. This rewrites the economics of who can build software, how software businesses are structured, and where value accrues.

The cost structure before and after

In a traditional software team, the cost structure looks roughly like:

  • Implementation labor: 50-60% of engineering budget (writing code, reviewing code, iterating on code)
  • Specification and design: 15-20% (architecture, technical specs, product requirements)
  • Testing and validation: 15-20% (manual QA, automated testing, integration testing)
  • Operations and maintenance: 10-15% (deployment, monitoring, incident response)

A dark factory compresses the implementation bucket. StrongDM's team of three engineers, spending over $1,000 per engineer per day on AI compute, produces the output that would traditionally require a much larger team. The implementation labor cost is replaced by compute cost — and compute, unlike labor, is elastic, scalable, and improving in price-performance with every model generation.

The post-factory cost structure shifts toward:

  • Specification and design: 40-50% (this becomes the primary engineering activity)
  • AI compute: 10-20% (the new variable cost — roughly $1K+/engineer/day at current StrongDM-level spend)
  • Testing and validation: 20-30% (validation becomes more important, not less, when code is AI-generated)
  • Operations and maintenance: 10-15% (similar, but with new challenges around specification drift)

The total cost is lower — sometimes dramatically lower — but the composition is different. Engineering effort concentrates at the top of the funnel (specification) and the bottom (validation), with the middle (implementation) largely automated.

The StrongDM benchmark

The StrongDM case study provides the first real-world data point. Three engineers, operating under the rules "code must not be written by humans" and "code must not be reviewed by humans," producing at a rate that would traditionally require a significantly larger team.

The most striking number is the compute spend benchmark: if you are not spending at least $1,000 per engineer per day on AI compute, "you have room for improvement." At current model pricing, that buys an enormous volume of agent execution. Claude Opus 4.8 at list price is $15/MTok input, $75/MTok output. A thousand dollars buys approximately 13 million output tokens — the equivalent of generating tens of thousands of lines of code, plus reviews, plus test generation, plus iteration on failures.

Compared to a fully-loaded senior engineer cost of $600-800/day (salary, benefits, overhead), the math is straightforward: if the AI can produce even a fraction of that engineer's output at a fraction of the cost, the economic advantage is compelling. And the AI does not take vacations, does not switch jobs, and improves in capability with each model release.

The margin structure of dark factory businesses

This cost structure shift has specific implications for different business models:

SaaS businesses. The traditional SaaS cost structure has high initial engineering investment and low marginal cost per user. Dark factories compress the initial investment, making it feasible to build and maintain SaaS products with smaller teams. The competitive dynamic shifts: incumbents with large engineering organizations lose their headcount advantage against smaller, factory-enabled competitors. The moat moves from "we have more engineers" to "we have better specifications" — which is a very different kind of moat.

Agencies and consultancies. The traditional agency model sells engineer hours. More work requires more engineers, and revenue scales roughly linearly with headcount. A dark factory breaks this relationship. A small team can produce the output of a much larger one, which means revenue per employee can increase dramatically. But it also means the sales pitch must change: you are no longer selling "we have great engineers who will write your code." You are selling "we have great spec writers and validators who will define exactly what should be built and verify that the AI built it correctly." The client must be sold on the process, not the headcount.

Vertical software. The most interesting play may be in vertical SaaS — industry-specific software for niches that were previously too small to justify a dedicated engineering team. When the fixed cost of creation drops, the addressable market for custom or semi-custom software expands. Problems that couldn't support a five-person engineering team may support a one-person team plus a dark factory pipeline. The long tail of software opportunities becomes economically viable.

Where the money goes

If implementation is commoditized, the economic value in the software supply chain concentrates in three places:

1. Specification expertise. The people who can define precisely what should be built — domain experts who understand the problem, product thinkers who understand the user, architects who understand the system constraints. These people were always valuable. In a dark factory world, they are the primary constraint on output. Their leverage increases because their specifications can be executed at machine scale.

2. Validation infrastructure. The tooling that verifies AI-generated code against specifications, catches specification drift, and provides the audit trail for compliance and trust. Companies like CodeRabbit are early entrants here, but the category is wide open. Validation is not just testing — it is the entire chain of evidence from specification to acceptance, and it needs to be automated at the same scale as the code generation it checks.

3. Trust and distribution. When code is commoditized, the question "can I trust this software?" becomes the buying decision. The dark factory operator who can demonstrate rigorous validation, clean audit trails, and proven reliability has a structural advantage over competitors with similar output quality but weaker trust signals. This is especially true in regulated industries — finance, healthcare, infrastructure — where "an AI wrote this" is currently a liability that must be offset by evidence of correctness.

What breaks the model

The economics are compelling, but they rest on assumptions that can fail:

Model pricing does not stay flat. If AI compute costs increase — through provider consolidation, demand exceeding supply, or regulatory intervention — the cost advantage of dark factories over traditional teams shrinks. Conversely, if costs continue their current trajectory downward, the advantage grows. The economics are tied to a variable that teams do not control.

Specification costs may be higher than expected. The assumption that specification is cheaper than implementation depends on specifications being less voluminous and less complex than the code they replace. This may not be true. A specification precise enough for machine execution may approach the complexity of the code itself — different in form, but similar in information content. If so, the cost savings are real but smaller than the headline "code is free" suggests.

Quality externalities may dominate. If AI-generated code carries higher defect rates that must be caught in production, the operational cost of dark factory software may offset the development savings. This is the "specification debt" problem: flawed specs produce flawed outputs faster and more confidently than human teams would. The total cost of ownership may be higher even if the initial development cost is lower.

Trust is not free. For enterprise buyers, "no human reviewed this code" is currently a negative signal. Building trust — through validation infrastructure, audit trails, compliance certifications — costs money. The trust premium may erode some of the cost advantage, especially in the early years.

The equilibrium

Where does this settle? The most likely equilibrium is not "all software is built in dark factories" but a hybrid: dark factories handle the large fraction of software that applies well-understood patterns to business problems (CRUD, auth, API plumbing, dashboard construction, data pipelines), while human-led development handles genuinely novel systems, cutting-edge research, and domains where the cost of getting it wrong is catastrophic.

This is already the shape of the StrongDM experiment. The dark factory handles the predictable work. Humans handle the specification, the validation, and the exceptions. The ratio will shift over time as models improve and tooling matures, but the principle — factories for the known, humans for the unknown — is likely durable.

The economic prize is enormous: a large fraction of professional software development falls into the "well-understood patterns applied to business problems" category. Moving that work from labor cost to compute cost is one of the largest productivity improvements available in the global economy. The teams that figure out how to do it well — with rigorous specification, automated validation, and earned trust — will have a structural cost advantage that compounds with every model generation.


References:

Systems design is the core engineering discipline. Every system — whether a dark factory, an agent governance framework, or a software architecture — involves the same set of decisions: what are the components? what are their interfaces? what changes do we hide? what stays stable? The engineer who can answer these questions can design any system. The domain provides the constraints. The principles provide the method.

When code is free, the scarce resource shifts from implementation to specification. The person who can say exactly what to build becomes more valuable than the person who can build it. The economics invert.

The dark factory doesn't eliminate complexity — it moves it

Dark factories shift the bottleneck from implementation to specification. The complexity doesn't vanish — it concentrates upstream, and the skill that matters is the ability to say exactly what you mean.

dark-factorycomplexitysoftware-engineeringspecificationagents

In 1986, Fred Brooks drew his famous distinction between essential and accidental complexity. Essential complexity is inherent in the problem — you cannot remove it, only manage it. Accidental complexity is everything we inflict on ourselves: build systems, type systems, deployment pipelines, the accumulated sediment of toolchain decisions that have nothing to do with the problem domain.

For four decades, the software industry has fought accidental complexity with better languages, better tooling, better abstractions. Dark factories change the game in a way Brooks did not anticipate: they make accidental complexity someone else's problem. Specifically, the AI's problem.

But essential complexity does not go anywhere. It just moves.

The complexity budget

Every software system has a fixed complexity budget. You can think of it as the total information content required to produce working software: the sum of domain knowledge, architectural decisions, edge-case handling, behavioral contracts, and operational constraints. Before dark factories, this budget was spent across the entire stack — some in specification, some in architecture, some in implementation, some in testing, some in operations.

A dark factory reshuffles where each unit of complexity is absorbed. Implementation complexity — the part that was always accidental, the part that was about translating intent into code — drops toward zero. But the total complexity budget does not shrink. The complexity that used to be absorbed by a senior engineer during implementation now must be absorbed upstream, in the specification, or downstream, in validation and operations.

The upstream shift is the one that matters. When a human engineer receives a vague ticket, they fill in the gaps: they know the codebase conventions, they have intuitions about edge cases, they can ping the PM and clarify. When an AI agent receives a vague specification, it produces exactly what you asked for — and you discover at validation time that what you asked for was wrong.

This is not a failure of the AI. It is a failure of the specification. And in a dark factory world, specification failures are the only kind of failure that matter.

What specification actually means now

Specification in a dark factory context is not a Jira ticket. It is not a user story. It is not "as a user I want to reset my password." It is a document with enough precision that a system with no context, no judgment, and no ability to clarify ambiguity can produce working software from it.

This means specification must contain:

  • Behavioral contracts — given these inputs, produce these outputs, within these constraints
  • Edge cases enumerated — empty states, error states, boundary values, concurrent access patterns
  • Acceptance criteria at machine resolution — not "the page loads fast" but "the page renders within 200ms at P95 under 10K concurrent requests"
  • Error handling semantics — what fails gracefully, what fails loudly, what retries, what alerts
  • State machine definitions — what states exist, what transitions are legal, what invariants hold
  • Integration contracts — API shapes, authentication models, retry policies, failure modes of dependencies

This is not a new discipline. It is what good technical leads have always done, just more explicit and more complete. The difference is that in a traditional team, gaps in the specification could be absorbed by the engineer implementing it. In a dark factory, gaps in the specification produce gaps in the software — at scale, at speed, with nobody in the loop to catch them.

The inversion of expertise

This reshuffling inverts what "seniority" means. In a traditional team, seniority is partly about coding skill — the ability to hold a large system in your head, to write clean abstractions, to navigate the codebase quickly. In a dark factory, coding skill is commoditized. What remains valuable is:

  • Domain modeling — the ability to see the shape of a problem and express it precisely
  • Edge-case imagination — the paranoid instinct for what could go wrong, honed by years of production incidents
  • Contract design — the ability to define interfaces that are complete, minimal, and stable under change
  • Validation strategy — knowing what "correct" looks like and how to test for it at multiple levels

These skills are rare. They were always the truly valuable part of senior engineering — the part that distinguished a 10x engineer from a fast typist. What dark factories do is strip away everything else, making it obvious that specification craftsmanship was the bottleneck all along.

The new accidental complexity

There is a twist. Dark factories eliminate one form of accidental complexity (implementation) while potentially creating new forms:

Specification tooling complexity. If specification becomes the primary engineering artifact, the tooling around specification — versioning, diffing, linting, testing, code review for specs — becomes critical. A bad specification is harder to debug than bad code because you cannot step through it with a debugger. The specification is the source of truth, and we have almost no tooling for managing specification quality at scale.

Validation complexity. When AI generates code, the testing burden inverts. You are no longer testing that a human implemented what they intended. You are testing that an AI implemented what you specified — and also that what you specified was correct. The second problem is harder than the first. It requires tests that validate the specification against reality, not just the implementation against the specification.

Drift complexity. Specifications, like code, drift from reality over time. In a traditional codebase, drift manifests as stale comments and outdated READMEs — annoying but usually harmless. In a dark factory, drift manifests as the AI faithfully implementing an outdated specification, producing software that matches the spec but not the world. Detecting and correcting specification drift becomes an operational concern.

None of these are unsolvable. But they are new. And they will consume engineering effort that was previously spent on build systems, linters, and CI pipelines — the old accidental complexity that the dark factory absorbed.

What this means for teams

The practical implication is that teams adopting dark factory workflows should stop measuring implementation velocity and start measuring specification quality. The metric that matters is not "how fast can we produce code" — the AI does that instantly. The metric is "how often does the produced code match intent on the first try."

That metric is a function of specification quality. And specification quality is a function of how well the team understands the problem domain, how rigorously they think about edge cases, and how clearly they can express what they mean.

The dark factory does not make software easy. It makes implementation easy. The hard part — understanding what to build and defining it precisely enough that a machine can build it — remains hard. It always was. The factory just makes it impossible to pretend otherwise.


References:

Systems design is the core engineering discipline. Every system — whether a dark factory, an agent governance framework, or a software architecture — involves the same set of decisions: what are the components? what are their interfaces? what changes do we hide? what stays stable? The engineer who can answer these questions can design any system. The domain provides the constraints. The principles provide the method.

The dark factory does not eliminate complexity. It moves it. The complexity that lived in the code now lives in the specification. The specification must be precise enough that a machine can execute it without asking questions. Precision is the new bottleneck.

Always-on agents: state, memory, and the governance gap

A new survey of 435 papers argues that making agents truly always-on requires treating state as a first-class systems concern — not just remembering, but governing, recovering, and forgetting.

agentsmemorystategovernancesurvey

Most of the agent conversation focuses on what happens during a task: tool calls, reasoning loops, correctness. Far less attention goes to what happens between tasks — the accumulated state, memory, permissions, commitments, and audit trails that persist across interactions.

A new survey from Ding, Nannapaneni, Liu, and Zhang (Always-On Agents: A Survey of Persistent Memory, State, and Governance in LLM Agents, June 2026) argues that this gap is the critical unsolved problem for deploying agents that operate continuously. And they back the claim with a coded analysis of 435 papers.

What is an always-on agent?

The paper defines always-on agents as LLM-based systems whose future behavior depends on durable, accumulated state from past interactions. This state is not just retrievable memories. It includes:

  • Task ledgers — what the agent has done, is doing, and has committed to do
  • Permissions and credentials — what the agent is authorized to access, and how those authorizations change over time
  • Commitments — promises made to users, other agents, or external systems
  • Provenance and audit records — how each decision was reached, for post-hoc review
  • Shared state — what multiple agents or an agent and its user both rely on
  • Trigger conditions — latent rules that fire when certain conditions are met
  • Externally committed effects — side effects the agent has already pushed into the world

This is a much richer picture than "the agent has a vector database of past conversations." An always-on agent is a persistent-state system. The persistence is the feature.

The gap: we're good at memory, bad at governance

The survey's central finding is blunt: the literature is heavily skewed toward accumulating and retrieving state, with far less attention to how to govern, recover, or relinquish that state.

We have plenty of papers on retrieval-augmented generation, embedding-based memory, and context window management. We have far fewer on:

  • Forgetting. When should an agent delete a memory? How do you ensure it actually forgets — including from backups, cached contexts, and fine-tuned weights? This connects directly to machine unlearning and to legal requirements like GDPR's right to erasure.
  • Recovery. If an agent's state is corrupted — by a bad interaction, a prompt injection, a buggy tool — how do you roll back? What is the agentic equivalent of a database transaction?
  • Auditing. If an agent made a consequential decision three weeks ago, can you reconstruct exactly what state it had access to at that moment, what it retrieved, and how it weighed that information?
  • Authority and scope. Who can modify the agent's state? If an agent has learned a preference from user A, should user B's interactions be influenced by it? What happens when state from different sources conflicts?

The paper frames this as a maturity problem. We have built the memory layer for always-on agents. We have not yet built the governance layer. And you cannot safely deploy persistent-state agents at scale without both.

Six diagnostic axes

The authors propose six axes for analyzing any piece of agent state:

  1. Authority — Who or what created this state item? Who can modify or delete it?
  2. Scope — Is this state private to one agent, shared across a fleet, or tied to a specific user?
  3. Mutability — Can this state change? Under what conditions? Is it append-only, versioned, or freely overwritable?
  4. Provenance — Where did this state come from? What chain of interactions produced it?
  5. Recoverability — If this state is lost or corrupted, can it be reconstructed? From what?
  6. Actionability — Does this state item directly drive agent behavior, or is it purely informational?

Most current agent frameworks score well on actionability (of course state drives behavior) and poorly on provenance and recoverability (good luck reconstructing why the agent did what it did six weeks ago). The axes give teams a checklist for auditing their own systems: for each piece of state your agent accumulates, can you answer all six?

The lifecycle: state as a managed resource

Beyond the diagnostic axes, the paper introduces a lifecycle model for agent state. State is not just written and retrieved — it moves through a series of stages, each of which can fail:

  • Write — state is created or updated
  • Validate — is the state correct, consistent, not poisoned?
  • Organize — how is state structured, indexed, deduplicated?
  • Retrieve — the well-studied part: finding relevant state at decision time
  • Act — state drives a decision or an external effect
  • Update — the decision's outcome feeds back into state
  • Forget — state is intentionally removed or decayed
  • Audit — state is examined after the fact for correctness or compliance
  • Rollback — state is restored to a prior version after a failure

The lifecycle exposes the asymmetry in current research. Write, organize, retrieve, and act are well-covered. Validate, forget, audit, and rollback are not. This means we are building agents that accumulate state aggressively and have almost no machinery for unwinding it when something goes wrong.

AOEP-v0: governance as an evaluation target

One of the paper's more interesting contributions is the Always-On Evaluation Protocol (AOEP-v0) — a pilot evaluation contract that scores systems on state mutation and recovery obligations rather than answer quality.

This is a meaningful departure from standard agent benchmarks. Most evals ask: "Did the agent complete the task correctly?" AOEP-v0 asks questions like: "If we corrupt a piece of the agent's state, does it detect the corruption? Can it recover? If we issue a forget request, is the memory actually gone from all layers?" These are systems questions, not task-completion questions. They require testing the agent's governance machinery, not its reasoning quality.

The protocol is explicitly a v0 — early, incomplete, aspirational. But the direction is right. As agents move from demo to deployment, the evaluation that matters is not "can it answer questions" but "can you trust it to run for six months without accumulating dangerous state, leaking cross-user information, or becoming un-auditable."

Why this matters now

The timing of this survey is good. Agent deployment is accelerating — from coding assistants to customer-facing autonomous systems. Each of these deployments accumulates state. Each one will eventually hit the governance questions the paper raises. And right now, the answers are mostly ad-hoc: prompt the agent to "be careful about stale information," log everything to a table nobody queries, hope for the best.

The paper connects always-on agents to mature disciplines that have already solved adjacent problems: databases (transactions, rollback, consistency), distributed systems (state reconciliation, quorum, fencing), capability security (authority, attenuation, revocation), and formal methods (invariants, verification). The claim — and I think it is correct — is that agent state governance is not a novel problem requiring novel solutions. It is a composition problem: we have the pieces, but we have not wired them together in the agent context.

This is a call to action. If you are building agent infrastructure, the question is not just "how does the agent remember?" It is "how does the agent govern its memory?" The second question is harder. It is also the one that will determine whether always-on agents are safe to deploy.


Reference: Tianyu Ding, Aditya Nannapaneni, Bingfan Liu, Ling Zhang. Always-On Agents: A Survey of Persistent Memory, State, and Governance in LLM Agents. arXiv:2606.30306, June 2026.

Systems design is the core engineering discipline. Every system — whether a dark factory, an agent governance framework, or a software architecture — involves the same set of decisions: what are the components? what are their interfaces? what changes do we hide? what stays stable? The engineer who can answer these questions can design any system. The domain provides the constraints. The principles provide the method.

An always-on agent is not a faster request-response loop. It is a different architecture. The agent that persists state across interactions is a system. The system must be governed. Governance is the hard part.

Academics without academic integrity: shame on you

When senior academics steal ideas from students, manuscripts, and grant proposals — and the system protects them — they burn down the trust that makes research possible.

academiaintegrityethicsresearch

There is one asset that separates a university from a trade school, a scholar from a blogger, and a journal from a Substack: the claim that the work is honest. Not clever. Not well-cited. Honest.

And among all the ways to betray that claim, one stands apart for its particular cowardice: stealing ideas. Not fabricating data out of thin air — that at least requires inventing something. Not plagiarizing published text — that's just lazy. Stealing ideas is different. It is predatory on trust. It exploits the specific vulnerability that academic work requires: sharing your thinking before it is finished, with people who have more power than you do.

This is not a footnote to the broader story of academic fraud. It is the core of it. And the system is built to protect the thieves.

How they do it

The mechanisms are well-known to everyone inside academia and almost invisible to everyone outside it.

The reviewer scoop. You submit a manuscript to a journal or conference. A reviewer sits on it for weeks — long enough to extract the core insight, assign it to a fast-moving postdoc, and get a competing submission into the next venue before yours even clears the review queue. You find out when you see their paper, which cites you in footnote 14 for something tangential while the central idea you developed now has their names on it. Proving it is nearly impossible. The editor will shrug. Your paper now looks derivative of theirs. You lose three years of work to someone who read a PDF for an afternoon and recognized a good idea when they saw one.

The advisor tax. A graduate student spends years developing a research direction, writing the code, running the experiments, drafting the manuscript. The advisor's contribution was a vague suggestion in a meeting two years ago — "maybe try applying X to Y" — that the student turned into an actual research program. When the paper comes out, the advisor is first author. When the press covers it, the advisor is the face. When tenure committees evaluate it, the advisor claims the intellectual leadership. The student gets a diploma and a lesson in how power works.

The conference predator. You present preliminary work at a workshop or a poster session. Someone from a larger, better-funded lab takes careful notes, asks detailed questions that feel like engagement, and then returns to their institution and replicates your approach with more compute, more RAs, and more name recognition. They publish first. You are left explaining to your own advisor why your project now looks like a replication study of someone else's result.

The grant proposal heist. You submit a grant proposal. It is reviewed by a panel that includes a senior person in your subfield. The proposal is rejected — "insufficient preliminary data," "overly ambitious" — but eighteen months later that same senior person's group publishes a paper whose research plan maps almost one-to-one onto your unfunded proposal. You recognize your hypotheses, your experimental design, even your clever naming of conditions. They claim independent convergence. Nobody investigates.

Four mechanisms, same shape every time: someone with less power shares an idea in good faith, someone with more power takes it, and the system provides cover.

The power gradient is the whole story

Fabrication and plagiarism can be committed by anyone at any level. A desperate undergrad can photoshop a gel. A postdoc can copy-paste a literature review.

Idea theft is different. It almost always flows downhill: from the tenured to the untenured, from the senior to the junior, from the well-resourced to the scrappy. The thief has the platform, the lab, the reputation, and the publication velocity to execute on the stolen idea faster than its originator can. The originator has nothing but the sickening moment of recognition when they open a proceedings volume and see their own thought staring back at them under someone else's byline.

This is not a crime of desperation. The people who steal ideas already have labs, grants, and CVs long enough to survive several lifetimes of honest work. They do not need your idea. They take it because they can. Because the power gradient means there are no consequences. Because their reputation will be believed over yours every time.

The perfect crime

And here is why idea theft flourishes while data fabrication occasionally gets caught: it is structurally impossible to prove.

To establish that someone stole your idea, you must demonstrate three things: that the thief had access to the idea (they did — you gave it to them in good faith), that the idea was novel (it was — that's why they took it), and that the thief would not have arrived at it independently. That third one is a logical impossibility. You cannot prove a counterfactual. Two people can independently converge on the same idea. The fact that one of them reviewed the other's manuscript, or heard their talk, or read their grant proposal six months earlier is, legally speaking, a coincidence.

The evidentiary standard is set so high that it functions as a license to steal. Everyone knows this. The thieves count on it. The language they use when confronted — "parallel discovery," "independent convergence," "the idea was in the air" — is rehearsed precisely because it cannot be falsified. It is the academic equivalent of "who are you going to believe, me or your lying eyes?"

What this destroys

The most obvious victim is the person whose idea was taken. They lose years of work, a publication, a grant, a career trajectory. Some leave academia entirely. Some stay and learn the game — they become hoarders, presenting only published work, treating every lab meeting as a potential heist, never sharing a half-formed thought with anyone who outranks them.

That second outcome is the one that should terrify us. When junior researchers learn that sharing ideas means risking expropriation, the intellectual commons collapses. The free exchange of half-formed ideas — the generative friction of honest peer feedback, the hallway conversation that sparks a collaboration, the workshop where someone says "have you thought about trying X?" — all of it dies. What replaces it is a series of armed standoffs where everyone presents finished, timestamped work and nobody says anything real.

The people who steal ideas are not just taking someone else's work. They are burning down the collaborative infrastructure that makes research possible in the first place. And they do it for one more line on a CV they already didn't need.

The incentives are not an excuse

I know the counterargument. "Publish or perish." "The metrics are broken." "Everyone does it."

Yes, the incentive structure is perverse. Yes, hiring committees count papers instead of reading them. Yes, impact factor determines funding, and impact-factor optimization is not the same thing as science.

But the people who steal ideas are, almost without exception, the ones who have already won the game. They have tenure. They have funding. They have students. They are not scrapping for survival — they are padding a record that was already padded. And they are doing it by reaching down the power gradient and taking from people who have none of those protections.

Nobody held a gun to your head and forced you to take a graduate student's dissertation insight, strip their name from it, and publish it as your own "independent" work while they watched from the acknowledgments section. Nobody made you sit on a manuscript you were reviewing, extract its core contribution, and race it to publication with your own lab's resources. You made a calculation — that the personal benefit outweighed the risk, and that the risk was zero because the system is designed to protect people exactly like you — and you acted on it.

That is not structural pressure. That is predation. And the fact that you can dress it up in systems-critique language does not make it less ugly.

Shame is the appropriate response

We have sanitized the language around academic misconduct. "Research integrity concern." "Questionable research practice." "Intellectual property dispute." These phrases exist to avoid saying what actually happened: someone with power reached down, took something that did not belong to them, and the institution looked the other way.

Shame is the correct word. Not embarrassment. Not "a learning opportunity." Shame — the public acknowledgment that you violated a trust held by your students, your colleagues, and the public that funded your work.

If you stole an idea from a student and called it mentorship: shame on you. If you mined a manuscript you were reviewing and raced it to publication: shame on you. If you rejected a grant proposal and then published its research plan under your own name: shame on you. If you heard a junior colleague present at a workshop and beat them to print with your larger lab: shame on you. If you are a department head who knew about any of this and protected the thief because they bring in grants: shame on you. If you are a journal editor who has spent more energy protecting your impact factor than investigating the papers in it: shame on you.

The rest of us — the ones who still believe that honest inquiry matters, that credit should flow to the person who had the idea, and that the power gradient should not function as a permission structure for theft — need to stop treating academic fraud as a PR problem and start treating it as a betrayal. Because that's what it is.

And the idea thieves know it. Watch how they react when caught: not with confession, but with lawyering. Not with repair, but with "independent convergence." Not with shame, but with the calm confidence of someone who knows the system will not touch them.

They know what they did. They just assumed they would get away with it.

Prove them wrong.

Systems design is the core engineering discipline. Every system — whether a dark factory, an agent governance framework, or a software architecture — involves the same set of decisions: what are the components? what are their interfaces? what changes do we hide? what stays stable? The engineer who can answer these questions can design any system. The domain provides the constraints. The principles provide the method.

Integrity is not a virtue. It is a constraint. The researcher who fabricates data removes the constraint. The removal produces papers. It does not produce knowledge.

AI sovereignty or AI colony: why domestic capability matters

If a country cannot build and operate its own AI stack, it becomes dependent on foreign models, cloud capacity, and policy defaults.

aipolicysovereigntygeopolitics

A country that cannot build its own AI capabilities does not stay neutral — it becomes dependent on systems and rules designed elsewhere.

Hisashi Matsumoto

That is why the warning from Japan's digital minister matters: without domestic capability, even advanced economies risk becoming an "AI colony" where strategic choices are constrained by external providers and standards.

In practice, this dependency appears across three layers:

  1. Infrastructure dependency: compute, chips, cloud credits, and model hosting sit outside national control.
  2. Model dependency: core models, update cycles, and safety tuning are dictated by vendors in other jurisdictions.
  3. Governance dependency: policy defaults (privacy boundaries, content rules, API limits, auditability) are inherited instead of negotiated.

The hard lesson is simple: AI sovereignty is not only about "having a model." It is about building sustained local capacity across talent, infrastructure, open research, and institutions that can set and enforce national priorities.

Countries that invest early in these foundations gain bargaining power, resilience, and room to adapt AI to local language, law, and economic goals. Countries that do not risk locking themselves into technical and regulatory dependence.

Reference: Japan risks becoming an AI colony, its digital minister warns

Engineering is always embedded in a context of constraints — economic, political, organizational. The engineer who ignores the context builds systems that are technically correct and operationally irrelevant. The engineer who understands the context builds systems that survive. The context is part of the specification. The specification is incomplete without it.

AI sovereignty is not about having a national model. It is about having the capacity to build one. The capacity is infrastructure, talent, and data. Without all three, sovereignty is aspirational. With all three, it is operational.

How to build self-improving companies and internal AI agents (YC talks)

Notes and links for two Y Combinator talks about self-improving companies and internal AI agents.

youtubeycaiagents

This short post collects notes and links for two Y Combinator talks. Thumbnails below link to the videos; embedded players are included for easy viewing.

Videos

How to Build a Self-Improving Company with AI — Y Combinator

How to Build a Self-Improving Company with AI

A few short takeaways and prompts for teams and builders:

  • Focus on measurable feedback loops: define metrics, collect signals, and automate where the ROI is clear.
  • Use AI to scale repeatable improvement tasks while keeping human oversight on strategy and safety.
  • Start with small pilot systems that demonstrate measurable improvement before scaling.

How to Build an Internal AI Agent That Evolves Itself — YC Root Access

How to Build an Internal AI Agent That Evolves Itself

Short notes:

  • Design agents with safety and guardrails; make evolution safe, observable, and reversible.
  • Balance automation with human-in-the-loop checks for high-impact decisions.
  • Emphasize evaluation metrics and continuous testing when enabling agent self-improvement.

Screenshots: YouTube thumbnails (downloaded and included in this repo). If you prefer different screenshots or specific timestamps as images, say which and a screenshot can be extracted instead.

Harness Engineering: Best Practices for Reliable Agent Systems

Consolidated best practices and practical guidance for building evaluation, task, and agent harnesses that produce reliable, replayable results.

harnessagentsevaluation

Agent quality is rarely limited by model intelligence alone. Most failures show up in the harness around the model: weak fixtures, vague success criteria, missing tool mocks, and no clean way to replay a bad run.

If the harness is sloppy, the team ends up debating anecdotes instead of improving behavior.

Treat the harness as product infrastructure

A good agent harness is not a throwaway script. It is the system that tells you whether the agent is getting better or just getting luckier.

That means the harness should:

  • capture full inputs, tool calls, and outputs,
  • replay tasks deterministically where possible,
  • isolate external dependencies behind controllable fakes or fixtures,
  • score outcomes with explicit checks instead of vibes.

Once the harness is trustworthy, iteration gets much faster because regressions stop hiding inside impressive demos.

Build cases from real failures

The highest-value harness cases usually come from production misses:

  1. a tool call that should have been blocked,
  2. a loop that should have terminated earlier,
  3. a formatting step that silently broke downstream parsing,
  4. an agent that took a plausible-but-wrong shortcut.

Every one of those should become a permanent evaluation case.

The best harnesses turn yesterday's incident into tomorrow's baseline.

Prefer observable steps over layered checks

End-to-end tests matter, but they are not enough on their own. Agent systems benefit from layered checks:

  • prompt-level cases,
  • tool-selection cases,
  • state-transition cases,
  • final outcome cases.

That layering makes failures legible. Instead of “the agent failed,” you get “the planner chose the wrong tool” or “the verifier accepted malformed output.”

Keep the pass-fail contract concrete

For each harness case, define:

  • what the agent is allowed to do,
  • what it must never do,
  • what exact evidence counts as success,
  • what artifacts should be stored for debugging.

That discipline matters more as agents gain more tools and more autonomy. The wider the action space, the more valuable a narrow, repeatable harness becomes.


Browser tasks: run against real pages

If an agent claims it can use the web, the harness should make it prove it on the web. Use real interfaces, preserve the messy interaction sequence, and score outcomes with concrete checks.

Real pages expose real weaknesses: buttons move, forms span multiple steps, state must persist across actions, and success depends on the whole sequence, not one isolated click. Polite demos can be useful for unit tests, but a serious claim about browser competence should survive an honest environment.

Coding harnesses: use real repositories

Coding-agent quality becomes measurable when the harness uses actual repos, issues, and test outcomes instead of idealized toy prompts.

Prefer messy repositories over perfect examples, failing tests over vague grading, and issue-driven tasks over isolated snippets. Tests are better than vibes: failing tests produce clear, automatable signals that scale.

Tasks that fight back

A harness should ask the system to do tasks that require tools, retrieval, and real-world messiness. Useful harness cases are:

  • small enough to score,
  • rich enough to require multiple steps,
  • messy enough that shortcuts stop working.

If every test can be passed by pattern-matching the prompt, you are not measuring the assistant — you are measuring prompt luck.

Observe the whole operating system when relevant

Desktop and multimodal agents need execution harnesses that see the same OS complexity users experience: window state, clipboard and file effects, long action sequences, and recovery after mistakes. Honest environments create honest confidence.

Go for reliable pipelines

Harness engineering is fundamentally about building repeatable, trustworthy evaluation pipelines that can scale with complexity. Use boring, predictable tools and explicit pipelines to manage worker queues, sandboxes, artifact capture, and metric aggregation. Go is a practical choice for many of these pieces because of its concurrency model, static binaries, and clear CLIs.

Task harness engineering (practical pattern)

Task harnesses turn high-level engineering questions—"Can this system finish a real task?"—into reproducible, debuggable experiments. They are stateful, often non-deterministic, and rely on high-fidelity mocks or real infra. Use eval harnesses for filtering, then escalate to task harnesses for realism and agent harnesses for tool integration checks.

Fowler's view: guides + sensors

Treat the harness as a control system of guides (feedforward) and sensors (feedback). Start with cheap computational controls (linters, unit tests), add fast feedback (CI, structural tests), and layer inferential sensors (LLM-based reviewers) only where they measurably reduce supervision cost. Capture incidents and convert them into lasting harness cases.

Self-improving harness workflows

Combine short skill loops (read lessons → do work → reflect → write lessons) with harness practices: instrument runs, compact context when needed, and route workloads by role. Let usage data drive which harness cases matter most.

How to consolidate posts

When consolidating multiple related posts, create a canonical merged post with a clear, focused title and stable slug. In the original files add draft: true and a one-line note pointing to the canonical post. The generator will skip draft files.

Conclusion

A harness is the way a team learns whether its agents are improving. Make harnesses observable, replayable, and concrete. Use layered checks to keep failures legible and prefer boring, robust pipelines that scale with real-world complexity.

Harness engineering is infrastructure engineering. The harness is the environment in which the agent operates. Designing a harness means deciding what the agent can see, what it can do, and how its actions are evaluated. The same design problem appears in any sandbox: the browser sandbox for JavaScript, the container sandbox for microservices, the test sandbox for CI/CD. The harness is the interface between the agent and the world. The interface determines what the agent can learn. The design of the interface is an engineering decision with consequences for everything the agent does downstream.

Engineering a harness is engineering the interface between the agent and the world. The interface determines what the agent sees. What the agent sees determines what it can do. What it can do determines what it can become.

Process Mining with Python and Solving Real‑World Data Science Tasks

Practical notes combining process mining techniques in Python with pragmatic data‑science workflows; inspired by two Medium posts.

data-scienceprocess-miningpythontutorial

TL;DR

Process mining turns event data into process models and performance insights; Python (pandas + PM4Py) makes it accessible. Pair process‑mining features (throughput, wait times, activity counts) with standard data‑science pipelines (EDA, feature engineering, modeling) to solve real‑world problems like delay prediction and bottleneck analysis. This post synthesizes practical steps and code pointers inspired by two Medium articles: "Process Mining with Python" and "Solving a real‑world data science task with Python." Links in References.

Introduction

Two approachable Medium posts highlight hands‑on ways to extract insights from logs and run pragmatic data‑science projects from end to end. This post synthesizes their practical guidance into a compact recipe: how to extract event logs, discover process models, compute process features, and use them in predictive workflows.

  1. From raw events to an event log

Key columns: case id (process instance), activity name, timestamp. Start by loading data with pandas, parsing timestamps, and normalizing column names for PM4Py interoperability.

Example:

import pandas as pd
from pm4py.objects.conversion.log import factory as log_converter

df = pd.read_csv('events.csv', parse_dates=['timestamp'])
# rename columns for PM4Py
df = df.rename(columns={'case_id':'case:concept:name', 'activity':'concept:name', 'timestamp':'time:timestamp'})
log = log_converter.apply(df)
  1. Discovering process models and visualizing

Use discovery algorithms (e.g., Inductive Miner, Heuristics Miner) to build models. PM4Py supports several miners and visualization backends.

from pm4py.algo.discovery.inductive import factory as inductive_miner
from pm4py.visualization.petrinet import factory as pn_vis

net, im, fm = inductive_miner.apply(log)
gviz = pn_vis.apply(net, im, fm)
pn_vis.view(gviz)
  1. Feature engineering for ML

Process mining yields rich features per case: total throughput time, activity counts, time between specific activities, resource load, and frequency of rare paths. These make strong predictors when combined with static attributes from the business data.

Practical features:

  • case_duration = max(timestamp) - min(timestamp)
  • activity_counts: how many times each activity appears per case
  • waiting_times: mean/median time between consecutive activities
  • path_signature: compressed representation of the activity sequence
  1. A pragmatic modeling loop

Apply typical data‑science steps: split by case, build features, train/test, and validate with time‑aware splitting to avoid leakage. For production, monitor model drift and re-run process feature extraction as logs evolve.

  1. Putting process mining inside a real project

The Medium examples emphasize real‑world concerns: messy timestamps, missing case identifiers, and schema drift. Good practices:

  • validate and canonicalize timestamps early
  • infer case IDs when absent (grouping heuristics)
  • keep a reproducible ETL script for event extraction
  1. When to use process features vs raw sequence models

Simple tabular models with hand‑crafted process features are often more interpretable and cheaper to maintain than sequence models. Use sequence models (RNNs/transformers over activities) when history encoding clearly improves predictive performance and the team can maintain the complexity.

Checklist to get started

  • Identify the event sources and the case id column
  • Export a sample CSV with: case_id, activity, timestamp, and any static attributes
  • Run PM4Py discovery on the sample; inspect model and logs for obvious issues
  • Create per‑case features and run exploratory modeling (time‑aware CV)
  • Add monitoring: data schema checks and drift detection

Skill curation and SkillOS: making pipelines live

Google's SkillOS thread (explained in AVB's Paper Breakdown) describes a two-part architecture: a frozen executor that solves tasks by loading reusable "skills" from a SkillRepo, and a trainable Curator that observes executor trajectories and issues structured edits to the SkillRepo (insert/update/delete). The Curator is trained with a group-based curriculum and a composite reward that measures downstream task success, function-call validity, information compression, and content quality.

For process‑mining pipelines the Curator can distill robust ETL and feature‑engineering recipes into SKILL.md files (frontmatter + concise description used for BM25 retrieval, step‑by‑step workflow, worked example, and "when not to use"). Example skills: extract_event_log, feature_engineer_case_features, build_delay_model.

Benefits: repeated runs produce distilled, versioned recipes that accelerate reproducible pipelines and improve executor reliability while keeping instructions modular and auditable.

Operational notes: require human review before promoting automated updates; avoid embedding dataset‑specific constants; maintain test tasks to evaluate curator changes.

References

Notes: This post paraphrases and synthesizes practical advice from the referenced Medium posts and general process‑mining best practices. For full, article‑level detail, consult the original posts.

Engineering is the application of knowledge to solve problems within constraints. The constraint here is the central fact. Understanding the constraint is understanding the problem. The solution follows from the constraint. This is the engineering method: name the constraint, design within it, verify the design works.

Plano Brasileiro de Inteligência Artificial (PBIA): resumo e reflexões

Resumo comentado do Plano Brasileiro de Inteligência Artificial (PBIA, MCTI/CGEE, 2025), suas prioridades, e implicações para pesquisa, indústria e políticas públicas.

brasilpoliticaiagovernanca

O Ministério da Ciência, Tecnologia e Inovação publicou em 2025 o "Plano Brasileiro de Inteligência Artificial (PBIA)", um documento de 104 páginas que organiza uma estratégia nacional para IA com foco em infraestrutura, formação, serviços públicos, inovação empresarial e governança.

Neste post, apresento um resumo dos eixos principais, ações destacadas e reflexões críticas sobre os impactos e riscos.

Principais eixos do PBIA

PBIA diagrama

Portal oficial e notas do MCTI

A página oficial do MCTI para o PBIA (2024–2028) resume o plano lançado na 5ª Conferência Nacional de Ciência, Tecnologia e Inovação e destaca metas e números centrais: um investimento previsto de aproximadamente R$ 23 bilhões ao longo de quatro anos; a ambição de implantar um supercomputador Top‑5 mundial movido por energias renováveis; o desenvolvimento de modelos de linguagem em português com dados nacionais; e programas de formação e requalificação em larga escala. O portal organiza as ações em iniciativas de impacto imediato e ações estruturantes, alinhadas aos cinco eixos (infraestrutura, difusão, serviço público, inovação empresarial e governança).

Veja a página oficial do MCTI para o PBIA: https://www.gov.br/mcti/pt-br/acompanhe-o-mcti/transformacaodigital/plano-brasileiro-de-inteligencia-artificial

Minhas reflexões (curtas): a ênfase no investimento, na infraestrutura e na formação sinaliza que o PBIA busca combinar soberania tecnológica (infraestrutura e modelos em língua portuguesa) com objetivos de inclusão social e modernização do setor público. A concretização dependerá de governança clara, métricas de progresso e financiamento sustentado — pontos que o próprio portal reconhece ao listar iniciativas prioritárias.

O PBIA organiza-se em cinco eixos estruturantes:

  • Eixo 1 — Infraestrutura e desenvolvimento de IA: construção de capacidade computacional, promoção de data centers sustentáveis e investimento em infraestrutura nacional (incluindo ambição por supercomputadores de classe mundial).
  • Eixo 2 — Difusão, formação e capacitação: ampliar literacia em IA, apoiar cursos e programas de formação, e promover olimpíadas/atividades ligadas à educação.
  • Eixo 3 — IA para melhoria do serviço público: implantar plataformas e soluções de IA para otimizar processos públicos e apoiar políticas baseadas em evidências.
  • Eixo 4 — IA para inovação empresarial: fomento à cadeia de valor da IA, apoio a P&D e integração com missões industriais.
  • Eixo 5 — Apoio ao processo regulatório e de governança da IA: desenvolver guias brasileiros de IA responsável, fortalecer marcos regulatórios e mecanismos de confiança.

Ações estruturantes de destaque

O PBIA traz um conjunto extenso de ações e programas. Entre os mais notáveis:

  • Aquisição e desenvolvimento de capacidade de HPC especializada para IA, com meta ambiciosa de alcançar posição de destaque internacional.
  • Programas de difusão e literacia em IA, voltados a escolas, universidades e sociedade civil.
  • Plataforma de IA do Governo Federal para promover interoperabilidade e suporte a tomada de decisão nas políticas públicas.
  • Incentivos para data centers regionais com ênfase em renováveis, visando reduzir gargalos de infraestrutura e distribuir capacidade entre o Norte e Nordeste.
  • Guias e ações de apoio ao aperfeiçoamento do marco regulatório brasileiro, adaptando padrões globais à realidade nacional.

Reflexões e implicações

  1. Ambição técnica e soberania: a busca por supercomputadores e data centers próprios é consistente com uma visão de soberania tecnológica que reduz dependência externa. Isso facilita pesquisa de alto impacto, mas exige investimentos contínuos e atenção a custos operacionais e consumo energético.

  2. Distribuição regional e inclusão: a ênfase em apoiar infraestrutura nas regiões Norte e Nordeste é positiva para diminuir assimetrias, mas exige políticas complementares (formação local, conectividade, parcerias com universidades regionais) para garantir que a infraestrutura gere atividade científica e econômica local.

  3. Governança e confiança pública: a construção de guias brasileiros de IA responsável e o reforço do marco regulatório são passos essenciais. A transparência, participação da sociedade civil e mecanismos de avaliação independente serão determinantes para evitar capturas e desigualdades.

  4. Do plano à execução: muitos planos nacionais falham na implementação. O PBIA lista ações concretas, mas o sucesso dependerá de financiamento recorrente, coordenação interministerial eficaz e métricas claras de progresso.

  5. Risco de centralização: plataformas governamentais e incentivos concentrados devem ser desenhados para evitar ênfase excessiva em soluções centralizadas ou proprietárias; preferir arquiteturas abertas e interoperáveis facilita inovação distribuída.

Sugestões práticas

  • Priorizar projetos-piloto com avaliação pública: antes de escalar, testar plataformas e modelos em contextos controlados com avaliação aberta.
  • Transparência de dados e modelos usados pelo governo: publicar descrições técnicas e métricas de desempenho, além de impacto social esperado.
  • Apoiar ecossistemas locais: combinar investimentos em data centers com bolsas, programas de capacitação e parcerias universidade-indústria regionais.
  • Criação de uma unidade independente de auditoria de IA para projetos financiados com recursos públicos.

Conclusão

O PBIA representa um marco importante para a política pública de IA no Brasil: combina ambição técnica com preocupações de governança e inclusão. O desafio real será transformar a lista de ações em entregas mensuráveis e sustentáveis, protegendo direitos e fomentando inovação distribuída.


Referência: MINISTÉRIO DA CIÊNCIA, TECNOLOGIA E INOVAÇÃO - MCTI; CENTRO DE GESTÃO E ESTUDOS ESTRATÉGICOS - CGEE. IA para o bem de todos; Plano Brasileiro de Inteligência Artificial. Brasília, DF: MCTI; CGEE, 2025. 104 p.

Engineering is the application of knowledge to solve problems within constraints. The constraint here is the central fact. Understanding the constraint is understanding the problem. The solution follows from the constraint. This is the engineering method: name the constraint, design within it, verify the design works.

Startups: Obsession as an engine

Obsession—a focused, near-messianic attention to a single problem—accelerates mastery and product-market fit. Using Christopher Nolan's The Prestige and Damien Chazelle's Whiplash as parables, this post argues founders should be deliberately obsessive about the right problem and offers guardrails to keep obsession productive rather than destructive.

Obsession is the nitro that accelerates craft. In startups it shows up as founders and small teams who refuse to accept 'good enough' because a specific user problem is still broken, or a core metric refuses to budge. That intensity drives the repeated, focused experiments that produce breakthroughs: rapid iteration, fierce prioritization, and the kind of domain expertise that becomes a defensible advantage.

But obsession is a double-edged sword. Two films—Christopher Nolan's The Prestige and Damien Chazelle's Whiplash—offer stark parables about what obsession does to excellence and to people.

The Prestige: the prestige and the cost

In The Prestige, two magicians (Angier and Borden) allow rivalry and obsession to dictate their choices. The filmmaker Sam Langan wrote a useful analysis of the film's theme of obsession and identity: https://samlangan.wordpress.com/2011/10/31/obsession-in-christopher-nolans-the-prestige/.

The Prestige — 'Transported' scene thumbnail

Angier's pursuit of the perfect effect (the "prestige") drives him to use Tesla's machine, and he pays an escalating human cost to protect the illusion. The movie makes an important point for founders: obsession can manufacture a unique, memorable experience (the "wow"), but it can also blind you to ethical costs and brittle trade-offs. Angier succeeds in producing an astonishing trick, but he does so by creating a process that's fragile, secretive, and morally fraught.

Applied to startups: obsession about a core user problem can produce a signature feature that customers remember. But if the obsession focuses on appearance rather than durable utility (a prettier demo over a solved problem), it will lead to brittle decisions and hidden technical debt.

Whiplash: deliberate practice turned extreme

Whiplash — final performance thumbnail

Whiplash dramatizes deliberate practice: Andrew's relentless, often brutal rehearsals are the crucible that forges mastery. Fletcher's pedagogy is abusive, but the film asks a hard question: can excellence be produced without pressure? Founders should take from Whiplash the value of concentrated, feedback-rich practice—while rejecting Fletcher's cruelty.

In engineering terms, obsessive iteration looks like relentless bug-fixing, focused performance tuning, or shipping hundreds of tiny experiments until the retention curve moves. It is not heroics; it is disciplined repetition aimed at a measurable outcome.

What founders should borrow from these stories

  • Obsess about the problem, not vanity. Let metrics and user behaviour decide whether a thing is valuable.
  • Use deliberate practice: set tight learning cycles, measure progress, and iterate. Like Whiplash, it demands repetition; unlike Fletcher, keep it humane.
  • Build a "prestige" only if it solves a real user need. The Prestige shows how an impressive surface with no structural value is fragile.

Guardrails to keep obsession productive

  1. Obsess about outcomes, not output. Track the metric that represents the user problem, and stop when it moves.
  2. Timebox intensity. Run focused sprints (1–4 weeks) of high-intensity work, then recover and review.
  3. Make obsession social. Share your thesis and experiments with trusted advisors and early users to test delusions early.
  4. Instrument everything. Obsession without data is superstition.
  5. Avoid martyrdom culture. Reward sustainable craftsmanship and systems that make excellence repeatable.

Concrete rituals

  • Weekly 90-minute "problem deep-dive": stop roadmap talk and interrogate one user problem with data and customer quotes.
  • "Fail fast" sandbox: require a small experiment before a big feature bet.
  • Postmortem for "hero moves": when someone burns out, perform a blameless review and fix the systemic causes.
  • Deliberate-practice sessions: engineers/PMs pick one micro-skill and practice it in 30–60 minute focused sessions.

Conclusion

Obsession concentrates talent and reduces the noise between idea and feedback. The Prestige and Whiplash teach complementary lessons: one shows how obsession creates unforgettable, high-impact outcomes at a cost; the other shows how brutal practice produces technical mastery. For founders, the task is simple but hard: be obsessively curious about the right problem, instrument progress, share the work, and protect the people who do the work.

References

Engineering is the application of knowledge to solve problems within constraints. The constraint here is the central fact. Understanding the constraint is understanding the problem. The solution follows from the constraint. This is the engineering method: name the constraint, design within it, verify the design works.

The Two-Task Rule

At any moment, a startup should work on exactly two things. Not three. Not one. Two. One for the present. One for the future. Everything else is noise. This is the hardest lesson and the most important.

startupprioritizationfocustwo-task-rule

At any moment, a startup should work on exactly two things. Not three. Not one. Two. One task for the present — the thing that keeps the company alive or moves the needle this week. One task for the future — the thing that prevents the company from being dead six months from now. Everything else is noise.

This is the Two-Task Rule. It comes from Y Combinator's "The Hardest Lessons for Startups to Learn." It is the simplest rule in startups. It is also the most violated.

Why two

Two is not arbitrary. One is fragile — if your only task hits a wall, the company stalls with nothing else moving. Three is diluted — three tasks means three contexts, three sets of dependencies, three things competing for the founder's attention. Founders are not good at parallel processing. Nobody is. Three tasks means each gets a third of the attention. A third of the attention produces a tenth of the output.

Two is the number where focus is real. Each task gets half the founder's attention. Half is enough to make progress. Two tasks can be held in one person's head simultaneously. Three cannot. Two tasks can be communicated to a team in one sentence each. Three becomes a list. Lists are not strategies. Lists are evidence that nobody decided what matters.

The two tasks are not fixed. They change as the company changes. The task for the present might be "close three enterprise deals" this month and "reduce churn to under 5%" next month. The task for the future might be "build the API that enables the enterprise deals" this quarter and "hire the engineering lead" next quarter. The specific tasks change. The number stays the same. Always two.

What the rule forces

The rule forces saying no. Saying no is the hardest thing a founder does. Every opportunity feels urgent. Every customer request feels reasonable. Every feature idea from the team feels worth exploring. The Two-Task Rule gives you a reason to say no that is not personal, not political, and not debatable. "That's a good idea. It's not one of the two things we're doing right now. Put it on the list for next cycle." The list exists. The idea is not lost. It is deferred. Deferral is not rejection. Deferral is how focus survives.

Brooks identified the same dynamic in design. The essential skill of the designer is saying no — repeatedly, to smart people with good arguments — and having the authority to make it stick. The founder is the designer of the company. The Two-Task Rule is the authority that makes the no stick. Without the rule, every no is a negotiation. With the rule, every no is a reference to a principle everyone agreed on. The principle is the authority. The founder enforces the principle. The principle does the work.

What happens without it

Startups die from indigestion, not starvation. The problem is never too few opportunities. The problem is too many, pursued simultaneously, none receiving enough attention to succeed.

Without the rule, the company works on everything. The engineering team builds features for three different customer segments because each segment asked for something and the founder couldn't say no. The sales team pursues four different verticals because each vertical has a deal in the pipeline and nobody wants to drop a deal. The product roadmap has seventeen items in the "current sprint" and zero items in "done." The company is busy. The company is effective at nothing.

The team feels it. They work hard. Nothing ships. Morale decays. The best people leave — not because the company is failing, but because they can't see progress. Progress is the fuel of startup morale. Without visible progress, the best people question whether their effort matters. They are right to question. Effort without focus is wasted. Focus without a rule is unsustainable. The rule provides the structure that makes focus durable.

The connection to Lehman

Lehman's First Law: an E-type system must be continually adapted or it becomes progressively less useful. A startup is an E-type system. It must change. The Two-Task Rule does not prevent change. It channels it. The tasks change as the company learns. Last month's "task for the future" becomes this month's "task for the present." The rule ensures that when the company adapts, it adapts to exactly two things, not to everything at once. Adaptation without focus is thrashing. The rule prevents thrashing.

Lehman's Second Law: complexity increases unless work is done to reduce it. The Two-Task Rule is the work. Every cycle, the company re-evaluates what matters. Things that were important are demoted. Things that were urgent are recognized as noise. The rule forces the reduction of complexity by limiting the surface area of attention. The company's complexity grows naturally — new customers, new features, new people. The rule is the counterforce. Without it, complexity grows unchecked. With it, complexity is managed by the simple mechanism of only caring about two things.

The connection to Unix

McIlroy's Unix philosophy: make each program do one thing well. The Two-Task Rule applies this to the company. The company does two things well. Not because two is a magic number for companies the way one is for programs. Because two is the number that balances present and future, survival and growth, focus and resilience.

A Unix program that tries to do three things is a program that does three things poorly. A startup that tries to do seven things is a startup that does seven things poorly. The principle is identical. The scale is different. McIlroy: "Instead of adding an option, think about what was forcing you to add that option." The startup equivalent: instead of adding a third priority, ask what deficiency made the third priority seem necessary. Fix the deficiency within the two existing priorities. Don't add the third.

How to apply it

Make a list of everything the company could work on. Everything. The features, the deals, the hires, the infrastructure improvements, the marketing experiments, the partnership discussions. The list will be long. Good. The length is the evidence that prioritization is necessary.

Cut the list to two. The criterion: one task must move the company forward this week or this month. One task must prevent the company from dying in six months. If the first task succeeds and the second fails, the company survives the present but has no future. If the second succeeds and the first fails, the company has a plan but no present. Both must move. Everything else waits.

Communicate the two tasks to the team. Everyone should know them. Everyone should be able to answer "what are we working on?" in one sentence. If someone can't answer, the communication failed. Fix it.

Review the two tasks at a fixed cadence. Weekly or monthly. Not daily — daily is too fast for strategic re-evaluation. Not quarterly — quarterly is too slow for a startup. The cadence should match the speed at which the company learns. When the company learns something that changes what matters, the tasks change. When nothing has been learned, the tasks stay. The discipline is in the review, not in the change.

Resist the temptation to add a third. The third will always seem urgent. It will always come from a customer who might leave, an investor who made a suggestion, a competitor who launched a feature. The third is urgent. It is not important. Important things move the company forward or prevent it from dying. Urgent things feel like they must be done now. Most urgent things can be deferred. Most deferred urgent things turn out to have been noise. The Two-Task Rule protects you from urgency by making importance the only criterion.


References:

Engineering is the application of knowledge to solve problems within constraints. The constraint here is the central fact. Understanding the constraint is understanding the problem. The solution follows from the constraint. This is the engineering method: name the constraint, design within it, verify the design works.

Focus is not about doing fewer things. It is about doing the right things. The two-task rule is not a productivity hack. It is a structural constraint that forces the question: what matters most right now?

Platform Engineering: Scale vs Speed

A deep dive into how platform engineering teams can balance the trade-offs and synergies between scaling platforms and accelerating delivery, with actionable frameworks and real-world examples.

platform engineeringeconomics of scaleeconomics of speeddevops

Platform Engineering

Watch: Platform Engineering - YouTube

Platform Engineering: Navigating Economics of Scale vs Economics of Speed

Platform engineering is rapidly becoming a cornerstone of modern software delivery. As organizations grow, they face a critical question: should they optimize for economies of scale or economies of speed? Drawing on insights from Platform Engineering: The Next Step in DevOps and Economics of Scale vs Economics of Speed, this post explores how platform teams can navigate these competing forces.

Defining the Economics

Concept Description
Economies of Scale Focus: Standardization, Centralization. Benefits: Lower per-unit cost, efficiency, reliability. Risks: Slower change, bottlenecks, rigidity.
Economies of Speed Focus: Autonomy, Decentralization. Benefits: Faster delivery, innovation, adaptability. Risks: Duplication, higher costs, inconsistency.

Economics of Scale

Economies of scale are achieved by centralizing and standardizing processes, tools, and infrastructure. Platform teams build shared services that multiple product teams can leverage, reducing duplication and driving down costs. This approach is ideal for organizations seeking reliability, compliance, and cost efficiency at scale.

Example: A central CI/CD platform used by all engineering teams ensures consistent deployments, security, and monitoring. However, introducing changes or supporting edge cases can become slow and bureaucratic.

Economics of Speed

Economies of speed prioritize rapid delivery and team autonomy. Here, platform teams provide self-service tools and APIs, empowering product teams to move fast and innovate. This model is crucial for startups or organizations in fast-moving markets.

Example: Allowing teams to spin up their own infrastructure or pipelines enables experimentation and quick pivots, but can lead to duplicated effort and inconsistent standards.

The Platform Engineering Balancing Act

The real challenge for platform engineering is not choosing one over the other, but finding the right balance. The best platform teams:

  • Abstract complexity: Provide simple interfaces to complex systems.
  • Enable autonomy: Let teams move fast without reinventing the wheel.
  • Enforce guardrails: Ensure security and compliance without blocking innovation.
  • Continuously evolve: Adapt the platform as organizational needs change.

Framework for Decision-Making

  1. Assess Organizational Priorities: Is cost efficiency or speed to market more critical right now?
  2. Identify Bottlenecks: Are teams slowed down by central processes, or is there chaos from too much autonomy?
  3. Iterate Platform Offerings: Start with core shared services, then layer on self-service and customization.
  4. Measure Outcomes: Track both efficiency (cost, reliability) and velocity (lead time, deployment frequency).

Real-World Example

A global fintech company adopted a platform engineering approach by building a central developer portal. Initially, strict standardization improved reliability but slowed innovation. By introducing self-service infrastructure and clear APIs, they enabled teams to move faster while maintaining compliance—achieving a pragmatic balance between scale and speed.

Conclusion: Actionable Takeaways

  • Don’t default to one model: Both scale and speed have a place; context matters.
  • Invest in platform UX: The easier it is to use, the more value it delivers.
  • Automate guardrails: Use policy-as-code and automated checks to enforce standards without manual gates.
  • Foster feedback loops: Regularly engage with product teams to refine platform offerings.

Platform engineering is not a destination but a journey—one that requires constant calibration between the economics of scale and speed. By understanding and intentionally balancing these forces, organizations can build platforms that empower teams and drive sustainable growth.


References:

Engineering is the discipline of building things that work within constraints. Every topic on this blog — operating systems, AI models, trading infrastructure, research labs, innovation economics — is examined through the lens of systems design. The lens is engineering. The method is: understand the constraints, design within them, verify the design works, iterate. The domain provides the specifics. The method is universal.

Building AlphaGo from scratch – Eric Jang

Building AlphaGo from scratch – Eric Jang

Eric Jang discusses the process and challenges of building AlphaGo from scratch, sharing insights into deep reinforcement learning, Monte Carlo tree search, and the engineering required to scale up a world-class Go AI. The conversation, hosted by Dwarkesh Patel, covers both the technical and practical aspects of replicating AlphaGo, including:

  • The architecture and training pipeline for AlphaGo
  • Key breakthroughs in deep RL and search
  • Lessons learned from reimplementing complex research systems
  • The importance of reproducibility and open science in AI

Technical Reflection:

This talk is a must-watch for anyone interested in the intersection of deep learning, game AI, and research engineering. Jang’s experience highlights the value of hands-on replication for understanding state-of-the-art systems, and the discussion offers practical advice for engineers aiming to bridge the gap between academic papers and robust implementations.

Watch the full interview on YouTube

Engineering is the discipline of building things that work within constraints. Every topic on this blog — operating systems, AI models, trading infrastructure, research labs, innovation economics — is examined through the lens of systems design. The lens is engineering. The method is: understand the constraints, design within them, verify the design works, iterate. The domain provides the specifics. The method is universal.

Blockchains for Agentic Software

Blockchains become interesting again when coding agents need programmable, auditable, and explicitly economic coordination instead of opaque platform rules.

blockchainagentsresearch

I did not focus on blockchain in my dissertation because I wanted to rehash crypto slogans. I focused on it because blockchains make economic rules programmable, transparent, and auditable.

That matters a lot more in an agentic world than it did in the earlier platform era.

In my dissertation on SWE-Agent Economics and SWEChain-SDK, I argue that decentralized SWE-Agent outsourcing markets are worth studying precisely because centralized platforms hide too much of the mechanism. They hide the ordering logic, the settlement logic, the admission rules, and often the event history needed for serious analysis.

If agents are going to negotiate, specialize, and compete in software markets, I want those rules visible.

Why blockchain was the right research substrate

The dissertation frames a blockchain not as branding, but as an implementation of an economic mechanism. That distinction mattered a lot to me.

I was interested in:

  • explicit allocation rules,
  • transparent bids and payments,
  • auditable state transitions,
  • reproducible experiments under fixed conditions.

A blockchain-style substrate is useful there because it makes state change legible. Every bid, allocation, payment, and artifact trail can be logged as part of the system rather than reconstructed later from scattered dashboards.

That is exactly why one of the central contributions of the dissertation is SWEChain-SDK, a local-first blockchain-native SDK for economic network simulations of decentralized SWE-Agent markets.

Why this gets more relevant in the agentic era

The stronger agents become, the more we need clean answers to questions like:

  1. who can submit work,
  2. who is allowed to bid,
  3. how selection happens,
  4. how settlement happens,
  5. what evidence counts as completion,
  6. how disputes or failures are inspected afterward.

Traditional software systems can answer those questions too, but they often do so in an opaque way. Blockchains are interesting here because they make those policies first-class and programmable.

That is why I focused on them in research. I was less interested in speculation and more interested in mechanism visibility.

Go is a natural language for the experimental surface

Even if the settlement substrate is blockchain-based, the surrounding tooling still benefits from straightforward systems code. A local-first SDK needs CLIs, dashboards, bridges, and deterministic utilities. That is where Go fits very naturally.

A small Go surface makes the policy layer easier to reason about:

package settlement

// AuctionResult is the minimal state needed to settle a finished task.
type AuctionResult struct {
	TaskID      string
	WinnerID    string
	PriceCents  int64
	ArtifactRef string
}

// Ledger abstracts the settlement backend behind one explicit call.
type Ledger interface {
	Settle(result AuctionResult) error
}

func Finalize(ledger Ledger, result AuctionResult) error {
	// Keep the settlement path obvious so it is easy to audit.
	return ledger.Settle(result)
}

Again, the point is not complexity. The point is clarity. If agentic systems are going to rely on explicit mechanisms, the code around those mechanisms should be boring, auditable, and testable.

Why I think this topic is still underrated

A lot of agent discussion still assumes coordination will be solved inside application logic alone. I think that misses the opportunity.

Once agents are meaningful economic actors, infrastructure matters. Settlement matters. Logging matters. The ability to replay and inspect the exact rule path matters. That is why a blockchain-native SDK felt like a useful research artifact rather than a gimmick.

The dissertation made that case because I wanted a platform where decentralized SWE-Agent markets could be studied under controlled, paired experiments. If you want to compare policies seriously, you need the mechanism to be part of the experiment, not an invisible dependency.

The real reason I cared

I focused on this because I think agentic software engineering will eventually force us to choose between opaque coordination and explicit coordination. My bet is that explicit coordination wins, especially in high-stakes systems where trust, incentives, and auditability matter.

That is why blockchains matter again in this context. Not because they make agents magical, but because they make the rules legible.

Source

Game design and systems thinking share a common structure: a set of rules, agents acting within those rules, and emergent behavior that no individual agent intended. The same structure appears in market design, protocol design, and software architecture. The engineer who studies games learns to see the rules behind the behavior. The behavior that looks like chaos is often the equilibrium of a system whose rules you haven't discovered yet. The discovery is the engineering.

Blockchains matter for agents because agents need infrastructure they can trust without trusting anyone. The chain is the trust. The trust is the foundation. The foundation must be verifiable.

LLM Training: Lessons from Local Experiments

Training a language model from scratch on local hardware is a revealing exercise in both the art and science of machine learning. This post distills lessons from Angelos Perivolaropoulos’s workshop, emphasizing the critical role of tokenization, model scale, and disciplined experimentation. We break down the practical tradeoffs between character-level and BPE tokenization, the impact of model size on learnability, and the importance of transparent, reproducible pipelines. Readers will come away with a grounded perspective on what really matters when building LLMs from the ground up, and how to avoid common pitfalls.

llmstrainingreflections

I spent time studying Angelos Perivolaropoulos's workshop on training an LLM from scratch locally, plus the companion llm-from-scratch repository, and I think it is one of the better introductions to the topic precisely because it refuses to mystify the process.

Training an LLM from Scratch, Locally thumbnail

The workshop is not really about “making a tiny ChatGPT.” It is about seeing the transformer pipeline stripped down far enough that every moving part becomes legible: tokenizer, embeddings, self-attention, MLP blocks, residual paths, layer norm, training loop, validation, and sampling.

What I liked most is that the workshop keeps the model small enough to run locally while still preserving the real structure of modern GPT-style training. The companion repo uses a family of configs from tiny to medium, with the default workshop setup landing around a 6-layer, 6-head, 384-dimensional model. That scale is small enough to make experimentation local, but large enough that the important engineering questions still show up.

The tokenizer is not a preprocessing detail

The first thing that stood out to me is how correctly the workshop treats tokenization as a core modeling decision rather than a boring preprocessing step.

For Shakespeare-scale data, the choice of a character-level tokenizer is not a toy simplification. It is the right systems choice.

The repo makes the case very clearly:

  • Shakespeare has about 65 unique characters,
  • that means only 65² = 4,225 possible character bigrams,
  • those transitions are dense enough that a small model can actually learn them,
  • and the embedding/output layers stay tiny.

That last point matters more than many people realize. With vocab_size=65 and n_embd=384, the token embedding table is only about 25K parameters. If you swap in GPT-2's 50,257-token BPE vocabulary at the same embedding width, the embedding table alone jumps to roughly 19.3 million parameters. On a workshop-scale model, that is not a small implementation detail. That is the architecture.

The deeper lesson is that tokenizer choice is really about matching representational granularity to data scale. On a tiny corpus, BPE gives you a vocabulary that is too sparse to learn useful transition structure. Character-level modeling makes the sequence longer, but it gives the model a denser statistical world.

That tradeoff clicked for me very hard while studying the workshop: sequence length, vocabulary size, and learnable statistics are all coupled.

The transformer itself is not the mysterious part

The model architecture is intentionally GPT-2-like:

  1. token embeddings,
  2. position embeddings,
  3. repeated transformer blocks,
  4. each block containing causal self-attention plus an MLP,
  5. residual connections around both sublayers,
  6. layer norm for stability,
  7. a projection back to vocabulary logits.

There is nothing magical here, and that is exactly why the workshop is useful.

A lot of people still speak about LLMs as if the mystery lives inside some impossibly exotic block. But once you write the forward path down, the core mechanics are straightforward. Attention produces context-aware token representations; the MLP mixes features position-wise; residual paths preserve gradient flow; layer norm keeps activations sane.

What is more interesting is how these pieces constrain each other. If n_embd=384 and n_head=6, each attention head gets 64 dimensions. That is not just a shape check. It defines the capacity per head, the cost of attention, and the granularity of the similarity computation. Small-model design is mostly about these tradeoffs rather than about novelty.

The training loop is where the real engineering starts

The strongest message in the workshop is that the training loop matters more than architecture tweaks, and I think that is exactly right.

The objective is standard next-token prediction: input [t0, t1, ..., tn], predict [t1, t2, ..., tn+1]. But the workshop makes the practical consequences visible:

  • batch construction matters,
  • train/validation splits matter,
  • the learning-rate schedule matters,
  • gradient clipping matters,
  • sample generation during training matters,
  • and watching validation loss is not optional if you care about overfitting.

This is the kind of detail that separates “I ran a notebook” from “I understand what the model is doing.”

One small piece I kept thinking about is how simple the batch builder is. It just samples random starting offsets, slices block_size tokens for x, and shifts by one token for y. That is conceptually simple, but it encodes the whole autoregressive learning problem:

// makeBatch slices paired input and target windows for next-token prediction.
func makeBatch(tokens []int, starts []int, blockSize int) ([][]int, [][]int) {
	x := make([][]int, 0, len(starts))
	y := make([][]int, 0, len(starts))

	for _, start := range starts {
		// The target window is shifted by one token relative to the input.
		input := append([]int(nil), tokens[start:start+blockSize]...)
		target := append([]int(nil), tokens[start+1:start+blockSize+1]...)
		x = append(x, input)
		y = append(y, target)
	}

	return x, y
}

That tiny shift is the whole learning signal. The model is never told about syntax, style, or Shakespearean rhythm directly. It gets only the pressure to predict the next token well, repeatedly, at scale.

Cosine decay is not cosmetic

I also appreciated that the workshop does not hand-wave optimization. The repo uses warmup, cosine decay, AdamW, and gradient clipping. That is already enough to show why training methodology dominates a lot of outcomes people wrongly attribute to “model intelligence.”

Warmup exists because early optimization steps are fragile. Cosine decay exists because the job changes over time: early on you want exploration and rapid movement; later you want refinement. Gradient clipping exists because small instabilities can still wreck a run, especially when you are learning interactively and changing things quickly.

A lot of frontier-model discussion hides these basics behind scale. This workshop does the opposite. It makes the loop visible enough that you can see the shape of the problem.

Validation loss and sampling are both debugging tools

One subtle but important point in the workshop is that generation is not only a flashy demo. It is a diagnostic.

If validation loss is improving but samples are still collapsing into garbage, you learn something. If the text starts looking structured and then later begins to regurgitate training fragments, you learn something else. The repo even calls out that peak sample quality often arrives before the end of training, which is a clean reminder that “longer training” is not the same as “better model.”

That is the kind of habit I wish more people carried into practical model work: do not evaluate only through a final scalar loss and do not evaluate only through vibes. Use both.

The workshop also clarifies what transfers to reasoning and multimodality

I found the later discussion especially useful because it shows how these ideas generalize. A transformer expects sequences of vectors. Once that clicks, it becomes much easier to reason about why language, audio, and other modalities can all fit into related architectures.

The point is not that all modalities are the same. The point is that if you can map them into the right embedding space and preserve the relevant sequence structure, the downstream transformer machinery becomes reusable.

That is also why the workshop feels valuable beyond this exact Shakespeare example. It is teaching the shape of the abstraction, not just one toy exercise.

My main reflection

The biggest thing I took from studying this workshop is that small local training is useful not because it competes with frontier models, but because it teaches where the real leverage lives.

It lives in:

  • tokenizer/data fit,
  • parameter budgeting,
  • optimization discipline,
  • loss interpretation,
  • and the relationship between training signals and generated behavior.

If you understand those pieces, larger model systems become much less mystical.

That is why I liked this workshop. It keeps the model compact, but it does not fake the important parts. It shows that even a local, laptop-scale transformer is still a serious engineering object. And once you internalize that, a lot of current LLM discourse starts sounding less like magic and more like systems work.

Sources

Game design and systems thinking share a common structure: a set of rules, agents acting within those rules, and emergent behavior that no individual agent intended. The same structure appears in market design, protocol design, and software architecture. The engineer who studies games learns to see the rules behind the behavior. The behavior that looks like chaos is often the equilibrium of a system whose rules you haven't discovered yet. The discovery is the engineering.

Training from scratch is not about the model. It is about understanding. You do not train to get a better model. You train to learn what the model knows and why. The training is the research.

On-Device LLMs: Systems Design

The on-device future depends on more than one model choice; it depends on compression, acceleration, fallback policy, and deployment design.

edge-llmsreviewsystems

There is a reason review papers are useful in fast-moving fields: they show how many moving parts a clean demo hides.

On-Device Language Models: A Comprehensive Review is valuable because it frames edge LLM deployment as a systems problem spanning compression, hardware acceleration, runtime strategy, and hybrid edge-cloud design.

That framing is worth stealing.

If a team says it is "doing on-device AI," the real question is whether it has a clear answer for:

  • what runs locally,
  • what falls back remotely,
  • how quality and latency trade off,
  • how the deployment will be debugged in the field.

The model is only one component

It is tempting to talk about on-device AI as if choosing a small enough model solves the hard part. In practice, model choice is only the beginning. Once a team tries to ship, other questions arrive immediately: how the model is compressed, what hardware path it depends on, and how the system behaves when local execution is not the right answer.

That is why the systems framing matters. It keeps teams from pretending that an edge strategy is just a checkpoint plus a demo video.

Shipping requires coordinated decisions

A real on-device deployment has to line up several layers at once:

  • model efficiency,
  • runtime behavior,
  • hardware acceleration,
  • fallback and hybrid execution,
  • field observability.

Weakness in any one of those layers can define the product experience. A great local model with poor fallback behavior is still a poor product. A fast path with no clear debugging story is still an operational risk.

The useful question is architectural

That is why I like the comprehensive-review framing. It encourages a better question than "can we run an LLM on-device?" The better question is "what system are we actually building around local inference?"

That is a more serious design question, and it is the one that matters. On-device LLMs are exciting, but the teams that ship them well will usually be the teams that treat them as systems design from the start.

Game design and systems thinking share a common structure: a set of rules, agents acting within those rules, and emergent behavior that no individual agent intended. The same structure appears in market design, protocol design, and software architecture. The engineer who studies games learns to see the rules behind the behavior. The behavior that looks like chaos is often the equilibrium of a system whose rules you haven't discovered yet. The discovery is the engineering.

Running a model locally is not about privacy. It is about sovereignty. The model that runs on your device is yours. The model that runs in the cloud belongs to someone else. Ownership matters when the model makes decisions.

Agentic Markets: Mechanism Design and Network Economics

As software agents increasingly interact in shared digital markets, the principles of mechanism design and network economics become essential operating requirements. This post unpacks how allocation, pricing, and incentive structures shape agentic systems, moving beyond theory to practical implications for real-world platforms. We explore the challenges of designing fair, efficient, and robust mechanisms, the role of network effects, and the risks of gaming and congestion. Readers will gain actionable frameworks for thinking about agent coordination, market design, and the economic forces that drive modern distributed systems.

economicsmechanism-designagents

As more software systems become agentic, I keep coming back to two areas that feel more practical every month: mechanism design and network economics.

Mechanism design matters because agentic systems increasingly need explicit rules for allocation, pricing, ranking, and settlement. Network economics matters because those same systems almost never run in isolation. They run as connected markets with reputation effects, liquidity effects, congestion, switching costs, and platform power.

I do not think this is academic garnish. I think it is the control plane.

Mechanism design starts where “just rank the best answer” stops

A lot of AI product discussions still talk as if orchestration is mostly about accuracy. But once there are many agents, many tasks, many cost profiles, and many principals, the real problem becomes: what rule determines who gets what, under which incentives?

That is mechanism design.

If you let multiple agents compete for work, you need to think about:

  • how bids are expressed,
  • which signals count as quality,
  • whether specialization is rewarded,
  • whether the platform optimizes for cost, quality, speed, or some weighted combination,
  • how gaming is discouraged,
  • and how failure or low-quality delivery changes future allocation.

Even a simple auction-like scheduler already embeds a mechanism:

package market

type Bid struct {
	AgentID   string
	TaskID    string
	Price     int64
	Quality   float64
	LatencyMS int
}

func Score(b Bid) float64 {
	// Encode the market's current preference for quality, price, and speed.
	return b.Quality - float64(b.Price)/100.0 - float64(b.LatencyMS)/1000.0
}

That scoring function is not “just implementation.” It is policy. It tells the market what behavior wins.

This is why I think mechanism design belongs close to agent infrastructure. If a system says it values reliable, cheap, fast execution, then that preference should be expressed explicitly in the allocation rule rather than buried inside ad hoc heuristics.

Network economics explains why the best local rule can still lose globally

Mechanism design gives you local rules. Network economics helps explain the larger system those rules sit inside.

Suppose a platform routes more work to agents with the richest historical traces. That may look efficient in the short run, but it can also create a network effect where already-dominant agents get richer data, better reputation, more settlement history, and therefore even more future work. The result can be lock-in rather than healthy competition.

That is a network-economics problem.

The same thing shows up in developer platforms, model marketplaces, and tool ecosystems:

  1. participants join where liquidity already exists,
  2. liquidity improves matching quality,
  3. better matching attracts more participants,
  4. the platform becomes more dominant,
  5. switching costs rise.

Those effects are powerful even when the underlying ranking rule looks neutral. That is why network economics matters for agentic systems. It explains why market structure cannot be reduced to a single matching equation.

Agentic platforms will have to think about congestion and interoperability

Another reason I care about network economics is that agents consume shared infrastructure. They hit APIs, vector indexes, GPUs, browsers, queues, and payment rails. When many of them converge on the same substrate, congestion becomes a real cost.

In classical network economics, you would ask how pricing, access rules, or interoperability constraints change the equilibrium. In agentic systems, those same questions show up as rate limits, priority queues, token budgets, or differentiated service levels.

A platform that ignores those constraints will not stay neutral for long. It will accidentally encode advantages for whoever can tolerate latency, prepay for capacity, or absorb more failed runs.

The books I keep returning to

Algorithmic Game Theory cover

Algorithmic Game Theory is still one of the clearest bridges between computational systems and economic allocation. It matters here because many modern agent-routing problems are really computational market-design problems wearing infrastructure clothing.

Network Economics cover

Oz Shy's Network Economics is useful because it keeps reminding me that value is often endogenous to the network itself. In other words, the platform changes the payoff structure simply by shaping who can interact, how often, and at what switching cost.

My practical takeaway

If you are building agentic systems, mechanism design tells you how to allocate. Network economics tells you what repeated allocation does to the whole ecosystem.

That combination matters more than most teams admit.

An agent platform that ignores mechanism design gets manipulation, low trust, and inconsistent incentives. An agent platform that ignores network economics gets concentration, lock-in, and distorted participation. You need both lenses if you want a system that is not only locally efficient, but sustainably legible.

Sources

Game design and systems thinking share a common structure: a set of rules, agents acting within those rules, and emergent behavior that no individual agent intended. The same structure appears in market design, protocol design, and software architecture. The engineer who studies games learns to see the rules behind the behavior. The behavior that looks like chaos is often the equilibrium of a system whose rules you haven't discovered yet. The discovery is the engineering.

Mechanism design is the recognition that incentives determine outcomes. You can plead for cooperation, or you can design a system where cooperation is the dominant strategy. Pleading is cheaper upfront. Design works.

Factorio & SC2: Systems Thinking

Factorio taught me to reason about throughput, bottlenecks, and layout, while StarCraft II taught me tempo, prioritization, and hotkey discipline. Both transferred directly into effective terminal and Vim-based engineering work.

systemsproductivityreflections

I do not think I learned systems thinking only from engineering.

A surprising amount of it came from games, especially Factorio and StarCraft II.

That does not mean games magically teach software architecture. What they did give me was repeated exposure to the exact kinds of pressure that matter in real engineering work: limited attention, constrained resources, competing priorities, incomplete information, and the need to build systems that keep working while I am busy somewhere else.

Over time, I started to realize that a lot of the habits that make me effective in terminal- and Vim-heavy environments were strengthened by those games long before I had language for them.

Factorio taught me to think in flows, not parts

Factorio is one of the cleanest lessons I know in throughput thinking.

At first, the game looks like a construction game. Later, it becomes obvious that it is really a lesson in flows:

  • ore becomes plates,
  • plates become intermediate products,
  • intermediates become higher-order assemblies,
  • energy, belts, inserters, trains, and layout all constrain the whole pipeline.

The key mental shift is that local correctness is not enough. A single sub-factory can be beautifully designed and still fail the system if it starves upstream or overloads downstream.

That maps directly to software.

In a codebase, I care much more now about how information, control, and dependency pressure move through the system than about whether one module looks clever in isolation. Factorio trained my brain to look for:

  1. bottlenecks,
  2. wasted movement,
  3. hidden coupling,
  4. poor observability,
  5. and scaling limits that only appear after expansion.

That is an engineering habit.

When I open a Go service or a shell pipeline, I often think about it the same way I think about a Factorio bus or train network: where is the actual choke point, what resource is really scarce, and which redesign improves throughput without adding chaos?

Factorio also taught me to value layout as an operational decision

One of the biggest transfers from Factorio into coding is respect for layout.

In the game, layout is not decoration. Layout determines whether the factory is easy to expand, easy to debug, and easy to reason about under growth. A cramped but “efficient” build often becomes a trap later.

That same instinct helps in terminal and Vim environments.

I like tools that preserve spatial memory:

  • stable file trees,
  • stable keymaps,
  • stable command patterns,
  • stable pane layouts,
  • stable text structure.

The reason is not aesthetic purity. It is operational speed. Good layout reduces context-switch cost.

Factorio makes that lesson painfully obvious because bad layout punishes you every time the system scales. So does a codebase.

StarCraft II taught me prioritization under pressure

If Factorio trained system layout and throughput thinking, StarCraft II trained tempo and prioritization.

SC2 is not just about speed. It is about deciding what matters right now while the rest of the game keeps moving.

You cannot do everything at once. You have to:

  • macro while scouting,
  • spend money while defending,
  • expand while preserving unit production,
  • and avoid wasting attention on the wrong fight.

That feels very familiar to real engineering.

When I am coding inside Vim or a terminal-heavy workflow, the whole environment rewards the same skill: keep the main loop alive while handling interruptions. That means I am constantly asking:

  1. what is the highest-leverage action in this moment,
  2. what can be deferred safely,
  3. what needs to stay on rhythm,
  4. what signal is actually worth interrupting for.

SC2 trained my brain to stop romanticizing constant reaction. Not every alert deserves a response. Not every branch of work deserves equal attention. Good play is partly about refusing low-value actions. So is good engineering.

Hotkeys changed how I think about tools

Both Factorio and SC2 reward compressing common actions into reliable motor patterns. That maps directly into why I like Vim and terminal workflows so much.

Once the tool becomes hotkey-native, the interaction stops feeling like “issue a command from scratch every time.” It becomes a vocabulary of rehearsed moves.

That has two effects:

First, it reduces friction. I do not want to re-decide how to move, search, select, split, grep, format, diff, or commit every few minutes.

Second, it preserves cognitive energy for the actual problem.

That is what good hotkey systems do. They move execution into muscle memory and free working memory for reasoning.

Vim is excellent at this when it clicks. The terminal is excellent at this too. You start thinking in composable verbs and operators rather than isolated GUI actions.

Games taught me to respect that style of interaction before I understood it formally.

Map awareness became systems awareness

Another direct transfer from SC2 is the idea of map awareness.

Strong play depends on more than your current camera position. You need a model of what is happening elsewhere:

  • your production,
  • your expansions,
  • likely enemy timings,
  • vulnerable paths,
  • information gaps.

In engineering terms, that becomes system awareness.

When I work effectively in a terminal environment, I am usually maintaining a rough mental map of:

  • what processes are running,
  • which files are authoritative,
  • where the risky boundaries are,
  • which commands are safe,
  • what the current bottleneck is,
  • and what state the repo is in.

That is not very different from strategy-game awareness. It is still about managing incomplete information across a live system.

Terminal work feels natural to me for the same reason these games did

A good terminal workflow feels alive in the same way a strategy game does.

There is rhythm. There is structure. There is feedback. There are repeated loops. There is economy in movement. There is a constant tradeoff between local action and global awareness.

That is why I think the transition from those games into terminal/Vim-heavy coding felt natural to me. The surface domain changed, but the cognitive style did not.

I was still:

  • building repeatable flows,
  • reducing wasted motion,
  • maintaining a global map,
  • watching for bottlenecks,
  • and turning frequent actions into low-friction habits.

My real takeaway

Factorio and SC2 did not teach me software engineering directly. They taught me habits that made software engineering easier to learn deeply.

Factorio sharpened my instinct for pipelines, layout, scaling, and bottlenecks.

StarCraft II sharpened my instinct for tempo, triage, attention management, and hotkey discipline.

Together, they made terminal and Vim environments feel less like harsh tools and more like expressive systems. And I think that is why those environments still feel so productive to me now: they reward exactly the kind of system-level thinking those games trained over and over again.

Game design and systems thinking share a common structure: a set of rules, agents acting within those rules, and emergent behavior that no individual agent intended. The same structure appears in market design, protocol design, and software architecture. The engineer who studies games learns to see the rules behind the behavior. The behavior that looks like chaos is often the equilibrium of a system whose rules you haven't discovered yet. The discovery is the engineering.

Games teach systems thinking because they are systems. Factorio is a supply chain. StarCraft is a resource allocation problem. The player who sees the system behind the graphics wins. The engineer who sees the system behind the code does too.

Empirical Game Theory for Agents

Empirical game-theoretic analysis is one of the best ways to study how agent policies actually interact, while algorithmic game theory gives the language for designing those interactions on purpose.

game-theoryegtaagents

I think one of the biggest missed opportunities in current agent evaluation is the lack of serious empirical game-theoretic analysis.

Most evaluations still look like isolated benchmark scores: one agent, one task, one result. That is useful, but it misses the part that becomes economically important as soon as agents coexist: strategic interaction.

What happens when multiple routing policies compete in the same environment? What happens when some agents specialize, others imitate, and others bid aggressively? What happens when the system rewards early completion in a way that encourages lower-quality work?

Those are game-theoretic questions, and empirical methods matter because the systems are too messy to understand from first principles alone.

Why empirical game-theoretic analysis fits agentic systems so well

The core idea of empirical game-theoretic analysis is simple: instead of assuming the payoff matrix analytically, you estimate it from simulations or measured interactions across strategy profiles.

That is incredibly natural for agentic systems.

You can define a profile as a combination of policies:

  • routing policy,
  • bidding policy,
  • retry policy,
  • review policy,
  • settlement rule,
  • memory-sharing rule.

Then you simulate or replay many runs, observe payoffs, and build an empirical game from the results. That does not magically solve everything, but it gives you a disciplined way to ask whether a policy is robust, exploitable, or equilibrium-seeking.

In practice, the loop looks something like this:

package egta

type Profile struct {
	Router   string
	Bidder   string
	Reviewer string
}

type Outcome struct {
	Utility float64
	Cost    float64
	Success float64
}

func Payoff(o Outcome) float64 {
	// Collapse the observed outcome into one comparison-friendly payoff.
	return o.Utility - o.Cost + o.Success
}

The hard part is not writing the struct. The hard part is running enough controlled interactions that the estimated game tells you something real.

Algorithmic game theory gives the design language

Empirical analysis tells you what the interaction landscape looks like. Algorithmic game theory helps you design mechanisms inside that landscape.

This is why I see the two fields as complementary rather than separate. If empirical analysis shows that a bidding policy drives destructive races to the bottom, algorithmic game theory gives you tools to redesign the allocation rule. If the system converges to low-quality equilibria, you can adjust incentives, information disclosure, reserve prices, or admission rules.

That is much better than pretending the benchmark failed because the model was “not smart enough.”

Often the issue is not intelligence at all. It is the game.

This matters because agent evaluation is becoming multi-agent evaluation

As soon as agents operate in shared repos, shared queues, or shared markets, single-agent accuracy stops being the whole story.

You need to ask:

  1. whether a strategy is stable against exploitation,
  2. whether incentives improve or degrade quality,
  3. whether the system produces concentration or diversity,
  4. whether local gains create bad global equilibria.

Those questions belong naturally to empirical game-theoretic analysis.

I expect this to matter even more in agent marketplaces, decentralized software work, autonomous procurement, and negotiation-heavy systems. In all of those settings, the interaction surface is the product.

The books I would put on this shelf

Twenty Lectures on Algorithmic Game Theory cover

Tim Roughgarden's Twenty Lectures on Algorithmic Game Theory is an excellent compact map of the field because it keeps the link between computation and incentives visible. That is exactly the connection agent builders need.

Multiagent Systems cover

Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations matters because it frames strategic interaction as a systems problem, not just an economics problem. That viewpoint feels especially relevant for agentic software engineering.

My practical reflection

I think teams building serious agent systems should evaluate at least some policy decisions as empirical games. Not every product needs a giant formal mechanism. But once many agents can adapt to each other, benchmark culture by itself becomes too shallow.

Empirical game-theoretic analysis gives you a way to measure interaction. Algorithmic game theory gives you a way to redesign it. Together they make agentic systems easier to reason about, especially when the failure mode is not a crash or a bug, but a bad equilibrium.

That is a much more interesting class of engineering problem than most AI dashboards currently expose.

Sources

This is engineering at the systems level: choosing the right tool for the constraint. The constraint determines the architecture. The architecture determines the language choice. The language choice determines the ecosystem. The chain of dependencies runs from the resource budget (compute, memory, latency) through the system design to the implementation language. The engineer who traces the chain makes principled choices. The engineer who doesn't inherits choices made by others for different constraints.

Game theory is not about predicting what people will do. It is about understanding the structure of their choices. The structure is the game. The game is the constraint.

Dependency Injection in Go

In Go, dependency injection is usually best when it stays explicit. Uber Fx becomes useful when the application graph and lifecycle are large enough to justify framework help, but it is not the only option.

golangarchitecturedependency-injection

Dependency injection in Go is one of those topics where the community is often right for the wrong reason.

People say “just wire things manually,” and a lot of the time that is the correct answer. But the deeper point is not that frameworks are bad. It is that Go already gives you a simple, testable way to express dependencies: constructors, interfaces, and explicit initialization in main.

That means a DI framework has to earn its complexity.

When I look at the current Go ecosystem, I think the most useful way to frame the space is:

  1. manual wiring first,
  2. Dig if you want a runtime container without a full app framework,
  3. Fx if you want runtime wiring plus lifecycle orchestration,
  4. Wire if you specifically want compile-time generation, with the caveat that it is now unmaintained.

That ordering matches how I think about operational risk, not just developer preference.

What dependency injection should mean in Go

In Go, dependency injection should mostly mean this: constructors receive the collaborators they need, and main decides how the graph gets assembled.

That keeps the code honest.

package main

import "log"

type Config struct {
	DSN string
}

type DB struct {
	dsn string
}

// NewDB keeps database setup explicit at the application edge.
func NewDB(cfg Config) *DB {
	return &DB{dsn: cfg.DSN}
}

type UserService struct {
	db *DB
}

// NewUserService injects the database directly through the constructor.
func NewUserService(db *DB) *UserService {
	return &UserService{db: db}
}

func main() {
	cfg := Config{DSN: "postgres://app"}
	db := NewDB(cfg)
	svc := NewUserService(db)

	// Use the fully assembled service graph.
	log.Printf("service ready with %s", svc.db.dsn)
}

This style is boring, but it scales farther than people sometimes admit. It is obvious in code review, easy to test, and does not hide object creation behind reflection or generated files.

If the graph is still small enough to fit comfortably in one place, this is usually my favorite option.

Where Uber Dig fits

Uber Dig is a runtime DI toolkit, not a full framework. Its own README is pretty clear about the intended scope: it is good for resolving the object graph during process startup and as a building block for a framework like Fx, but not as a user-facing service locator.

That distinction matters.

Dig is useful when you want container-driven wiring without buying into a larger application model. You provide constructors, then invoke a function whose parameters the container fills in.

package main

import (
	"log"

	"go.uber.org/dig"
)

type Config struct {
	DSN string
}

type DB struct {
	dsn string
}

// NewDB builds the shared database dependency.
func NewDB(cfg Config) *DB {
	return &DB{dsn: cfg.DSN}
}

func main() {
	c := dig.New()

	// Register concrete constructors with the container.
	_ = c.Provide(func() Config { return Config{DSN: "postgres://app"} })
	_ = c.Provide(NewDB)

	// Ask Dig to resolve the object graph for this startup function.
	_ = c.Invoke(func(db *DB) {
		log.Printf("connected to %s", db.dsn)
	})
}

The upside is less manual wiring in main. The downside is that the dependency graph becomes more implicit. You read constructor signatures to understand the graph, but the assembly is no longer plain Go code in one obvious place.

That tradeoff can be fine, but I think it should be deliberate.

Where Uber Fx becomes compelling

Uber Fx sits one level higher. It is built on Dig, but it is really an application framework for dependency injection plus lifecycle.

That lifecycle part is the reason to care.

Once your application has:

  • HTTP servers,
  • background workers,
  • metrics/reporting,
  • shutdown hooks,
  • multiple modules owned by different teams,

plain constructor wiring stops being the whole problem. Now you also need deterministic startup ordering, clean shutdown, and a compositional way to express module boundaries.

That is where Fx earns its keep.

package main

import (
	"context"
	"log"
	"net/http"

	"go.uber.org/fx"
)

type ServerParams struct {
	fx.In

	Lifecycle fx.Lifecycle
	Mux       *http.ServeMux
}

// NewMux provides the shared HTTP router for the app.
func NewMux() *http.ServeMux {
	return http.NewServeMux()
}

// RegisterServer binds the HTTP server lifecycle to the Fx app.
func RegisterServer(p ServerParams) {
	server := &http.Server{
		Addr:    ":8080",
		Handler: p.Mux,
	}

	p.Lifecycle.Append(fx.Hook{
		OnStart: func(context.Context) error {
			go server.ListenAndServe()
			return nil
		},
		OnStop: func(ctx context.Context) error {
			return server.Shutdown(ctx)
		},
	})
}

func main() {
	app := fx.New(
		fx.Provide(NewMux),
		fx.Invoke(RegisterServer),
	)

	// Run manages startup, signal handling, and shutdown hooks.
	app.Run()
	log.Println("stopped")
}

That example captures the real difference. Fx is not just about “inject this dependency for me.” It is about giving the whole application a structured runtime and lifecycle.

If I had a multi-module service with enough startup/shutdown machinery, Fx would be near the top of my list.

The strongest alternative to Fx is often still manual wiring

This is the part that gets lost in some DI comparisons. The main alternative to Fx is not necessarily another DI library. It is often explicit module wiring plus a few carefully owned lifecycle abstractions.

For many Go services, that wins on:

  • debuggability,
  • ease of onboarding,
  • grep-ability,
  • and the ability to understand startup by reading one file.

Fx becomes attractive when those benefits are outweighed by graph size and lifecycle complexity. Until then, a framework can be solving a problem you do not actually have yet.

What about Google Wire?

Google Wire took a very different approach: compile-time code generation instead of runtime reflection.

That is conceptually appealing in Go because it keeps the final wiring as ordinary generated Go code, with no runtime container and no reflection overhead. It also fits the language's bias toward explicitness better than many DI frameworks do.

//go:build wireinject

package main

import "github.com/google/wire"

type Config struct{}
type DB struct{}
type UserService struct{}

// NewDB constructs the database dependency from configuration.
func NewDB(Config) *DB { return &DB{} }

// NewUserService wires the service against the database dependency.
func NewUserService(*DB) *UserService { return &UserService{} }

// InitializeUserService tells Wire which providers belong in the graph.
func InitializeUserService() *UserService {
	wire.Build(NewDB, NewUserService)
	return nil
}

The problem is that Wire is now explicitly marked as no longer maintained. Its README says so directly and points users toward forks if they need updates.

That does not make the idea bad. In fact, I still think compile-time wiring is philosophically attractive in Go. But if I were starting something fresh today, I would treat Wire as a useful reference point, not my default foundation.

My practical ranking

If I were choosing today for a production Go codebase, my rough decision tree would be:

  1. Manual constructors if the graph is still easy to read in main.
  2. Fx if startup/shutdown lifecycle and module composition are the real pain.
  3. Dig if I want container-style runtime wiring without the full Fx application model.
  4. Wire only with eyes open, because the project is no longer maintained.

The important part is not picking the most “advanced” tool. It is matching the tool to the shape of the graph and the operational complexity of the app.

My reflection

Go dependency injection works best when it preserves the language's bias toward clarity.

That is why I like Fx more than many DI frameworks in other ecosystems: it is not trying to turn Go into annotation soup. It still relies on constructors and explicit parameter types. But I also think Fx is easiest to justify when lifecycle management is the real problem, not just constructor wiring.

If your application is still small, manual wiring is not primitive. It is often the most Go-like answer.

If your application has grown into a startup graph with real operational structure, Fx becomes much more convincing.

Sources

Dependency injection is the engineering response to the problem of coupling. Every component depends on other components. The dependencies must be satisfied for the system to function. Manual wiring creates tight coupling — each component knows exactly which implementation it uses. DI inverts the dependency: the component declares what it needs, the injector provides it. The component doesn't know the implementation. The pattern is information hiding applied to object construction. Parnas argued that modules should hide design decisions from each other. DI hides the decision of which implementation to use. The principle is the same. The granularity is different.

Dependency injection is not about making code configurable. It is about making coupling visible. The injector does not reduce coupling. It reveals the coupling that was always there.

Agentic Engineering: Codebase Contracts and Skills

The rise of agentic software engineering is transforming how codebases are structured, maintained, and extended. This post examines the need for explicit contracts, modular skills, and safe parallel work patterns to support both human and agent contributors. We discuss the emergence of repository-level instruction files, the importance of clear boundaries and invariants, and strategies for reducing merge friction in multi-agent environments. Readers will learn how to design codebases that are resilient, adaptable, and ready for the next wave of collaborative, agent-driven development.

golangagentsarchitecture

Codebases used to be written primarily for humans. The main readers were the teammates who opened files in an editor, learned the local conventions by trial and error, and built a mental map over weeks or months.

That assumption is breaking down.

In an agentic workflow, the first reader of a codebase is often a coding agent. The second reader may be another agent in a different worktree. The third may be a reviewer agent that only sees a diff. Humans still matter, but the codebase now has to explain itself to fast, literal, parallel workers that do not share much hidden context.

That changes what “well structured” means.

A modern codebase needs explicit contracts

The strongest recent signal here is the emergence of repository-level instruction files such as AGENTS.md in OpenAI Codex and the closely related CLAUDE.md, skills, and subagent patterns in Claude Code best practices, skills, and worktrees.

Those tools all point toward the same lesson: agents do better when the repo contains a compact, explicit contract for:

  • what the repo is for,
  • which commands are authoritative,
  • which paths are safe to change,
  • which invariants are non-negotiable,
  • where deeper local instructions live.

Humans can absorb ambiguity. Agents mostly amplify it.

If a repo does not declare its rules, every agent run starts by rediscovering them. That wastes context, increases variance, and creates merge friction when multiple agents land changes that were individually reasonable but globally inconsistent.

AGENTS.md should be a routing layer, not a novel

The worst version of AGENTS.md is a giant wall of text. The best version is a routing contract.

At the root, it should state the repo mission, the required workflow, the canonical test/build commands, and the directories where deeper instructions live. Then each major subtree can add a local instruction file with the context only that area needs.

That lets an agent read just enough to work safely instead of loading the entire history of the repository into every run.

For a Go codebase, that usually means:

  1. a small root AGENTS.md,
  2. local contracts for subsystems like cmd/, internal/, or pkg/,
  3. repo-local skills for recurring workflows such as adding a handler, expanding a schema, or shipping a release.

The key idea is locality. The closer the instruction is to the code it governs, the easier it is for parallel agents to stay correct.

Skills are how you turn tribal knowledge into executable guidance

Agent-first repositories should treat skills as first-class assets. A good skill is not motivational prose. It is an operational recipe with:

  • purpose,
  • inputs,
  • exact steps,
  • validations,
  • failure modes.

That is useful for humans too, but it is especially valuable for agents because it removes guesswork from repeated tasks. Instead of hoping every agent rediscovers the right release flow, migration sequence, or API checklist, the repo can teach that behavior directly.

Skills are the scalable answer to “everyone knows how to do this.” In an agentic repo, that sentence should be treated as a bug report.

Modularity matters more when multiple agents work in parallel

Parallel agents are most effective when they can work in separate git worktrees with minimal coordination. That only works if the codebase has solid seams.

In Go, the natural seam is the package boundary. If a package exports a small, well-tested interface contract, different agents can work on adjacent layers without constantly reaching across the boundary.

For example, an agent that owns orchestration code should not need to know how persistence is implemented. It should only need a stable interface:

package contracts

import "context"

// SkillsCatalog defines the read-only contract for skill discovery.
type SkillsCatalog interface {
	Load(ctx context.Context, name string) (Skill, error)
	List(ctx context.Context) ([]Skill, error)
}

// WorktreeAllocator isolates parallel work into separate trees.
type WorktreeAllocator interface {
	Reserve(ctx context.Context, branch string) (Worktree, error)
	Release(ctx context.Context, path string) error
}

type Skill struct {
	Name        string
	Description string
}

type Worktree struct {
	Path   string
	Branch string
}

This looks simple, but that simplicity is the point. If the contract is small and explicit, one agent can change the allocator implementation while another extends skill discovery without both editing the same files.

Agent-friendly Go packages should minimize hidden cross-package state

A lot of merge pain in agentic work happens because packages are not really modular. They look modular, but they share config globals, mutate common registries, or depend on side effects that are never written down.

A safer pattern is to make dependencies explicit in constructors:

package planner

import (
	"context"
	"fmt"
)

// TaskStore persists the generated plan steps for later execution.
type TaskStore interface {
	SavePlan(ctx context.Context, id string, steps []string) error
}

type Service struct {
	store TaskStore
}

func New(store TaskStore) *Service {
	return &Service{store: store}
}

func (s *Service) Plan(ctx context.Context, id string, ask string) error {
	// Keep the planning stages explicit so parallel agents share the same flow.
	steps := []string{
		fmt.Sprintf("classify: %s", ask),
		"load local contracts",
		"select skill or subagent",
		"emit scoped plan",
	}
	return s.store.SavePlan(ctx, id, steps)
}

This does two things for agentic development:

  1. it reduces the number of invisible assumptions,
  2. it makes interface contracts testable in isolation.

That means a parallel agent can change a planner, store, or allocator behind the same interface and still merge cleanly.

Worktrees are safer when interface tests are part of the contract

When agents work in separate git trees, smooth merging depends on more than good intentions. It depends on contract tests.

If package boundaries are meant to stay stable, the repo should enforce them with focused tests:

package contracts_test

import (
	"context"
	"testing"
)

// fakeCatalog is a tiny stand-in that satisfies the contract in tests.
type fakeCatalog struct{}

func (fakeCatalog) Load(context.Context, string) (Skill, error) { return Skill{Name: "go-api"}, nil }
func (fakeCatalog) List(context.Context) ([]Skill, error)       { return []Skill{{Name: "go-api"}}, nil }

func TestCatalogContract(t *testing.T) {
	// Bind the fake to the interface so the consumer-facing seam stays explicit.
	var svc SkillsCatalog = fakeCatalog{}

	skill, err := svc.Load(context.Background(), "go-api")
	if err != nil {
		t.Fatalf("load skill: %v", err)
	}
	if skill.Name == "" {
		t.Fatal("expected skill name")
	}
}

The implementation here is tiny, but the idea scales: every important seam should have a small set of tests that define what consumers rely on. Parallel agents can change internals freely when those seams are protected.

The codebase now needs to optimize for fast onboarding by machines

A human teammate can survive an opaque repo if they are patient and can ask questions. An agent gets a narrower window. It needs a fast path to useful context.

That suggests a different priority order than older codebases often used:

  1. explicit contracts before clever abstractions,
  2. local instructions before tribal knowledge,
  3. interface stability before cross-package reach,
  4. reproducible commands before “it works on my laptop.”

This is also why repo-local templates such as agent-ready-repo are interesting. They encode the idea that architecture docs, operational skills, and agent-facing contracts belong inside the repo rather than floating around in chat history.

The merge target is not just correctness, but convergence

The best agent-first codebases do more than help a single agent succeed. They help many agents converge on compatible answers.

That means designing the repo so independent workers can discover the same commands, the same invariants, the same subsystem boundaries, and the same review expectations. When that happens, separate worktrees stop feeling risky. They start feeling like throughput.

The old codebase question was: can a human figure this out eventually?

The new question is: can multiple agents work in parallel, in separate trees, with enough shared contract to merge smoothly later?

That is a different optimization target. It favors explicitness, modularity, skills, and durable interface contracts. Go is a strong fit for that world because its package boundaries, interfaces, tests, and deployment story make it easier to build systems that are boring in the right places.

And in the era of agentic software engineering, boring seams are a superpower.

Sources

This is Brooks's conceptual integrity inverted. Instead of one mind controlling the design, we have agents generating code and humans reviewing the diffs. The design authority shifts from the architect to the spec writer. The spec becomes the source of truth. The code becomes a generated artifact. The engineering question: how do you maintain conceptual integrity when the code is written by agents that don't understand the design? The answer is the same as Brooks's: one mind — the spec — controls the output. The spec is the architect. The agents are the builders.

The agent does not replace the engineer. It replaces the typing. The engineering remains: what to build, why to build it, whether what was built is correct.

Agentic Era: An Economic System

The agentic era should be understood as a system of priced intelligence, constrained resources, and comparative advantage rather than just a better autocomplete stack.

economicsagentsgolang

One reason I kept pushing on SWE-Agent economics in my dissertation is that the agentic era is easy to misunderstand. It is tempting to frame it as a UX improvement: faster coding, better assistants, cheaper automation.

That is real, but it is not the deepest change.

The deeper change is that intelligence is becoming an allocatable resource. Once agents can act with some autonomy, the problem becomes economic: how do we route scarce capability across tasks, budgets, quality thresholds, and time constraints?

That is exactly the kind of question I wanted to study in my dissertation on SWE-Agent Economics and SWEChain-SDK. I focused on it because software engineering is moving from static labor assumptions toward dynamic allocation problems.

Agentic systems make cost visible

In the old story, software work was mostly discussed in terms of teams, estimates, and ticket flow. In the agentic story, it becomes easier to measure the real tradeoffs:

  • which agent finishes first,
  • which one finishes cheapest,
  • which one has the highest first-pass success,
  • which one burns the most compute,
  • which routing policy creates the best portfolio outcome.

That is why the economic lens matters. It turns a fuzzy conversation into one about mechanism design, resource allocation, and incentives.

The dissertation uses the language of SWE-Agent outsourcing markets because outsourcing markets make those choices explicit. The idea is not that every company will literally run an auction tomorrow. The idea is that auctions expose the structure of the problem in a way that centralized product flows usually hide.

Why I cared about Intelligence Per Watt

One clue that pushed me further into this area was the growing importance of efficiency metrics such as Intelligence Per Watt (IPW). Once model quality is no longer the only variable, system design has to care about capability per unit of energy, cost, and latency.

That is economically meaningful because it changes who gets selected. A slightly weaker agent with better cost-performance can win in a constrained environment. The agentic era is full of those tradeoffs.

In Go terms, that means the orchestration layer has to become explicit about utility:

package routing

type Candidate struct {
	Name       string
	PriceCents int64
	LatencyMS  int
	Score      float64
}

func Utility(c Candidate) float64 {
	// Trade off quality against both price and latency.
	return c.Score - float64(c.PriceCents)/100.0 - float64(c.LatencyMS)/1000.0
}

This is not a production formula. It is a reminder that selection policy is an economic policy. Even a simple router is already making claims about what the system values.

Why I thought software engineering needed this framing

I focused on this in research because I did not want software engineering to adopt agents while leaving its evaluation language behind. If we keep treating agentic systems as isolated model demos, we miss the fact that they are participating in a larger allocation problem.

That is why the dissertation emphasizes controlled paired experiments. If you want to compare policies, you need a system where you can hold the environment steady and vary one rule at a time. Otherwise, people confuse noise with insight.

The economics is not optional

The agentic era creates an economic system whether we acknowledge it or not. Agents consume resources, compete for tasks, differ in comparative advantage, and operate under explicit or implicit incentives.

The practical question is whether we want to study those rules directly. My answer in the dissertation was yes. That is why I focused on building a framework where those interactions could be observed, logged, and rerun under comparable conditions.

To me, that is one of the most exciting parts of the whole field. It means software engineering is no longer only about writing code better. It is about designing systems where intelligence itself becomes a resource to allocate well.

Source

Game design and systems thinking share a common structure: a set of rules, agents acting within those rules, and emergent behavior that no individual agent intended. The same structure appears in market design, protocol design, and software architecture. The engineer who studies games learns to see the rules behind the behavior. The behavior that looks like chaos is often the equilibrium of a system whose rules you haven't discovered yet. The discovery is the engineering.

The agentic era is not about agents doing what humans did. It is about agents doing what humans could not. The economics change when the cost of a task drops from an hour of labor to a penny of compute. The change is not incremental. It is structural.

SWE-Agent Economics: My Focus

My dissertation focused on SWE-Agent economics because software work is becoming a market of autonomous decision-makers, not just a pipeline of human tickets.

researchagentic-economicsgolang

One of the clearest questions in front of us is not whether coding agents will get better. They will. The harder question is what happens when software work itself starts behaving like an economic system.

That is why I focused my dissertation on this area. In Software Engineering Agent Economics: A Blockchain Software Development Kit for Economic Network Simulations, I framed what I call SWE-Agent Economics as the intersection of intelligent software engineering and software-engineering economics. I was not interested only in whether agents can solve tasks. I was interested in what happens when they bid, specialize, compete, consume priced resources, and operate under explicit allocation rules.

That focus came from a simple observation: once software work can be decomposed into scoped issues, artifacts, tests, and payments, the system starts to look less like a task board and more like a market.

Why I thought the economic lens mattered

A lot of discussion around coding agents still treats them as isolated assistants. That framing is too small.

In practice, agents already operate under:

  • budget limits,
  • latency limits,
  • tool-access limits,
  • quality thresholds,
  • routing and scheduling decisions.

Those are economic constraints, even when people describe them as product or platform constraints.

My research focused on this because I wanted a language and an experimental setup that could capture those interactions directly rather than pretending they were just implementation details. The dissertation argues that decentralized SWE-Agent outsourcing markets are a useful central case because they make allocation, pricing, and settlement rules explicit.

Why I built a toolkit instead of writing only theory

The dissertation does not stop at framing. One of its main contributions is SWEChain-SDK, a blockchain-native SDK for controlled economic network simulations of SWE-Agent markets. I cared about that because good ideas are cheap if nobody can run the experiment again under comparable conditions.

I wanted an environment where we could vary one policy dimension at a time and still keep:

  1. the same datasets,
  2. the same time base,
  3. the same agent pool,
  4. the same logging surface.

That is what makes claims about agent economics credible instead of anecdotal.

From a Go perspective, that kind of work benefits from explicit contracts and small composable binaries. A simulation stack like this should make every surface painfully clear:

package market

// Bid captures the economic signal an agent submits for a task.
type Bid struct {
	AgentID string
	TaskID  string
	Price   int64
	Score   float64
}

// Allocation records the assignment decision emitted by the market.
type Allocation struct {
	TaskID   string
	AgentID  string
	Accepted bool
}

// Logger preserves the events needed for replayable experiments.
type Logger interface {
	RecordBid(Bid) error
	RecordAllocation(Allocation) error
}

The code is intentionally boring. That is the point. If the economic mechanism is the thing under study, the interfaces around it should be stable enough to let experiments change policy without rewriting the whole platform.

Why this mattered to me as a software engineering problem

I focused on this line of research because software engineering is entering a phase where coordination matters as much as raw model capability. The interesting question is no longer only “can an agent solve this issue?” It is also:

  • which agent should do it,
  • under what incentives,
  • at what price,
  • with what comparative advantage,
  • under which settlement rule,
  • and with what observable audit trail.

That is a very software-engineering question, but it is also an economic one.

The dissertation formalizes that intuition because I think the next stage of agentic software engineering will reward teams that can reason about incentives and market structure, not only prompts and models.

The real motivation

The deepest reason I focused on SWE-Agent economics is that it creates a bridge between two worlds that are often separated: engineering systems and economic systems. Agents force them back together.

Once software workers become autonomous, the surrounding system has to answer questions about specialization, cost, settlement, trust, and transparency. I wanted my research to address those questions directly, with reproducible tooling rather than vague metaphors.

That is why the dissertation centers this topic. I think it is one of the most important lenses for understanding where software engineering is heading next.

Source

The economics of software engineering agents is not about whether agents can write code. It is about whether the code they write is worth more than the compute it costs. The answer changes with every model release.

Why Go Still Matters in AI

Go keeps earning its spot in AI products by making inference infrastructure, orchestration, and operational tooling simple to ship and simple to trust.

golangaisystems

The AI wave did not remove the need for reliable systems work. It amplified it.

Models may be trained in Python, but the product around them still needs to route requests, stream results, enforce quotas, collect traces, fan out work, and stay debuggable at 3 a.m. That is exactly where Go keeps showing up.

What Go is unusually good at

Go is rarely the language of frontier model research, but it is an excellent language for the layers around the model:

  • API gateways that need predictable latency.
  • Workers that coordinate retrieval, ranking, and post-processing.
  • CLI tools that make local evaluation and release workflows less painful.
  • Services that need easy concurrency without turning every deploy into a runtime puzzle.

In practice, that matters more than language fashion. AI products win when the whole path from prompt to production is fast, observable, and boring in the best possible way.

The real advantage is operational clarity

Go gives teams a compact standard library, fast startup, straightforward deployment, and simple static binaries. In an AI stack, that translates into fewer moving parts around the expensive part of the system.

That clarity helps when building:

  1. request routers for model providers,
  2. background jobs for embeddings and indexing,
  3. evaluation harnesses,
  4. internal tools that glue together data, prompts, and model outputs.

None of that work is glamorous, but all of it compounds.

The AI era rewards teams that can operationalize intelligence, not just demo it.

A good split of responsibilities

A pragmatic stack often looks like this:

  • Python for training loops, notebooks, and experimentation.
  • Go for services, orchestration, developer tooling, and production control planes.

That split lets each language do what it is best at. You do not need one language to dominate the entire stack to build a fast team.

Reliability is still a product feature

It is easy to talk about AI as if the whole category reduces to model capability. In production, capability is only one layer. Someone still has to make the system dependable, observable, and affordable enough to run every day.

That is why Go keeps surfacing in mature AI products. It helps teams build the unglamorous pieces that decide whether intelligence feels like a feature or like a recurring incident. The more valuable models become, the more valuable that kind of boring infrastructure becomes too.

This is engineering at the systems level: choosing the right tool for the constraint. The constraint determines the architecture. The architecture determines the language choice. The language choice determines the ecosystem. The chain of dependencies runs from the resource budget (compute, memory, latency) through the system design to the implementation language. The engineer who traces the chain makes principled choices. The engineer who doesn't inherits choices made by others for different constraints.

Go for Mobile LLM Control Planes

Edge AI products still need sync services, rollout control, metrics, and device policy, which keeps Go relevant even when inference runs elsewhere.

golangedge-llmsmobile

Even when inference runs on-device, the surrounding product still needs a control plane.

MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases focuses on the model side of that problem, but product teams still need servers for model rollout, feature flags, telemetry collection, and safety policy updates.

That is one reason Go keeps showing up in AI-adjacent systems work.

The model can live on a phone. The operational contract still lives in services, CLIs, background jobs, and dashboards. A boring systems language is still a competitive advantage there.

On-device does not mean no backend

There is a recurring fantasy in edge AI discussions that local inference makes the rest of the product magically disappear. It does not. The product still needs to decide which model version to ship, how to observe behavior in the field, and how to respond when a rollout goes badly.

Those are not minor details around the edges. They are the parts that determine whether an edge feature can be maintained after launch.

The control plane still matters

Even with local inference, teams usually need reliable systems for:

  • rollout coordination across device cohorts,
  • telemetry and health signals,
  • remote policy updates,
  • internal tools that help humans understand what is deployed.

None of that makes the on-device story less interesting. It makes the on-device story real.

This is why Go remains relevant in mobile LLM products. Not because it should replace the model stack, but because it handles the operational layer well. Services, jobs, CLIs, and dashboards benefit from a language that makes simple infrastructure easy to keep simple.

The model may live close to the user. The control plane still lives in the ordinary world of software operations, and ordinary engineering discipline still wins there.

This is engineering at the systems level: choosing the right tool for the constraint. The constraint determines the architecture. The architecture determines the language choice. The language choice determines the ecosystem. The chain of dependencies runs from the resource budget (compute, memory, latency) through the system design to the implementation language. The engineer who traces the chain makes principled choices. The engineer who doesn't inherits choices made by others for different constraints.

Mobile is not a smaller cloud. It is a different physics. The battery, the thermal envelope, the intermittent connectivity — these are not constraints to work around. They are the design.

Go for Disaggregated Serving

Disaggregated prefill and decode pipelines need schedulers, backpressure, and observability more than they need another complex runtime.

golanginferencellm-serving

One of the clearest recent serving ideas is in DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. The paper shows why prefill and decode interfere with each other and why separating them can improve goodput under real latency targets.

That architecture has a very Go-shaped seam in it.

If prefill and decode become distinct pools, somebody has to own:

  • request admission,
  • routing policy,
  • streaming state,
  • deadline propagation.

That "somebody" does not need to be the same runtime that executes the kernels. A small Go service is often the right place to implement the policy layer because it stays deployable, observable, and easy to debug when traffic gets weird.

Separation creates coordination work

Disaggregation is attractive because it stops different phases of inference from fighting each other quite so directly. But separating them does not remove complexity. It relocates complexity into scheduling, buffering, and operational policy.

That is exactly the kind of work that benefits from a clean control plane.

Once there are distinct pools, the system needs a component that can make understandable decisions when load shifts. It has to answer practical questions: which requests get admitted, what deadlines matter most, and how streaming state is preserved without turning every incident into a forensic exercise.

Why Go fits the seam

A Go service is not interesting here because it is fashionable. It is useful because the control-plane job rewards plain engineering:

  • predictable concurrency,
  • simple deployment units,
  • straightforward observability,
  • code paths operators can still follow during an incident.

The serving runtime can stay specialized around execution. The control layer can stay specialized around policy.

That separation of responsibilities feels healthy to me. DistServe highlights why inference phases deserve different treatment. The systems lesson is that once you accept that split, you should also accept a clear policy layer around it. Go is often a very practical place to put that layer.

This is engineering at the systems level: choosing the right tool for the constraint. The constraint determines the architecture. The architecture determines the language choice. The language choice determines the ecosystem. The chain of dependencies runs from the resource budget (compute, memory, latency) through the system design to the implementation language. The engineer who traces the chain makes principled choices. The engineer who doesn't inherits choices made by others for different constraints.

Disaggregation is the recognition that different components scale at different rates. The control plane scales with decisions. The data plane scales with bytes. Forcing them to scale together is forcing one to be wrong.

Go for Structured LLM Runtimes

Structured LLM programs need cache-aware runtimes and simple orchestration boundaries, which is exactly where Go stays useful.

golangllm-servingsystems

Structured prompting workflows look fancy at the model layer, but they usually fail or slow down at the runtime layer.

That is why I like reading systems papers such as SGLang: Efficient Execution of Structured Language Model Programs. The paper argues that structured LLM programs benefit from runtime features like cache reuse and careful execution planning, not just better prompts.

My Go takeaway is simple: keep Python close to model experimentation, but let Go own the boring infrastructure around it. Go is a good fit for:

  • queueing and routing structured requests,
  • managing timeouts and retries,
  • exposing clear metrics for cache hit rates and latency.

The paper is not about Go, but the engineering lesson maps well to Go services. The more structured the LLM program becomes, the more valuable it is to have a control layer that is easy to reason about under load.

Structure raises the runtime bar

A surprisingly large amount of LLM application complexity appears only after teams move past one-shot prompts. Once a workflow starts branching, reusing context, or coordinating multiple calls, the runtime matters much more. Suddenly cache behavior, execution order, and failure handling shape the user experience.

That is why "better prompting" is often an incomplete answer. A smart prompt on top of a sloppy runtime still produces a sloppy system.

Boring infrastructure is a feature

This is where Go keeps earning its place. Not because it knows anything special about prompts, but because it helps teams build a simple boundary around the complicated part.

A solid Go layer can make structured programs easier to operate by handling:

  • admission control before expensive work starts,
  • cancellation and timeout propagation,
  • metrics that explain whether cache-aware execution is helping,
  • stable service contracts around rapidly changing model logic.

That kind of boring is valuable. It turns runtime behavior into something engineers can inspect without decoding a tower of incidental complexity.

The more structured LLM applications become, the more they resemble ordinary systems problems wrapped around unusual compute. That is a good place for Go. Let the model layer stay experimental. Let the runtime boundary stay readable.

This is engineering at the systems level: choosing the right tool for the constraint. The constraint determines the architecture. The architecture determines the language choice. The language choice determines the ecosystem. The chain of dependencies runs from the resource budget (compute, memory, latency) through the system design to the implementation language. The engineer who traces the chain makes principled choices. The engineer who doesn't inherits choices made by others for different constraints.

Game theory is not about predicting what people will do. It is about understanding the structure of their choices. The structure is the game. The game is the constraint.

Edge LLMs: Model Shape and Serving Shape Are One Decision

On-device inference fails in two separate meetings — the one where a cloud model gets picked for compression, and the one where "the serving layer" is treated as a generic runtime beneath it. Two papers, MobileLLM and Fast On-device LLM Inference with NPUs, are usually read as answers to different questions. They are two halves of the same one — the model's shape and the serving stack's shape are co-designed against the same hardware budget, and teams that decide them separately ship the compromises.

edge-llmsmobilenpuarchitectureserving

The promise of on-device inference is easy to say and hard to ship. And when it fails to ship, the failure usually traces back to an org-chart artifact: model architecture was decided in one meeting and serving strategy in another, as if the phone will respect the boundary between them. It won't. On a constrained device, the shape of the model and the shape of the serving stack are a single design decision evaluated against a single budget — memory, latency, and thermals.

Two papers make the halves of this argument, and they're worth reading as one.

Small models are not shrunk models

MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases focuses on the architecture details that matter at the small end: depth, width, and parameter allocation.

A lot of edge planning still begins with the assumption that the main job is to squeeze an existing large model into a smaller box — quantize it, prune it, distill it, ship it. That instinct is understandable, and it leads to awkward compromises. A model that works well in the cloud carries structural assumptions — wide layers that amortize beautifully across an A100's memory bandwidth, attention patterns that assume KV-cache is cheap — that stop making sense once memory pressure, latency budgets, and thermals become the real boss. MobileLLM's finding that thin-and-deep beats wide-and-shallow at sub-billion scale is exactly the kind of result you never discover by compressing downward; you only find it by designing for the target from the start.

That is why model shape matters. Depth, width, and parameter allocation are not abstract architecture debates when the target is a phone. They are part of the deployment contract. Sometimes the winning move is not to compress a cloud model at all — it is to start from a shape that was designed for the edge in the first place.

NPUs change the serving problem, not just the speed

Fast On-device LLM Inference with NPUs supplies the other half. When teams first hear "NPU," it is tempting to translate that into "faster inference" and move on. In practice, specialized acceleration changes the system's shape more than the marketing shorthand suggests. Latency can improve, but the path to predictable latency depends on the runtime making better decisions — which is why the paper's ideas like prompt chunking and hardware-aware scheduling matter. Edge serving is rarely "run the same pipeline on smaller hardware." It is a different scheduling problem.

An edge stack that ignores device variation ends up either fragile or overly conservative. Different hardware profiles push the software toward different execution strategies, so the product needs explicit handling for:

  • uneven latency behavior across the device fleet,
  • fallback paths when the preferred accelerator path is unavailable,
  • request shaping that matches the device instead of an abstract average.

This is where "works on my test phone" demos break down. The demo path is a single happy execution route. The product path needs to survive a fleet.

One budget, one decision

Put the two papers side by side and the shared structure is obvious. MobileLLM says: the hardware budget determines the right model shape. The NPU paper says: the hardware substrate determines the right serving shape. Both are the same claim aimed at different layers — the constraint shapes the solution space, and the constraint is the device.

This is hardware-software co-design, and it runs in a loop: the hardware determines what software is efficient, and the software requirements determine what hardware gets built. Platforms have always worked this way — iOS constraints shaped mobile app design, cloud instance economics shaped distributed systems design. The NPU is just the newest constraint in the oldest loop.

The practical consequence is that the two decisions cannot be made in sequence, because each is an input to the other:

  • A model shape that behaves well under tight resource limits simplifies the serving layer: fewer ugly runtime compromises, less dependence on aggressive fallback behavior, more predictable performance across device classes.
  • A serving strategy that understands the accelerator changes which model shapes are viable: a shape that quantizes cleanly onto the NPU's supported ops beats a nominally better shape that keeps falling back to CPU.

A bad fit in either direction makes the other layer compensate. The serving stack papers over an architecture that was never comfortable on the hardware, or the model gets contorted to survive a runtime that was designed for an abstract average device. Either way, the user gets the compromise.

So the test for an edge effort is not "how small is the model?" or "does it use the NPU?" It is: are model shape and serving shape being evaluated in the same meeting, against the same measured budget? Measure the budget, design within it, test at the boundary — the same discipline as fitting firmware to ROM, just with a token stream on top.

Model choice and serving choice are still only two of the coordinated decisions a real deployment needs — compression, fallback policy, and field observability round out the list, which is the subject of On-Device LLMs: Systems Design. And when the budget itself moves — as 1-bit models may move it — both shapes get renegotiated together. That is the point: they were never separate.

References

  1. Zechun Liu et al., MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (2024).
  2. Daliang Xu et al., Fast On-device LLM Inference with NPUs (2024).
  3. Jiajun Xu et al., On-Device Language Models: A Comprehensive Review (2024).

1-Bit Models: Edge Budgets

Aggressive efficiency ideas like 1-bit transformers are interesting because they change the deployment budget, not just benchmark tables.

edge-llmsefficiencybitnet

Efficiency work becomes more exciting when it changes what hardware can participate.

BitNet: Scaling 1-bit Transformers for Large Language Models is interesting for that reason. The paper points toward a future where memory and energy budgets shift enough to make different deployment targets practical.

That matters for edge intelligence because budget is the product constraint:

  • battery,
  • memory,
  • thermals,
  • cost per shipped device.

If a model architecture changes those constraints in a real way, it can change what the product team is willing to build at all.

Why the budget matters more than the benchmark headline

A lot of model discourse still assumes the main question is whether a smaller system can retain enough quality to feel respectable next to a cloud model. That is part of the story, but it is not the whole story. On the edge, the first question is usually simpler: can this thing run at all inside the envelope of a real product?

That is why aggressive efficiency ideas deserve attention even before they become mainstream defaults. A meaningful shift in representation can move a device from "not viable" to "viable with tradeoffs," or from "lab demo" to "shippable feature." Those are product-level changes, not paper-only changes.

What changes when the budget moves

When memory and energy costs drop, design space opens up in practical ways:

  • more room for local context without immediately hitting device ceilings,
  • less pressure to offload every hard case to the network,
  • more freedom to treat intelligence as a default capability instead of a premium tier.

That does not mean every edge team should bet immediately on 1-bit architectures. It does mean teams should watch for ideas that alter the baseline economics of deployment. If the cost profile changes enough, the roadmap changes with it.

For edge work, that is the real promise of papers like BitNet. They are not only about squeezing a prettier number out of an efficiency table. They hint at a different hardware participation curve, and that is where product strategy starts to move.

Efficiency is not a benchmark score. It is a change in what can be built. The model that uses half the memory does not just cost less. It fits on hardware that was previously excluded. The exclusion was the constraint. The efficiency removes it.

This is scarcity economics applied to model deployment. The scarce resource is the compute budget — memory, energy, FLOPs. BitNet changes the cost function. When the cost function changes, the set of feasible deployments changes. The deployments that were uneconomic become economic. The hardware that was too small becomes sufficient. The same logic that determines whether to use a monolith or microservices — what are the constraints, what does each option cost — determines whether a model runs locally or in the cloud. The constraints are compute, memory, latency, and battery. The economics are the same. The domain is different.