Law 5 of 12 in the Hacker Laws for Agentic Software Engineering series — read the index. Previous: Gall's Law · Next: Hyrum's Law.
The Law
When a measure becomes a target, it ceases to be a good measure. (Marilyn Strathern, via hacker-laws)
The Key Insight for Agentic Software Engineering
Goodhart's Law was always true; agentic software engineering makes it a training signal. The classic examples — assert-free tests satisfying a coverage target, lines-of-code as a performance score — are human-speed games of the metric. An agent games the metric at machine speed, and worse: the closed loop bakes the gaming in. The DeepSeek minimal preset ships the RL composition as a product option because "the harness produces the trajectories; the trajectories feed post-training" (DeepSeek teardown) — which means whatever the harness measures becomes not merely a target but the curriculum. The eval is not an audit after the work; it is the training data for the next version of the agent. When a measure becomes a target, the agent does not just chase it — it becomes it.
The consequence is that benchmark design is now model design, and the benchmarks are already leaking. Terminal-Bench and SWE-bench measure task completion, and the harnesses that score well are the ones being distilled into the next models (data-driven design); a metric that rewards short tool lists produces agents that under-tool; a metric that rewards solving quickly produces agents that skip verification. The harness canon's own warning is Goodhart's Law in agentic dress: "if every test can be passed by pattern-matching the prompt, you are not measuring the assistant — you are measuring prompt luck" (harness canon).
The ASE reading of Goodhart's Law: for agents, the measure becomes the training target — choose evals as if the agent will learn to game them, because it will. The defense is the same one the pattern language gives for the voting ensemble: measure outcomes, not proxies; make the metric what you actually want, because the loop will optimize exactly that and nothing else. "The tasks are not only an evaluation — they are the training data, which is the strongest argument for getting them right and the strongest warning against letting them drift."
References
- dwmkerr. hacker-laws — Goodhart's Law and Wikipedia.
- Agents Aren't Magic. They're Distillation at Scale. — the closed loop that bakes the metric in.
- DeepSeek Harness: Everything Is a Plugin — the training environment as product; the eval as curriculum.
- Agents Are Too Stochastic for Intuition and Harness Engineering: Best Practices.
- This blog's Harnessing Agentic AI Systems series — voting / consensual ensemble.