--- title: "Better Harnesses, Smaller Models: A Method Review of the CMU Harness-Adaptation Study" date: 2026-08-30 slug: better-harnesses-smaller-models-review summary: "Method review of Yang, Zhao, Wu & Kästner (CMU), 'Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation' (arXiv:2607.08938) — the companion to this blog's earlier insights post. The findings were already covered; this review examines the method that produced them: the failure-mode→adaptation framework, the $20-per-task harness search, and the limits the first post did not address — 21 pairs on benchmark-grounded tasks, one optimizer bounded by one SDK, selection-on-validation risk, and the honest ceiling for weak models. Also: the paper is the constructive answer to Don't Blame the LLM." tags: [review, harness-adaptation, small-language-models, harness-optimization, meta-agents, cost-efficiency, slm, empirical-software-engineering] --- *[Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation](https://arxiv.org/abs/2607.08938)* — Chenyang Yang, Xinran Zhao, Tongshuang Wu, Christian Kästner (Carnegie Mellon), arXiv:2607.08938v1 (cs.SE), July 2026. ## Why this paper is getting a second post This blog covered the findings of this paper in [Better Harnesses, Smaller Models: 90% Cheaper Agents via Automated Harness Adaptation](https://blog.hackspree.com/#better-harnesses-smaller-models) when it appeared — the five insights, the headline numbers, the economics. That post was written before this blog's paper-review series existed. This review is the companion that the earlier post deliberately was not: a critical examination of the *method* that produced those numbers, the limits the first post did not name, and the paper's place in the constellation of the three papers reviewed here this month — the [SASE vision](https://blog.hackspree.com/#sase-vision-review), the [Inside the Scaffold taxonomy](https://blog.hackspree.com/#inside-the-scaffold-review), and the [Don't Blame the LLM longitudinal study](https://blog.hackspree.com/#dont-blame-the-llm-review). The reason this paper earns the treatment is that it is the constructive answer to the darkest finding in that last one. "Don't Blame the LLM" showed that *uncontrolled* harness evolution — 10 releases a week, features piling onto prompts — produced no quality gain while doubling token costs. This paper shows that *controlled, evaluation-driven* harness adaptation produces exactly the quality gains that uncontrolled evolution failed to deliver: 16 of 21 task-model pairs significantly improved, seven closed the SLM-LLM gap, and the best SLM agent recovered 89% of frontier performance at 4% of the cost. The difference between the two outcomes is not the harness — it is the presence of a search loop that measures, diagnoses, and gates. That is the same conclusion, reached from opposite directions. ## What the paper did For readers of the earlier post, a condensation. The paper's thesis: for routine business tasks, much of the difficulty is *shared across instances* — every budget request hits the same policy lookup, the same pricing table, the same approval flow — so it can be lifted out of the model and into the harness (tailored instructions, tools, hooks, loops), once, offline, and amortized over every run. To make this systematic rather than hand-crafted, they built: 1. **A failure-mode→adaptation framework.** Five capability-indexed failure modes (tool-use, instruction-following, knowledge, long-context, planning/reasoning) mapped to three adaptation categories (context adaptations, tool adaptations, agent-loop adaptations). The indexing by *capability* rather than by module is the right choice — it is what lets a diagnosis connect to a fix. 2. **A harness optimizer.** A meta-agent (gemini-3.1-pro) running a GEPA-style search over a concrete design space instantiated in the [software-agent-sdk](https://blog.hackspree.com/#openhands-software-agent-sdk): it samples a harness from a Pareto-front pool, evaluates it on a batch of training instances, diagnoses failures from raw trajectories, proposes harness edits, sanity-checks them, and validates against a validation set. Each optimization run cost $20; three runs per pair, 21 pairs — $1,260 for the entire study. 3. **A controlled evaluation.** Seven benchmark-grounded business tasks (attendance auditing, budget approval, stock alerts, anomaly detection, playwright testing, website management, code refactoring), three SLMs from different families (qwen3-coder-30b-a3b, ministral-3-8b, gemma-4-26b-a4b), a frontier-LLM baseline (gemini-3.1-pro), 20/20/60 train/validation/test splits, three agent runs per configuration, cost from official API pricing, and task diversity operationalized as the average pairwise Levenshtein distance between LLM tool-call sequences. The headline results, for orientation: gemma's average accuracy rose from 31.4% with a generic harness to 80.2% optimized (89% of the frontier's 89.7%, at $0.071 vs $1.735 per instance — 4% of the cost, with 25% lower latency); qwen3-coder went 26.9% → 74.8%; ministral went 9.5% → 25.0%. The budget-approval showcase — 75.0% → 98.3%, beating the frontier model at 8% of its cost — was achieved after 23 search iterations with a rewritten step-by-step prompt, a tool filter, and an anti-loop hook. ## What the method does well The study's methodological hygiene is genuinely good, and it should be said plainly before the criticisms. **The cost accounting is honest and complete.** Most agent papers report accuracy and stop. This one reports accuracy, cost per instance, and end-to-end latency for every configuration, prices costs from official API rates, and includes the optimizer's own cost in the amortization calculation. The $20-per-task budget is justified by observed optimization saturation, and the total is disclosed: $1,260. The breakeven claim — the $20 optimization is recovered after 13 runs on average — is the kind of number that makes the paper useful to a CFO, and it is exactly the [unit-economics discipline](https://blog.hackspree.com/#every-token-has-a-price-tag) this blog keeps asking for. **The task-diversity experiments are well-designed.** The cross-task Levenshtein correlation (ρ = −0.96 with optimized performance) is supported by a controlled follow-up that varies the number of workflow templates (3 → 20 drops accuracy 89.1% → 68.0%) — turning a correlation into a manipulated variable. And the capability result (+48.8% improvement for stronger SLMs vs. +15.5% for the weakest) is checked against an independent capability index rather than asserted. **The failures are reported as well as the successes.** The paper does not hide the ceiling: ministral stays at 0.0% on stock-alert and anomaly-detection no matter what the harness does; its code-refactoring score actually *fell* slightly after optimization (31.7% → 30.0%); its costs rose on some tasks. "Harnesses can't manufacture capability" is the paper's own conclusion, and the "no successful sub-agent adaptation" negative result is reported with two plausible explanations rather than one. This is the honesty that makes the positive results credible. ## What the method does not address The critical review starts here, because the first post — fairly, for its purpose — did not go this far. **One optimizer, one SDK, one institution.** The search space is "a practical subset" of adaptation strategies, instantiated through the API surface of software-agent-sdk. The claim "harness design can be automated as a search problem" is therefore bounded by (a) the meta-agent's diagnosis quality — a frontier model whose own failures gate the whole loop — and (b) the SDK's editable components. A different SDK with a different component model, or a different meta-model, could yield a different Pareto front. The paper acknowledges the optimizer implementation as a threat; the deeper point is that *the search space is itself a harness design decision*, made by humans, and the paper does not explore how its findings depend on that choice. The [Inside the Scaffold](https://blog.hackspree.com/#inside-the-scaffold-review) taxonomy is precisely the reminder that the space of harness designs is far larger than any one SDK's component set — twelve dimensions, not five editable categories. **Twenty-one pairs is a small n, and the tasks are benchmark-grounded, not real business workflows.** Seven tasks × three SLMs is enough to establish that the effect exists and correlates with task diversity and model capability; it is not enough to characterize the boundary conditions. And the tasks are *recreations* of business workflows from public benchmarks — TheAgentCompany, LOCA-Bench, WebGenBench, WebArena, RefactorBench — with clean environments, verifiable outcomes, and no distribution drift. Real routine business workflows are messier: they change weekly, they have legacy tools with undocumented behavior, and their failure modes drift as the underlying systems evolve. The paper's own future-work section (mixtures-of-harnesses with routing, online monitoring and adaptation) is an admission that the static single-harness-per-task design is the study's simplifying assumption, not its endpoint. **Selection-on-validation risk is real but underexplored.** The optimizer runs three times per pair and keeps the harness with the best validation score, and the paper does hold out a test set — good. But with $20 budgets, small validation batches, and three runs, the selected harness is the best of a small, noisy sample; the held-out test numbers are the headline, and the variance around them (three agent runs per configuration, averaged) is reported as a single point. The paper's own threat section is a paragraph; the reviewer's instinct is that the between-optimization-run variance deserved a table of its own. **The breakeven math is volume-dependent.** Thirteen runs is the average across tasks and models, and it is optimistic precisely where the paper's own findings are weakest: for a task with a tiny per-run cost differential (the anomaly-detection case, where optimized gemma costs $0.033 vs. the frontier's $1.45 — a 44× differential, fine — but ministral's optimized cost on several tasks is nearly unchanged), the $20 optimization is a longer bet. The amortization argument assumes a stable task distribution; the moment the workflow changes, the harness needs re-optimization, and the paper's hyper-churn context — see [Don't Blame the LLM](https://blog.hackspree.com/#dont-blame-the-llm-review) for what uncontrolled change does — makes distribution stability a strong assumption. **Reproducibility is bounded by closed APIs.** The meta-agent and the frontier baseline are closed models whose behavior can shift underneath the results; the paper acknowledges this. It matters more than usual here because the optimizer's output is not a fixed algorithm but a frontier model's interpretation of trajectories — the exact thing the [empirical-SE discipline](https://blog.hackspree.com/#empirical-se-what-studies-say) warns is a moving target. ## The synthesis the paper earns Read against the trilogy, this paper is the missing constructive chapter. **It is the controlled-adaptation twin of Don't Blame the LLM.** That paper's punchline — harness evolution at hyper-churn velocity moves quality sideways while doubling tokens — is the failure mode this paper's setup avoids by construction: the optimizer *gates* every change on measured improvement, with a validation set and a budget. The two papers together define the axis: harness changes are quality changes, always — the only question is whether something is measuring the quality. This paper is what agentic QA (the prescription of both the [SASE vision](https://blog.hackspree.com/#sase-vision-review) and the longitudinal study) looks like when it is priced: $20 per task, three runs, evaluate-diagnose-gate. The MRP in SASE and the validation gate here are the same artifact at different scales. **Its failure-mode framework is the taxonomy made operational.** The [Inside the Scaffold](https://blog.hackspree.com/#inside-the-scaffold-review) paper's most useful finding was which dimensions converge and which diverge; this paper's RQ4 delivers the operational version — instruction-following (81%) and knowledge (81%) failures dominate, addressed by adding contexts (86%) and creating/managing tools (72% combined). The failure modes the optimizer keeps hitting are the diverging dimensions of the taxonomy: context management and tool design, the open questions. What the taxonomy names, the optimizer edits. **It is the bitter-lesson argument stated as an engineering result.** The paper cites Sutton explicitly — harness design as search, driven by data and evaluation, rather than human trial and error — which is precisely the position the SASE paper's Bitter Lesson section hedged on. Here the hedge is tested: a frontier meta-agent searching a bounded design space beat the human-engineered alternative on cost and, in the showcase case, on accuracy too. The meta-agent's own limits (evidence beats summaries; intelligence beats iteration count; explore diverse regions) are the practical lessons of making the search work, and they read like this blog's [tasks that fight back](https://blog.hackspree.com/#agent-harnesses-need-tasks-that-fight-back) argument turned inward: the optimizer is only as good as the diagnosis the trajectories make possible. ## Bottom line The first post on this paper told you what the study found. This one tells you what the study is: a well-costed, honestly-reported, controlled demonstration that the harness is not only the product — the harness is *searchable* — with the limits of that claim stated as clearly as the paper states them. Twenty-one pairs on benchmark-grounded tasks, one optimizer on one SDK, selection on small validation samples, closed models underneath: the finding generalizes in direction, not in magnitude. But the direction is the important part, and it is the answer to the trilogy's central question. Uncontrolled harness evolution does not improve quality; controlled, measured harness adaptation does. The difference is a loop that costs $20 and pays for itself after thirteen runs. That is the cheapest QA infrastructure in software engineering, and it is the reason this paper deserved a second look.