--- title: "Better Harnesses, Smaller Models: 90% Cheaper Agents via Automated Harness Adaptation" date: 2026-08-26 slug: better-harnesses-smaller-models summary: "Five insights from the CMU paper (arXiv:2607.08938): the naive model-swap failure is a harness artifact, not a model property; the deployable unit is the model+harness pair; shared task difficulty is the amortizable lever; harness design is a search problem; and diagnosis quality is the optimizer's bottleneck." tags: [slm, harness-engineering, harness-optimization, meta-agents, cost-efficiency, model-swap] --- [Better Harnesses, Smaller Models](https://arxiv.org/abs/2607.08938) (Yang, Zhao, Wu, Kästner — Carnegie Mellon, July 2026) starts from the null result every deployment engineer believes and inverts it. Swap a small model into a harness designed for a frontier LLM and it collapses; the paper shows the collapse is a **harness artifact, not a model property**. Adapt the harness automatically, and a 4B-parameter model matches the frontier model at a fraction of the cost. The opening example is the whole argument. A budget-approval agent with `gemini-3.1-pro` hits 97.3% at $0.22 per query. Swap in `gemma-4-26b-a4b` unchanged and accuracy drops to 75.0%. After automated harness adaptation — a step-by-step workflow in the system prompt, a filtered tool set, and a hook that blocks the agent from sending the same message twice — the same SLM hits 98.3%, beating the frontier model at 8% of the cost. Five insights follow. ## Insight 1 — The deployable unit is the model+harness pair, not the model The "model swap" mental model — replace one model, re-validate, ship — is wrong. Across seven business tasks and three SLMs, a generic harness averages 31.4% / 26.9% / 9.5% accuracy (gemma / qwen3-coder / ministral); the optimized harnesses reach 80.2% / 74.8% / 25.0%. **16 of 21 task-model pairs improved significantly, and seven closed the SLM-LLM gap entirely** — the best SLM recovering 89.7% of frontier performance at 4% of the cost, with 25% lower latency than the frontier agent. The entanglement runs both ways. Harness adaptations don't transfer across models: `ministral3-8b` needs workarounds for its file-editing failures, `qwen3-coder` occasionally emits raw XML where the protocol demands JSON tool calls. Each model needs its own harness, so the optimization is part of every deployment and every model upgrade — the pair is the unit of deployment. ## Insight 2 — Shared task difficulty is the lever, and it defines the boundary Why does this work at all? Routine business tasks have structure shared across instances: every budget request hits the same policy lookup, the same pricing table, the same reserve accounting. A frontier LLM privately reconstructs that structure on every run, token by token, at frontier prices; an SLM reconstructs it worse. The fix is to **lift the shared difficulty out of the model and into the harness — once, offline, and amortize it over every instance.** The $20 one-time optimization per task is recovered after 13 production runs on average. The same logic defines where the lever stops working: - **Repetitive tasks adapt; diverse tasks resist.** Task diversity (average Levenshtein distance between tool-call sequences) correlates with optimized performance at ρ = −0.96. Controlled: going from 3 workflow templates to 20 drops accuracy from 89.1% to 68.0%. Low-entropy tasks are exactly the ones where a harness can win the fight — the quantitative version of this blog's [tasks that fight back](https://blog.hackspree.com/#tasks-that-fight-back) argument. - **Capable SLMs benefit more.** Stronger models gain +48.8% from adaptation vs. +15.5% for the weakest — the harness offloads the repetitive parts; the model must still handle what can't be delegated. - **Harnesses can't manufacture capability.** The weakest model stays at 0.0% on two of seven tasks no matter what the harness does; website-management resists at 45.6% even for the best SLM. ## Insight 3 — Harness design is a search problem, not a craft The paper's methodological bet, in the spirit of the [bitter lesson](https://blog.hackspree.com/#hacker-laws-ase-bitter-lesson): harness design should be **automated as a search problem driven by data and evaluation**, not manual trial and error. The optimizer is a meta-agent running an evolutionary loop over the SDK's design space — system prompts, skills, tools, hooks, context management, sub-agents: 1. **Sample and evaluate** — pick a candidate GEPA-style from the Pareto front of tried harnesses; run it on a batch of training instances, logging full trajectories. 2. **Diagnose and propose** — the meta-agent reads trajectories plus the harness code and edits it, guided by a search memory of past proposals (so it stops re-proposing dead ends) and design-space documentation. Proposals pass a cheap sanity check with a repair loop. 3. **Validate and keep** — full validation-set run if the edit improved the batch; add to the pool. The loop is cheap by design: $20 per task-model pair, three runs each — $1,260 for the entire study. ## Insight 4 — The meta-agent's moves are legible The paper maps failures to adaptations so the search isn't a black box. Failures are indexed by capability — tool-use, instruction-following, knowledge, long-context, planning — and adaptations by harness component: contexts (add or manage), tools (create or manage), agent loops (instrument or orchestrate). The winning moves are consistent. The dominant addressed failure modes are **instruction-following (81%) and knowledge (81%)**; the dominant strategies are **adding contexts (86%), creating tools (43%), and managing tools (29%)**. The best anomaly-detection harness combines all three: a custom `query_mock_bigquery` tool that sidesteps the default tool's long-context behavior, a filter from 40+ MCP tools down to seven, and the environment's table-naming convention externalized into the system prompt. One negative result deserves emphasis: **no optimized harness successfully used sub-agents** — current SLMs can't track and coordinate sub-agent work. That is a bracing counterpoint to the industry reflex that answers every SLM weakness with a multi-agent topology. ## Insight 5 — Diagnosis quality is the optimizer's bottleneck The meta-agent loop is itself a system, and its developers learned where the leverage is: - **Evidence beats summaries.** Raw JSON trajectories made the meta-agent diagnose better than post-processed markdown — less human-friendly, richer evidence. - **Intelligence beats iteration count.** A cheaper meta-agent afforded more search steps but produced worse harnesses; lower-quality diagnoses outweighed the extra exploration. Hand-written heuristics (successful frontier trajectories, manual failure→fix maps) didn't help at all — given faithful evidence and an editable harness, the meta-agent infers repairs on its own. - **Explore diverse regions, not one long trajectory.** Search memory prevents rediscovering the same fixes; several independent short searches beat one long search. ## What this means The decision rule for practitioners is clean: for repetitive business workflows where the frontier-token bill is the constraint, take the best cheap MoE SLM you can find, spend a few dozen dollars on automated harness search, and expect to recover most of the frontier model's accuracy at ~5% of the cost — provided you treat "re-optimize the harness" as a normal part of every model upgrade. Economically, this is a distillation story. A frontier meta-agent spends $20 once, compresses the shared structure of a task into a harness, and every subsequent run of the cheap model executes that structure nearly free — the expensive intelligence is amortized away, the same economics as [agents as distillation at scale](https://blog.hackspree.com/#agents-are-distillation-at-scale). The cheap agent is not a model you buy; it is a system you build. The harness is the product — and as the next post shows, it is now a product you can build on top of: the OpenHands SDK is the substrate this optimizer searched, and [the harness is the product](https://blog.hackspree.com/#openhands-software-agent-sdk).