Problem 10 of 15 in the Harnessing Agentic AI Systems series — read the index for the framing. Previous: Asynchronous Tool Worker Queue Pattern · Next: Dynamic Tool Discovery Pattern.
The Problem — Making the fast loop repeatable
Real APIs are slow, flaky, and costly; the system needs deterministic replay at scale. There is no named anti-pattern; its failure mode — the mock that drifts from production — is in the tradeoffs.
| Field | P11 — Mock Tool Virtualization (pattern) |
|---|---|
| Forces | Fidelity vs frozen repeatability; speed vs drift detection; determinism vs live realism. |
| Solution | Swap production APIs out for lightweight mock responses inside development environments during multi-agent unit testing routines. |
| Consequences | Deterministic replay becomes possible — "a harness that cannot reproduce a run cannot measure a change" — and the sample sizes data-driven design demands become affordable. |
| Tradeoffs | A mock that drifts teaches the wrong lessons; mocks hide latency and rate limits; an agent trained only against mocks fails the first time it meets a 429. The honesty rule: mock for the unit test, record for the integration test, real for the eval. |
| Evidence | VCR.py, the record-and-replay reference (docs); the harness canon's layering (harness canon). |
| Related | Composes with P10; fast-loop complement to F5 (the slow loop). |
Discussion
Mocks are how the fast loop gets deterministic replay at the sample sizes data-driven design demands — "a single run is a data point; a thousand runs is a distribution." The honesty rule is the whole pattern: a drifting cassette teaches the wrong lessons, and the final gate must always be real. F5's live environment is this pattern's slow-loop counterpart.
Key Insight
The fast loop needs frozen reality. Deterministic replay is the enabling condition for measurement — a harness that cannot reproduce a run cannot measure a change — and the honesty rule is: mock for the unit test, record for the integration test, real for the eval.
References
VCR.py (docs); archive: harness canon, Agents Are Too Stochastic for Intuition.