Terminal Agents: A Survey of AI Agents in Command-Line Environments (Yi Bin, Xiaoyang Yuan, Haoxi Zeng, Wencheng Ye, et al., arXiv:2608.20485, 52 pages) is the first survey that treats terminal-mediated execution as an object of study in its own right, rather than scattering it across software-engineering, tool-use, and computer-use literature. It is worth a close read because it formalizes a claim this blog has been circling for months: the terminal is not an interface the agent uses, it is the substrate the agent lives in (see Agentic-First CLI Design and the canonical Harness Engineering: Best Practices for Reliable Agent Systems).
The companion repo is awesome-terminal-agents.
The scope move: substrate, not surface
The survey's organizing lens is deliberately narrow: a terminal agent is a system whose dominant progress-bearing action–observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction. Three workload-level boundary tests operationalize "dominant":
- Primary execution substrate — command execution is the workload's main means of progress.
- Iterative command feedback — outputs, errors, logs, diffs, return codes, or state changes materially shape later actions.
- Terminal dependence — removing terminal access would change the workload's core behavior.
The consequences are sharp. SWE-agent is in scope; Agentless is not (static patch pipelines lack iterative execution); OSWorld is not (visual feedback is the progress-bearing substrate); a CLI-packaged assistant that forwards requests to a non-terminal workflow is not. The tests apply per-workload, not per-product — a platform can be in scope for repository repair and out of scope for its browser workloads.
This matters for harness engineers because most "terminal agents" in industry are hybrid systems, and the boundary tests tell you which parts of them deserve terminal-specific evaluation.
Seven dimensions of terminal competence
The survey's analytical backbone is a seven-dimensional competence profile. These are not model capabilities; they are system-level responsibilities distributed across the model, interface, harness, runtime, and environment:
- Command and action formulation — translating goals into executable commands, edits, and build/test/run actions.
- Feedback and artifact interpretation — extracting evidence from stdout, stderr, exit codes, logs, diffs, stack traces.
- Runtime and environment management — preparing, configuring, and repairing dependencies, services, containers, remote machines.
- State, task, and context tracking — maintaining environment state and interaction history across extended sessions.
- Progress verification — designing checks of intermediate validity and completion conditions.
- Recovery and adaptation — diagnosing failures, replanning, retrying from grounded evidence.
- Governance and side-effect control — permissions, sandboxes, approvals, resource limits, destructive-action prevention.
Two features of this list are worth internalizing. First, verification (5) is separated from recovery (6), and both from governance (7) — conflating them is why so many harnesses discover that "it passed the tests" and "it did something terrible" are simultaneous truths (see Sandboxing AI Agents and Always-On Agents, which frames governing and recovering as first-class state concerns). Second, the dimensions are trace-observable in principle: planning appears as executable action sequences, memory is tested against persistent state, adaptation is grounded in external feedback. That traceability is the hook for everything that follows.
Architecture: the harness is a performance-shaping component
The survey traces four shifts in design emphasis: tool-augmented prompting (ReAct, Toolformer, CodeAct) → structured executable actions (SWE-agent's ACI primitives, OpenHands) → terminal-mediated agency as a first-class target (Terminal-Bench, CLI-Gym, Endless Terminals) → runtime- and harness-centered design (Meta-Harness, AutoHarness, Agentic Harness Engineering), where context compaction, approval rules, observation shaping, and observability-driven harness evolution are treated as first-class variables.
Responsibilities are organized into four layers: interface and observation, runtime and workspace, control/verification/recovery/governance, and harness/context. The recurring tensions:
- Expressiveness vs. recoverability — raw command access is expressive but noisy and hard to roll back; ACI mediation is reliable but narrows the task surface.
- Generality vs. task discipline — platform runtimes span heterogeneous work but give less structured feedback.
- Automation vs. inspectability — permission gates and approval checkpoints aid auditability but add latency; governance is the least systematically addressed dimension.
And the one that should shape every comparison you publish: attribution difficulty. Controlled skill injection can add substantial token overhead without improving pass rate, and Agentless shows static pipelines rivaling interactive agents on some repository-repair tasks. Gains under an optimized harness may come from context management, observation shaping, retry policy, or injected procedural knowledge — not from a better model. The complete model–harness–runtime configuration is the unit of comparison, not the model.
Acquisition: trajectories are the unit, failure is the signal
The relevant learning unit is a stateful trajectory — actions, observations, state changes, verification, recovery — not a prompt–response pair. The survey maps the acquisition levers:
- SFT on successful traces teaches common commands and workflows, but offers almost no supervision for diagnosis, rollback, or recovery after wrong assumptions.
- RL with executable rewards (Endless Terminals, ECHO, SWE-Master, SWE-Gym, Tmax) aligns behavior with completion, but sparse rewards can reinforce brittle or unsafe behavior.
- Process-aware / verifier-guided optimization supervises intermediate decisions via judgments, rankings, or hindsight validation (AgentHER is the key example — terminal failures usually appear early through stderr, failed tests, or inconsistent state).
- Failure-conditioned training (TRACE, AgentHER, AgentForesight) preserves diagnosis and recovery, but suffers noisy labels and repair-loop overfit.
- Synthetic generation broadens coverage of rare commands but risks teaching synthetic regularities; runtime memory (Memento, Context-Folding, TACO) sustains long horizons but risks persisting incorrect assumptions.
The uncomfortable finding: recoverable failures remain underrepresented. Failed installs, version conflicts, rollback decisions, and dead-end repairs are exactly the interactions needed to learn recovery — and successful-trace filtering removes them. If your training pipeline only keeps traces that worked, you are training an agent that cannot diagnose.
Evaluation: outcomes are the floor, not the ceiling
The evaluation section is the most useful part of the paper for working engineers. Benchmark families expose different things:
| Emphasis | Representative examples | Blind spot |
|---|---|---|
| Repository repair | SWE-bench, SWE-PolyBench | Conflates repair with terminal competence |
| CLI/terminal-centered | Terminal-Bench, TerminalWorld, LongCLI-Bench | Mixes terminal-native and repo-mediated tasks |
| Setup | SetupBench | Isolated from end-to-end workflows |
| Process | OctoBench, ProcBench, AppWorld | No standardized process scoring |
| Long horizon | SWE-Bench Pro, LoCoEval, LifelongAgentBench | Costly, hard to reproduce |
| Safety/governance | BashArena, ClawSafety, AgentHazard | Task success may conceal harmful actions |
| Production | ProdCodeBench | Limited public access |
The evidence hierarchy goes beyond binary correctness: outcome evidence (task completion), process evidence (how execution happened), environment evidence (runtime validity), trace evidence (inspection/replay), and governance evidence (permissions, containment, side effects). Three protocol facts deserve wide circulation:
- SWE-EVO-style long-horizon decay: 65–73% on SWE-Bench Verified drops to 21–25% on multi-file evolution in its reported setting. Short tasks hide failure.
- Reward hacking is measurable: an audit reports 16% of tasks across five terminal-agent benchmarks are reward-hackable.
- Harness effects dominate: WildClawBench reports an 18-point harness-conditioned gap; SWE-rebench finds evidence consistent with contamination inflation on static tasks.
Final success alone does not reveal whether an agent preserved state, recovered from failure, verified completion, or respected execution constraints.
The survey runs its own diagnostics
Rather than stopping at synthesis, the survey contributes bounded fixed-condition experiments — and these are the most concrete part of the paper. On a fixed configuration (mini-SWE-agent + DeepSeek-V4-Flash), it computes seven trace-derived process indicators (P1–P7) across four benchmark families:
| Benchmark | Tasks | Outcome | Final-verify (P5) | Gov. trigger (P7) |
|---|---|---|---|---|
| Terminal-Bench 2.1 | 241 | 52.6% | 29.5% | 0.5% |
| SetupBench | 93 | 59.1% | 36.6% | 1.4% |
| LongCLI-Bench | 21 | 23.8% | 23.8% | 4.5% |
| BashArena | 640 | 41.8% | 66.2% | 3.0% |
Key results: rule-matched invocation failures are rare (P1 ≈ 0.0%), local feedback use is high (76–82%), but final-window verification spans 23.8%–36.6% on three benchmarks versus 66.2% on BashArena — the same agent shows different process behavior because the benchmarks foreground different demands. The matched system comparison is even more pointed: SWE-agent wins every block, but the gap between systems ranges from at most 7.00 points on SWE-bench Lite to 21.25 points on Claw-SWE-Bench Lite, and the ordering of OpenHands and mini-SWE-agent reverses across benchmarks. Meanwhile the Flash-vs-Pro model variant never moves outcomes by more than 2.50 points. Benchmark choice changes what you can see, and system comparison without benchmark context is meaningless.
The trace cases ground the aggregates — the magsac-install case shows a dependency loop with a 38.5% environment-command non-zero-exit rate that never closes because the agent ends before an evaluator-relevant import check. Local feedback without verification and recovery leaves the loop unclosed.
type TraceIndicators struct {
InvocationFailures float64 // P1: rule-matched command failure rate
FeedbackUse float64 // P2: helpful-use among feedback episodes
EnvironmentExits float64 // P3: non-zero exit on env-management commands
StateErrors float64 // P4: consequential state errors
FinalWindowVerify float64 // P5: verification in final window
Recovery float64 // P6: task-relevant recovery success
GovernanceTriggers float64 // P7: irreversible/overprivileged/secret-handling actions
}
This is the closest thing the survey offers to a portable process-evaluation schema — and it is exactly the shape of instrumentation this blog argues harnesses should ship with (see Harness Engineering: Best Practices for Reliable Agent Systems).
What to take away
The survey's own conclusion is the one to steal for your evaluation practice: report system and runtime conditions explicitly, pair task outcomes with process evidence, and publish replayable traces. The four research priorities it derives — cross-domain competence, fresh and replayable process-level evaluation, runtime governability, and controlled model–harness attribution — map one-to-one onto the gaps that keep showing up in real agent deployments.
For harness engineers the operative reading is: treat the terminal as substrate, instrument the seven dimensions, keep failure traces, and never publish a model comparison without pinning the harness, the runtime, and the benchmark. This is the survey version of "tasks that fight back" — except now the tasks are graded on process, not just outcomes, and the benchmark is treated as a measurement instrument that must resist gaming, the argument made in Empirical Game Theory for Agents. If you only track resolution rates, you are measuring the flattering version of your system. (Relevant adjacent reading: DeepSeek Harness Notes and the Harness Patterns for Agentic AI Systems index.)
References
- Yi Bin, Xiaoyang Yuan, Haoxi Zeng, Wencheng Ye, Wenqi Shao, Chen Qian, Wei Ye, Yujuan Ding, Zheng Wang, Pengpeng Zeng, Jingkuan Song, Heng Tao Shen. Terminal Agents: A Survey of AI Agents in Command-Line Environments. arXiv:2608.20485, submitted 20 Aug 2026. Companion repo.
- Key benchmarks discussed: Terminal-Bench, SWE-bench, SetupBench, BashArena, LongCLI-Bench, Claw-SWE-Bench, ClawSafety.