How the Harness Around a Coding Agent Shapes Results

Run-Ze Fan, Zihao Zhang, Simin Ma, Yebowen Hu, Shouju Wang, Kaiqiang Song, Fei Liu, Hamed Zamani and Xiaoyang Wang submitted 'An Empirical Study of Harness Design for Coding Agents' to arXiv on 17 September 2026. arXiv The paper isolates harness design, the scaffolding around the model, from the capability of the base model itself.

The entry is listed under Artificial Intelligence and under Computation and Language, with pdf, html and other formats available. arXiv AI listing The study itself is a controlled comparison of that scaffolding.

The team used a lightweight coding harness with a fixed execution loop and varied only three elements. Those were planning, whether the agent lays out steps before acting, action space, which tools it can call, and context management, how past work is kept inside the context window, the amount of text the model can consider at once. The loop did not change. That fixed setup allows differences to be attributed to harness configuration rather than to prompt changes, retry logic or environment differences.

The evaluation covers four models on SWE-Bench Verified and Terminal-Bench 2.1, two benchmarks for software and terminal tasks, across 176 matched settings. arXiv Those settings span five context-management strategies, four context-window budgets and targeted ablations, or removals, of planning and action space. Each model faced the same task under the same harness variant, which keeps the comparison clean across model families and benchmarks.

On context management, the paper reports that value rises as the context-window budget tightens. Most of the gain comes from preventing context-overflow failures, where a run stops because history exceeds the limit. The gain comes less from improved reasoning over long histories and more from avoiding early termination. The paper notes this distinction matters for long runs under constrained windows.

Among the five strategies, the paper reports the strongest overall efficiency from rule-based elision, or deterministic deletion of low-value content, staged before LLM-based summarization. Elision removes low-value content first. Summarization then compresses what remains. That order reduces tokens, or text pieces, sent to the summarizer and to later model calls.

The paper also reports that making elided content recoverable adds tooling that models rarely use and yields no accuracy gain. It adds tool surface, extra decisions and extra context with little return.

Planning behaved differently by model strength. The paper reports planning shifts from an accuracy support for weaker models to a cost saver for stronger models, with little change in accuracy. Weaker models use the explicit structure to stay on task. Stronger models already plan internally. For them, explicit planning cuts wasted actions and token use without changing pass rates much.

Action space showed a similar split. The paper reports predefined tools improve performance for models with weaker bash proficiency, or skill with typed shell commands, while bash-capable models work effectively with a bash-only interface at substantially lower cost, especially on command-line-centric tasks. A curated toolset constrains and guides. A general shell rewards fluency. Terminal-Bench 2.1 shows that split directly because shell skill is central to the tasks.

The broader context here is cost control at scale. The findings point toward leaner harnesses tuned to the model in use rather than one maximal harness for all models. Tight budgets justify elision plus summarization, while loose budgets reduce the payoff. Weak bash skill justifies wrapper tools, while strong bash skill justifies removing them. Planning can remain as correctness support for weaker models and as efficiency support for stronger ones, which allows small per-run savings to compound across thousands of trajectories without a model upgrade.