---
title: "Frontier pass rate at a fraction of the cost"
description: "Why the harness sets the bill for coding agents, with head-to-head numbers on DeepSWE v1.1, Coding Agent Index v1.5, and UC Berkeley's HarnessTax study."
url: "https://reasonmachines.com/blog/cost-per-task"
section: "Blog"
---

# Frontier pass rate at a fraction of the cost

_Why the harness sets the bill for coding agents, with head-to-head numbers on DeepSWE v1.1, Coding Agent Index v1.5, and UC Berkeley's HarnessTax study._

By Reason

November 19, 2026

Most teams still compare coding agents by model name. In practice, the model sets the pass rate while the harness sets the bill. Across public benchmarks, the same model can cost up to 5x more depending on the harness around it, and matching a frontier pass rate can cost 68% to 96% less.

Below is a short breakdown of the numbers across DeepSWE v1.1, all 303 tasks of Coding Agent Index v1.5, and independent harness research. For full per-benchmark tables and methodology, see [Introducing Reason 1.1](/blog/introducing-reason-agent-1-1) and [The cheapest software factory](/blog/about).

## DeepSWE v1.1: same pass rate, 6x to 21x lower cost

On DeepSWE v1.1, Reason 1.1 reaches a 72.3% pass rate on both GPT-6 Astra (xhigh) at $3.96 per task and GPT-6.1 Sol (high) at $0.60 per task. On the exact same GPT-6 Astra model and Harbor benchmark, Codex and Pi score lower and cost more per task.

Against published Coding Agent Index v1.5 costs for Claude Code and Cognition's reported Devin Fusion runs, Reason 1.1 matches or beats the pass rate at a fraction of the spend: roughly 21x cheaper than Claude Code on Opus 5.5 and 13x cheaper than Devin Fusion on Claude Fable 5.1 + SWE-2.

### DeepSWE v1.1 · cost per task

Lower bar is cheaper · % above is pass rate

Hover or click any bar to open its source job or benchmark row.

Figure 1. Model cost per task in USD on DeepSWE v1.1 (113 tasks). Reason, Pi, and Codex measured on Harbor (113 tasks × 3 attempts). Devin Fusion: Cognition-reported. † Artificial Analysis published cost per task, averaged over its three Coding Agent Index v1.5 benchmarks.

## Across all 303 tasks: 68% to 96% cheaper at matched quality

Across all 303 tasks of Coding Agent Index v1.5 (DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas QnA), matching or beating a competitor's overall Index score with Reason 1.1 cuts cost per task by 68% to 96%.

At the 60 to 62 Index tier, Reason 1.1 on GPT-6.1 Sol costs $0.38 to $0.59 per task, compared to $4.54 to $7.90 on Devin Fusion, $7.47 on Codex (GPT-6 Astra max), and $10.79 to $12.39 on Claude Code. At the top end (65.1 Index), Reason 1.1 on GPT-6 Astra costs $3.96 per task, roughly 68% to 70% below Claude Code's flagship runs.

### Cost per task at matched or higher score

Coding Agent Index v1.5 · all 303 tasks

Matchup (Reason vs competitor)

Savings

GPT-6.1 Sol (med)

vs

Claude Code · Opus 5 (max)

Index 60.0 vs 59.7

$0.38 vs $10.79

−96%

GPT-6.1 Sol (high)

vs

Claude Code · Fable 5.1 (max)

Index 61.6 vs 62.2

$0.59 vs $12.39

−95%

GPT-6.1 Sol (high)

vs

Devin Fusion · Fable 5.1 + SWE-2

Index 61.6 vs 61.7

$0.59 vs $7.90

−93%

GPT-6.1 Sol (high)

vs

Codex · GPT-6 Astra (max)

Index 61.6 vs 61.6

$0.59 vs $7.47

−92%

GPT-6.1 Sol (med)

vs

Devin Fusion · Astra + SWE-2

Index 60.0 vs 58.9 (+1.1)

$0.38 vs $4.54

−92%

GPT-6.1 Sol (med)

vs

Cursor CLI · Opus 4.7 (med)

Index 60.0 vs 47.0 (+13.0)

$0.38 vs $2.72

−86%

GPT-6.1 Sol (med)

vs

Codex · GPT-6.1 Sol (max)

Index 60.0 vs 60.1

$0.38 vs $1.55

same model

−75%

GPT-6 Astra (xhigh)

vs

Claude Code · Opus 5.5 (max)

Index 65.1 vs 66.0

$3.96 vs $13.04

−70%

GPT-6 Astra (xhigh)

vs

Claude Code · Fable 5.1 (max)

Index 65.1 vs 62.2 (+2.9)

$3.96 vs $12.39

−68%

Reason 1.1 (Harbor Hub) Competitor (Artificial Analysis)

Figure 2. Head-to-head cost per task across all 303 tasks of Coding Agent Index v1.5 (DeepSWE v1.1, Terminal-Bench 4.0, SWE-Atlas QnA). Every Reason point comes from its public Harbor Hub job.

## Holding the model fixed: Claude Code to Pi to Reason

Cross-model comparisons mix model pricing with harness design. Holding the exact same model fixed isolates the harness alone, and the progression from a heavy harness (Claude Code) to a minimal harness (Pi) to Reason 1.1 is clear.

In UC Berkeley Sky Lab's [HarnessTax](https://harnesstax.github.io/) study (Pan et al., 2026), researchers evaluated 21 model-harness pairs across Claude Code, Codex CLI, and Pi on SWE-bench Lite and Terminal-Bench 2.0. Moving from Claude Code to Pi barely moved pass rates, yet cut cost per attempt by 50% to 80% on identical weights (up to 5x cheaper).

Reason 1.1 pushes past Pi on both quality and cost. On GPT-6 Astra (xhigh) across all 303 Harbor tasks, Reason 1.1 scores 65.1 vs Pi's 61.4 (+3.8 points) while spending 4.5% less per attempt ($3.96 vs $4.15) and 10% less per solved task (and 15% less per attempt on DeepSWE v1.1: $3.96 vs $4.67). Combining the HarnessTax drop from Claude Code to Pi with Reason's measured gain over Pi puts total same-model savings at roughly 52% to 81% versus Claude Code.

### Same model: Claude Code → Pi → Reason 1.1

Identical weights · cost per attempt

Model | Claude Code | Pi | Reason 1.1 | Savings |

GPT-5.6 Luna SWE-bench Lite · HarnessTax | $0.152 55.6% | $0.03 53.3% | ~$0.029 est.* | −81.2% vs Claude Code (Pi: −80.3%) |

GPT-5.6 Sol SWE-bench Lite · HarnessTax | $1.54 77.8% | $0.441 74.4% | ~$0.421 est.* | −72.7% vs Claude Code (Pi: −71.4%) |

Claude Fable 5 SWE-bench Lite · HarnessTax | $1.33 97.8% | $0.666 96.7% | ~$0.636 est.* | −52.1% vs Claude Code (Pi: −49.9%) |

GPT-6 Astra (xhigh) DeepSWE v1.1 · Harbor (113 × 3) | — | $4.67 69.6% | $3.96 72.3% | −15.2% vs Pi (+2.7 pts pass rate) |

GPT-6 Astra (xhigh) CAI v1.5 · Harbor (303 × 3) | — | $4.15 61.4 | $3.96 65.1 | −10.0% per solve vs Pi (+3.8 Index) |

Turn-1 prompt & tool overhead (HarnessTax)

Claude Code · 23 tools

27,011 tok

Minimal harness · 4 tools

1,972 tok (13.7× smaller)

Figure 3. Same-model progression from Claude Code to Pi (UC Berkeley Sky Lab's HarnessTax, SWE-bench Lite) and from Pi to Reason 1.1 (directly measured on Harbor, GPT-6 Astra xhigh). * Reason 1.1 estimates on SWE-bench Lite apply Reason's measured Harbor per-attempt cost ratio over Pi ($3.96 / $4.15 = 0.955×) to Pi's HarnessTax cost.

## Where the extra tokens come from

Three mechanical choices drive almost all of the cost spread between harnesses:

**Turn-1 context bloat** HarnessTax measured Claude Code injecting 23 tools and 27,011 tokens before the agent's first action, compared to 4 core tools and 1,972 tokens in a minimal harness. Because multi-turn agents re-read their history on every turn, that overhead compounds across the run.

**Turn inflation vs parallel execution** Artificial Analysis's comparison pages give turns per task for other harnesses: 78 for Cursor CLI on Coding Agent Index v1.3 ([copy archived on August 8, 2026](https://web.archive.org/web/20260808172832/https://artificialanalysis.ai/agents/coding-agents/comparisons/claude-code-vs-cursor-cli)) and 111 for Devin Fusion CLI on Index v2.1 ([Codex vs Devin Fusion CLI](https://artificialanalysis.ai/agents/coding-agents/comparisons/codex-vs-devin-fusion-cli), as read on November 21, 2026). Neither is a v1.5 measurement, and neither comes from the same runs as the numbers beside it in this post. Reason's `codemode` tool lets the model write one short JavaScript program that runs searches, file reads, and tests in parallel and returns only what matters. On GPT-6 Astra, the Reason 1.1 harness used 23% fewer input tokens per attempt than Codex ([Introducing Reason 1.1](/blog/introducing-reason-agent-1-1)); how much of that saving `codemode` accounts for has not been tested.

**Prompt cache hit rate** In long agent trajectories, cached input tokens make up most of the bill. Keeping the system prompt and tool schemas static and strictly append-only keeps every turn cache-eligible. As [Diogo's KV-cache analysis](https://www.completeskeptic.com/p/kv-cache-rules-everything-around) showed, raising cache hit rate from 95% to 98% cuts a modeled agent bill by roughly 17%.

## Reproduce the numbers

Every Reason 1.1 score and cost figure in this post links directly to its public Harbor Hub job, and our runner and scorer are open source in [reason-machines/benchmark](https://github.com/reason-machines/benchmark). For full per-benchmark tables, 95% bootstrap intervals, and interactive Pareto plots, read [Introducing Reason 1.1](/blog/introducing-reason-agent-1-1).

## Evaluation notes

- Updated November 21, 2026: Every Coding Agent Index result in this post is on v1.5: DeepSWE v1.1, Terminal-Bench 4.0 and SWE-Atlas QnA. Artificial Analysis’s index is now v2.1, which uses different benchmarks: FrontierCode-AA 2.0, Terminal-Bench 4.1 and SWE-Atlas-QnA-AA. Scores on its current leaderboard are not directly comparable with the ones here, and we have published no Reason result on v2.0 or v2.1. Every number we have published, with its method and raw data: [Reason benchmark results](/benchmarks). Index versions as of November 21, 2026: [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/coding-agents-benchmarking).

## References

<a id="ref-1"></a>
[1] Harbor Hub. Reason 1.1 public jobs (Coding Agent Index v1.5, 3 attempts per task): [GPT-6 Astra xhigh](https://hub.harborframework.com/jobs/0ac70bbf-10d8-5f45-8077-c428f10cb1a1) · [GPT-6.1 Sol xhigh](https://hub.harborframework.com/jobs/439dab82-597f-539e-9c98-7dde2ddd3072) · [GPT-6.1 Sol high](https://hub.harborframework.com/jobs/6c11b2fe-0dd9-53a7-9fe4-2dec41705472) · [GPT-6.1 Sol medium](https://hub.harborframework.com/jobs/b4405006-b07e-56bc-a427-e0c76c7bee12). Matched DeepSWE v1.1 Harbor jobs: [Codex GPT-6 Astra xhigh](https://hub.harborframework.com/jobs/e20ecafd-695a-417c-90f5-35fd36d2786f) · [Pi GPT-6 Astra xhigh](https://hub.harborframework.com/jobs/cd7d8de6-c152-48de-9840-b22bbea96d99).

<a id="ref-2"></a>
[2] Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica, and Matei Zaharia. HarnessTax: How Much Does the Harness Matter for Coding Agents? UC Berkeley Sky Lab, 2026. [Post](https://harnesstax.github.io/).

<a id="ref-3"></a>
[3] Reason Machines. [Introducing Reason 1.1](/blog/introducing-reason-agent-1-1), October 10, 2026, and [The cheapest software factory](/blog/about), August 28, 2026.

<a id="ref-4"></a>
[4] Artificial Analysis. [Coding Agents Leaderboard](https://web.archive.org/web/20261003030006/https://artificialanalysis.ai/agents/coding-agents), Coding Agent Index v1.5, as archived on October 3, 2026; the live leaderboard is now on v2.1. Comparison pages, retrieved October 2026: [Codex vs Devin Fusion CLI](https://artificialanalysis.ai/agents/coding-agents/comparisons/codex-vs-devin-fusion-cli), which showed Index v2.1 results when read on November 21, 2026 and has no archived copy, and Claude Code vs Cursor CLI, which is no longer served. Its [copy archived on August 8, 2026](https://web.archive.org/web/20260808172832/https://artificialanalysis.ai/agents/coding-agents/comparisons/claude-code-vs-cursor-cli) is on Index v1.3 and gives Cursor CLI 78 turns per task.

<a id="ref-5"></a>
[5] Cognition. Introducing Fusion in Devin Desktop & CLI. [Blog post](https://cognition.com/blog/local-fusion).

<a id="ref-6"></a>
[6] Diogo. (KV) Cache Rules Everything Around Me. Complete Skeptic, September 2026. [Post](https://www.completeskeptic.com/p/kv-cache-rules-everything-around).

<a id="ref-7"></a>
[7] Reason Machines. Open benchmark runner and scorer for DeepSWE v1.1, SWE-Atlas QnA, and Terminal-Bench 4.0 on Harbor. [GitHub repository](https://github.com/reason-machines/benchmark).

<a id="ref-8"></a>
[8] Methodology note: All Coding Agent Index figures here use v1.5 (DeepSWE v1.1, Terminal-Bench 4.0, SWE-Atlas QnA) following Artificial Analysis's [methodology](https://artificialanalysis.ai/methodology/coding-agents-benchmarking). Reason, Codex, and Pi Harbor runs are our own reproduction and are not endorsed by Artificial Analysis.
