Most teams still compare coding agents by model name. In practice, the model sets the pass rate while the harness sets the bill. Across public benchmarks, the same model can cost up to 5x more depending on the harness around it, and matching a frontier pass rate can cost 68% to 96% less.
Below is a short breakdown of the numbers across DeepSWE v1.1, all 303 tasks of Coding Agent Index v1.5, and independent harness research. For full per-benchmark tables and methodology, see Introducing Reason 1.1 and The cheapest software factory.
DeepSWE v1.1: same pass rate, 6x to 21x lower cost
On DeepSWE v1.1, Reason 1.1 reaches a 72.3% pass rate on both GPT-6 Astra (xhigh) at $3.96 per task and GPT-6.1 Sol (high) at $0.60 per task. On the exact same GPT-6 Astra model and Harbor benchmark, Codex and Pi score lower and cost more per task.
Against published Coding Agent Index v1.5 costs for Claude Code and Cognition's reported Devin Fusion runs, Reason 1.1 matches or beats the pass rate at a fraction of the spend: roughly 21x cheaper than Claude Code on Opus 5.5 and 13x cheaper than Devin Fusion on Claude Fable 5.1 + SWE-2.
DeepSWE v1.1 · cost per task
Lower bar is cheaper · % above is pass rateHover or click any bar to open its source job or benchmark row.
Across all 303 tasks: 68% to 96% cheaper at matched quality
Across all 303 tasks of Coding Agent Index v1.5 (DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas QnA), matching or beating a competitor's overall Index score with Reason 1.1 cuts cost per task by 68% to 96%.
At the 60 to 62 Index tier, Reason 1.1 on GPT-6.1 Sol costs $0.38 to $0.59 per task, compared to $4.54 to $7.90 on Devin Fusion, $7.47 on Codex (GPT-6 Astra max), and $10.79 to $12.39 on Claude Code. At the top end (65.1 Index), Reason 1.1 on GPT-6 Astra costs $3.96 per task, roughly 68% to 70% below Claude Code's flagship runs.
Cost per task at matched or higher score
Coding Agent Index v1.5 · all 303 tasksHolding the model fixed: Claude Code to Pi to Reason
Cross-model comparisons mix model pricing with harness design. Holding the exact same model fixed isolates the harness alone, and the progression from a heavy harness (Claude Code) to a minimal harness (Pi) to Reason 1.1 is clear.
In UC Berkeley Sky Lab's HarnessTax study (Pan et al., 2026), researchers evaluated 21 model-harness pairs across Claude Code, Codex CLI, and Pi on SWE-bench Lite and Terminal-Bench 2.0. Moving from Claude Code to Pi barely moved pass rates, yet cut cost per attempt by 50% to 80% on identical weights (up to 5x cheaper).
Reason 1.1 pushes past Pi on both quality and cost. On GPT-6 Astra (xhigh) across all 303 Harbor tasks, Reason 1.1 scores 65.1 vs Pi's 61.4 (+3.8 points) while spending 4.5% less per attempt ($3.96 vs $4.15) and 10% less per solved task (and 15% less per attempt on DeepSWE v1.1: $3.96 vs $4.67). Combining the HarnessTax drop from Claude Code to Pi with Reason's measured gain over Pi puts total same-model savings at roughly 52% to 81% versus Claude Code.
Same model: Claude Code → Pi → Reason 1.1
Identical weights · cost per attempt| Model | Claude Code | Pi | Reason 1.1 | Savings |
|---|---|---|---|---|
| GPT-5.6 LunaSWE-bench Lite · HarnessTax | $0.15255.6% | $0.0353.3% | ~$0.029est.* | −81.2%vs Claude Code (Pi: −80.3%) |
| GPT-5.6 SolSWE-bench Lite · HarnessTax | $1.5477.8% | $0.44174.4% | ~$0.421est.* | −72.7%vs Claude Code (Pi: −71.4%) |
| Claude Fable 5SWE-bench Lite · HarnessTax | $1.3397.8% | $0.66696.7% | ~$0.636est.* | −52.1%vs Claude Code (Pi: −49.9%) |
| GPT-6 Astra (xhigh)DeepSWE v1.1 · Harbor (113 × 3) | — | $4.6769.6% | $3.9672.3% | −15.2%vs Pi (+2.7 pts pass rate) |
| GPT-6 Astra (xhigh)CAI v1.5 · Harbor (303 × 3) | — | $4.1561.4 | $3.9665.1 | −10.0%per solve vs Pi (+3.8 Index) |
Turn-1 prompt & tool overhead (HarnessTax)
Where the extra tokens come from
Three mechanical choices drive almost all of the cost spread between harnesses:
Turn-1 context bloat HarnessTax measured Claude Code injecting 23 tools and 27,011 tokens before the agent's first action, compared to 4 core tools and 1,972 tokens in a minimal harness. Because multi-turn agents re-read their history on every turn, that overhead compounds across the run.
Turn inflation vs parallel execution Artificial Analysis's comparison pages give turns per task for other harnesses: 78 for Cursor CLI on Coding Agent Index v1.3 (copy archived on August 8, 2026) and 111 for Devin Fusion CLI on Index v2.1 (Codex vs Devin Fusion CLI, as read on November 21, 2026). Neither is a v1.5 measurement, and neither comes from the same runs as the numbers beside it in this post. Reason's codemode tool lets the model write one short JavaScript program that runs searches, file reads, and tests in parallel and returns only what matters. On GPT-6 Astra, the Reason 1.1 harness used 23% fewer input tokens per attempt than Codex (Introducing Reason 1.1); how much of that saving codemode accounts for has not been tested.
Prompt cache hit rate In long agent trajectories, cached input tokens make up most of the bill. Keeping the system prompt and tool schemas static and strictly append-only keeps every turn cache-eligible. As Diogo's KV-cache analysis showed, raising cache hit rate from 95% to 98% cuts a modeled agent bill by roughly 17%.
Reproduce the numbers
Every Reason 1.1 score and cost figure in this post links directly to its public Harbor Hub job, and our runner and scorer are open source in reason-machines/benchmark. For full per-benchmark tables, 95% bootstrap intervals, and interactive Pareto plots, read Introducing Reason 1.1.
Evaluation notes
- Updated November 21, 2026: Every Coding Agent Index result in this post is on v1.5: DeepSWE v1.1, Terminal-Bench 4.0 and SWE-Atlas QnA. Artificial Analysis’s index is now v2.1, which uses different benchmarks: FrontierCode-AA 2.0, Terminal-Bench 4.1 and SWE-Atlas-QnA-AA. Scores on its current leaderboard are not directly comparable with the ones here, and we have published no Reason result on v2.0 or v2.1. Every number we have published, with its method and raw data: Reason benchmark results. Index versions as of November 21, 2026: Artificial Analysis methodology.
References
- Harbor Hub. Reason 1.1 public jobs (Coding Agent Index v1.5, 3 attempts per task): GPT-6 Astra xhigh · GPT-6.1 Sol xhigh · GPT-6.1 Sol high · GPT-6.1 Sol medium. Matched DeepSWE v1.1 Harbor jobs: Codex GPT-6 Astra xhigh · Pi GPT-6 Astra xhigh.
- Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica, and Matei Zaharia. HarnessTax: How Much Does the Harness Matter for Coding Agents? UC Berkeley Sky Lab, 2026. Post.
- Reason Machines. Introducing Reason 1.1, October 10, 2026, and The cheapest software factory, August 28, 2026.
- Artificial Analysis. Coding Agents Leaderboard, Coding Agent Index v1.5, as archived on October 3, 2026; the live leaderboard is now on v2.1. Comparison pages, retrieved October 2026: Codex vs Devin Fusion CLI, which showed Index v2.1 results when read on November 21, 2026 and has no archived copy, and Claude Code vs Cursor CLI, which is no longer served. Its copy archived on August 8, 2026 is on Index v1.3 and gives Cursor CLI 78 turns per task.
- Cognition. Introducing Fusion in Devin Desktop & CLI. Blog post.
- Diogo. (KV) Cache Rules Everything Around Me. Complete Skeptic, September 2026. Post.
- Reason Machines. Open benchmark runner and scorer for DeepSWE v1.1, SWE-Atlas QnA, and Terminal-Bench 4.0 on Harbor. GitHub repository.
- Methodology note: All Coding Agent Index figures here use v1.5 (DeepSWE v1.1, Terminal-Bench 4.0, SWE-Atlas QnA) following Artificial Analysis's methodology. Reason, Codex, and Pi Harbor runs are our own reproduction and are not endorsed by Artificial Analysis.
