Backed by Y Combinator (F26)

Frontier pass rate at a fraction of the cost

Cover for Frontier pass rate at a fraction of the cost

Most teams still compare coding agents by model name. In practice, the model sets the pass rate while the harness sets the bill. Across public benchmarks, the same model can cost up to 5x more depending on the harness around it, and matching a frontier pass rate can cost 68% to 96% less.

Below is a short breakdown of the numbers across DeepSWE v1.1, all 303 tasks of Coding Agent Index v1.5, and independent harness research. For full per-benchmark tables and methodology, see Introducing Reason 1.1 and The cheapest software factory.

DeepSWE v1.1: same pass rate, 6x to 21x lower cost

On DeepSWE v1.1, Reason 1.1 reaches a 72.3% pass rate on both GPT-6 Astra (xhigh) at $3.96 per task and GPT-6.1 Sol (high) at $0.60 per task. On the exact same GPT-6 Astra model and Harbor benchmark, Codex and Pi score lower and cost more per task.

Against published Coding Agent Index v1.5 costs for Claude Code and Cognition's reported Devin Fusion runs, Reason 1.1 matches or beats the pass rate at a fraction of the spend: roughly 21x cheaper than Claude Code on Opus 5.5 and 13x cheaper than Devin Fusion on Claude Fable 5.1 + SWE-2.

DeepSWE v1.1 · cost per task

Lower bar is cheaper · % above is pass rate
Figure 1. Model cost per task in USD on DeepSWE v1.1 (113 tasks). Reason, Pi, and Codex measured on Harbor (113 tasks × 3 attempts). Devin Fusion: Cognition-reported. † Artificial Analysis published cost per task, averaged over its three Coding Agent Index v1.5 benchmarks.

Across all 303 tasks: 68% to 96% cheaper at matched quality

Across all 303 tasks of Coding Agent Index v1.5 (DeepSWE v1.1, Terminal-Bench 4.0, and SWE-Atlas QnA), matching or beating a competitor's overall Index score with Reason 1.1 cuts cost per task by 68% to 96%.

At the 60 to 62 Index tier, Reason 1.1 on GPT-6.1 Sol costs $0.38 to $0.59 per task, compared to $4.54 to $7.90 on Devin Fusion, $7.47 on Codex (GPT-6 Astra max), and $10.79 to $12.39 on Claude Code. At the top end (65.1 Index), Reason 1.1 on GPT-6 Astra costs $3.96 per task, roughly 68% to 70% below Claude Code's flagship runs.

Cost per task at matched or higher score

Coding Agent Index v1.5 · all 303 tasks
Matchup (Reason vs competitor)Savings
GPT-6.1 Sol (med)vsClaude Code · Opus 5 (max)
Index 60.0 vs 59.7$0.38 vs $10.79
−96%
GPT-6.1 Sol (high)vsClaude Code · Fable 5.1 (max)
Index 61.6 vs 62.2$0.59 vs $12.39
−95%
GPT-6.1 Sol (high)vsDevin Fusion · Fable 5.1 + SWE-2
Index 61.6 vs 61.7$0.59 vs $7.90
−93%
GPT-6.1 Sol (high)vsCodex · GPT-6 Astra (max)
Index 61.6 vs 61.6$0.59 vs $7.47
−92%
GPT-6.1 Sol (med)vsDevin Fusion · Astra + SWE-2
Index 60.0 vs 58.9 (+1.1)$0.38 vs $4.54
−92%
GPT-6.1 Sol (med)vsCursor CLI · Opus 4.7 (med)
Index 60.0 vs 47.0 (+13.0)$0.38 vs $2.72
−86%
GPT-6.1 Sol (med)vsCodex · GPT-6.1 Sol (max)
Index 60.0 vs 60.1$0.38 vs $1.55same model
−75%
GPT-6 Astra (xhigh)vsClaude Code · Opus 5.5 (max)
Index 65.1 vs 66.0$3.96 vs $13.04
−70%
GPT-6 Astra (xhigh)vsClaude Code · Fable 5.1 (max)
Index 65.1 vs 62.2 (+2.9)$3.96 vs $12.39
−68%
Reason 1.1 (Harbor Hub) Competitor (Artificial Analysis)
Figure 2. Head-to-head cost per task across all 303 tasks of Coding Agent Index v1.5 (DeepSWE v1.1, Terminal-Bench 4.0, SWE-Atlas QnA). Every Reason point comes from its public Harbor Hub job.

Holding the model fixed: Claude Code to Pi to Reason

Cross-model comparisons mix model pricing with harness design. Holding the exact same model fixed isolates the harness alone, and the progression from a heavy harness (Claude Code) to a minimal harness (Pi) to Reason 1.1 is clear.

In UC Berkeley Sky Lab's HarnessTax study (Pan et al., 2026), researchers evaluated 21 model-harness pairs across Claude Code, Codex CLI, and Pi on SWE-bench Lite and Terminal-Bench 2.0. Moving from Claude Code to Pi barely moved pass rates, yet cut cost per attempt by 50% to 80% on identical weights (up to 5x cheaper).

Reason 1.1 pushes past Pi on both quality and cost. On GPT-6 Astra (xhigh) across all 303 Harbor tasks, Reason 1.1 scores 65.1 vs Pi's 61.4 (+3.8 points) while spending 4.5% less per attempt ($3.96 vs $4.15) and 10% less per solved task (and 15% less per attempt on DeepSWE v1.1: $3.96 vs $4.67). Combining the HarnessTax drop from Claude Code to Pi with Reason's measured gain over Pi puts total same-model savings at roughly 52% to 81% versus Claude Code.

Same model: Claude Code → Pi → Reason 1.1

Identical weights · cost per attempt
ModelClaude CodePiReason 1.1Savings
GPT-5.6 LunaSWE-bench Lite · HarnessTax$0.15255.6%$0.0353.3%~$0.029est.*−81.2%vs Claude Code (Pi: −80.3%)
GPT-5.6 SolSWE-bench Lite · HarnessTax$1.5477.8%$0.44174.4%~$0.421est.*−72.7%vs Claude Code (Pi: −71.4%)
Claude Fable 5SWE-bench Lite · HarnessTax$1.3397.8%$0.66696.7%~$0.636est.*−52.1%vs Claude Code (Pi: −49.9%)
GPT-6 Astra (xhigh)DeepSWE v1.1 · Harbor (113 × 3)—$4.6769.6%$3.9672.3%−15.2%vs Pi (+2.7 pts pass rate)
GPT-6 Astra (xhigh)CAI v1.5 · Harbor (303 × 3)—$4.1561.4$3.9665.1−10.0%per solve vs Pi (+3.8 Index)

Turn-1 prompt & tool overhead (HarnessTax)

Claude Code · 23 tools
27,011 tok
Minimal harness · 4 tools
1,972 tok (13.7× smaller)
Figure 3. Same-model progression from Claude Code to Pi (UC Berkeley Sky Lab's HarnessTax, SWE-bench Lite) and from Pi to Reason 1.1 (directly measured on Harbor, GPT-6 Astra xhigh). * Reason 1.1 estimates on SWE-bench Lite apply Reason's measured Harbor per-attempt cost ratio over Pi ($3.96 / $4.15 = 0.955×) to Pi's HarnessTax cost.

Where the extra tokens come from

Three mechanical choices drive almost all of the cost spread between harnesses:

Turn-1 context bloat HarnessTax measured Claude Code injecting 23 tools and 27,011 tokens before the agent's first action, compared to 4 core tools and 1,972 tokens in a minimal harness. Because multi-turn agents re-read their history on every turn, that overhead compounds across the run.

Turn inflation vs parallel execution Artificial Analysis's comparison pages give turns per task for other harnesses: 78 for Cursor CLI on Coding Agent Index v1.3 (copy archived on August 8, 2026) and 111 for Devin Fusion CLI on Index v2.1 (Codex vs Devin Fusion CLI, as read on November 21, 2026). Neither is a v1.5 measurement, and neither comes from the same runs as the numbers beside it in this post. Reason's codemode tool lets the model write one short JavaScript program that runs searches, file reads, and tests in parallel and returns only what matters. On GPT-6 Astra, the Reason 1.1 harness used 23% fewer input tokens per attempt than Codex (Introducing Reason 1.1); how much of that saving codemode accounts for has not been tested.

Prompt cache hit rate In long agent trajectories, cached input tokens make up most of the bill. Keeping the system prompt and tool schemas static and strictly append-only keeps every turn cache-eligible. As Diogo's KV-cache analysis showed, raising cache hit rate from 95% to 98% cuts a modeled agent bill by roughly 17%.

Reproduce the numbers

Every Reason 1.1 score and cost figure in this post links directly to its public Harbor Hub job, and our runner and scorer are open source in reason-machines/benchmark. For full per-benchmark tables, 95% bootstrap intervals, and interactive Pareto plots, read Introducing Reason 1.1.

Evaluation notes

  • Updated November 21, 2026: Every Coding Agent Index result in this post is on v1.5: DeepSWE v1.1, Terminal-Bench 4.0 and SWE-Atlas QnA. Artificial Analysis’s index is now v2.1, which uses different benchmarks: FrontierCode-AA 2.0, Terminal-Bench 4.1 and SWE-Atlas-QnA-AA. Scores on its current leaderboard are not directly comparable with the ones here, and we have published no Reason result on v2.0 or v2.1. Every number we have published, with its method and raw data: Reason benchmark results. Index versions as of November 21, 2026: Artificial Analysis methodology.

References

  1. Harbor Hub. Reason 1.1 public jobs (Coding Agent Index v1.5, 3 attempts per task): GPT-6 Astra xhigh · GPT-6.1 Sol xhigh · GPT-6.1 Sol high · GPT-6.1 Sol medium. Matched DeepSWE v1.1 Harbor jobs: Codex GPT-6 Astra xhigh · Pi GPT-6 Astra xhigh.
  2. Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica, and Matei Zaharia. HarnessTax: How Much Does the Harness Matter for Coding Agents? UC Berkeley Sky Lab, 2026. Post.
  3. Reason Machines. Introducing Reason 1.1, October 10, 2026, and The cheapest software factory, August 28, 2026.
  4. Artificial Analysis. Coding Agents Leaderboard, Coding Agent Index v1.5, as archived on October 3, 2026; the live leaderboard is now on v2.1. Comparison pages, retrieved October 2026: Codex vs Devin Fusion CLI, which showed Index v2.1 results when read on November 21, 2026 and has no archived copy, and Claude Code vs Cursor CLI, which is no longer served. Its copy archived on August 8, 2026 is on Index v1.3 and gives Cursor CLI 78 turns per task.
  5. Cognition. Introducing Fusion in Devin Desktop & CLI. Blog post.
  6. Diogo. (KV) Cache Rules Everything Around Me. Complete Skeptic, September 2026. Post.
  7. Reason Machines. Open benchmark runner and scorer for DeepSWE v1.1, SWE-Atlas QnA, and Terminal-Bench 4.0 on Harbor. GitHub repository.
  8. Methodology note: All Coding Agent Index figures here use v1.5 (DeepSWE v1.1, Terminal-Bench 4.0, SWE-Atlas QnA) following Artificial Analysis's methodology. Reason, Codex, and Pi Harbor runs are our own reproduction and are not endorsed by Artificial Analysis.

Authors

Reason