Backed by Y Combinator (F26)

Reason benchmark results

On Oct 8–9, 2026, we ran Reason 1.1 on the three benchmarks of the Artificial Analysis Coding Agent Index v1.5: DeepSWE v1.1, Terminal-Bench 4.0 and SWE-Atlas QnA. The model was GPT-6 Astra at xhigh reasoning effort, with three attempts per task. Reason scored 65.1 on the Index (95% interval 60.0 to 70.2) at $3.96 per attempt. On the same tasks and graders, Codex scored 62.5 and Pi scored 61.4. This is our own reproduction of Artificial Analysis's methodology on public Harbor jobs. Artificial Analysis has not reviewed or endorsed these results.

Coding Agent Index v1.5

The Index is the equal-weight mean of the three benchmarks' pass rates. Reason's row is our own run. The Codex and Pi rows are public Harbor jobs on the same tasks that we did not run; frontier.json names who ran each. We scored every attempt in them by the same rules. Each row links its jobs, and gives its attempts per benchmark: DeepSWE v1.1, Terminal-Bench 4.0, SWE-Atlas QnA.

Coding Agent Index v1.5, all 303 tasks, GPT-6 Astra at xhigh
HarnessIndex95% intervalModelEffortAttemptsCost per attemptRun datesSource
Reason 1.165.160.0–70.2GPT-6 Astraxhigh339 +
198 +
372
$3.96–Harbor job 0ac70bbf
Codex62.557.5–67.5GPT-6 Astraxhigh338 +
330 +
372
$4.89–Harbor job e20ecafd, Harbor job caf6e433, Harbor job 11a440fb
Pi61.456.3–66.3GPT-6 Astraxhigh339 +
329 +
372
$4.15–Harbor job cd7d8de6, Harbor job 6ccd097a, Harbor job da4ac972

On the Index, Reason leads by 2.7 points over Codex (95% interval 0.1 to 5.2) and by 3.8 points over Pi (95% interval 1.1 to 6.6). A lead resamples the same tasks for both harnesses. On each single benchmark, the interval of Reason's lead includes zero; harbor-comparison.json has every one.

Results per benchmark

DeepSWE v1.1, 113 tasks, GPT-6 Astra at xhigh
HarnessPass rate95% intervalModelEffortAttemptsCost per attemptRun datesSource
Reason 1.172.3%64.3–79.4GPT-6 Astraxhigh339$3.96–Harbor job 0ac70bbf
Codex69.0%61.1–76.4GPT-6 Astraxhigh338$4.88–Harbor job e20ecafd
Pi69.6%62.0–77.0GPT-6 Astraxhigh339$4.67–Harbor job cd7d8de6

The two data files count Codex's job differently: harbor-comparison.json, which this table reads, has 338 attempts at $4.88 each, and frontier.json has 339 graded attempts, 234 of them passes, at $4.87 each. Both give a 69.0% pass rate.

Terminal-Bench 4.0, 66 tasks, GPT-6 Astra at xhigh
HarnessPass rate95% intervalModelEffortAttemptsCost per attemptRun datesSource
Reason 1.159.1%48.5–70.2GPT-6 Astraxhigh198$6.09–Harbor job 0ac70bbf
Codex57.9%47.3–68.5GPT-6 Astraxhigh330$7.12–Harbor job caf6e433
Pi55.0%44.2–65.5GPT-6 Astraxhigh329$5.53–Harbor job 6ccd097a

The Codex and Pi jobs are five attempts on each of the 66 tasks. Both data files count 330 attempts for Codex and 329 for Pi.

SWE-Atlas QnA, 124 tasks, GPT-6 Astra at xhigh
HarnessPass rate95% intervalModelEffortAttemptsCost per attemptRun datesSource
Reason 1.164.0%55.9–71.5GPT-6 Astraxhigh372$2.84†–Harbor job 0ac70bbf
Codex60.5%52.4–68.3GPT-6 Astraxhigh372$3.71–Harbor job 11a440fb
Pi59.4%51.6–66.9GPT-6 Astraxhigh372$2.94–Harbor job da4ac972

† Reason 1.1: 11 attempts the provider refused 1 + 10 times score 0; cost is over the 366 attempts with a recorded cost, as for the other rows.

Other people's published numbers

The homepage chart shows four harnesses on GPT-6 Astra. Its Reason and Pi rows, and Codex's DeepSWE value, are the xhigh rows above. Its other values are these: results other people published, at another effort or from another source. None of them is our run.

Other published GPT-6 Astra results shown on the homepage chart
HarnessBenchmarkPass rate95% intervalModelEffortAttemptsCost per attemptRun datesSource
CodexTerminal-Bench 4.058.2%52.8–63.4†GPT-6 Astramax330$9.90–Harbor job 0f01715e
CodexSWE-Atlas QnA61.8%—GPT-6 Astramax372——Artificial Analysis
Devin Fusion CLI (Cognition)DeepSWE v1.167.3%—GPT-6 Astra + SWE-2 Mediumxhigh339——Artificial Analysis
Devin Fusion CLI (Cognition)Terminal-Bench 4.050.0%—GPT-6 Astra + SWE-2 Mediumxhigh198——Artificial Analysis
Devin Fusion CLI (Cognition)SWE-Atlas QnA59.4%—GPT-6 Astra + SWE-2 Mediumxhigh372——Artificial Analysis

† Codex at max on Terminal-Bench 4.0 is the Terminal-Bench leaderboard's job: 192 of 330 attempts passed, five attempts per task. The interval is the one frontier.json gives for it, not the bootstrap of tasks used in the tables above.

A dash marks a value nobody has published. For Artificial Analysis's rows frontier.json records no run date and no interval. Artificial Analysis publishes one cost per task for a whole Index row, not one per benchmark: $7.47 for Codex on GPT-6 Astra at max and $4.54 for Devin Fusion CLI. frontier.json splits those into a cost per benchmark and marks each an estimate, so this table leaves them out. Artificial Analysis's numbers were retrieved on October 3, 2026, the date Introducing Reason 1.1 gives.

History: Reason 1.0 on DeepSWE v1.1

Reason 1.0's results cover one benchmark, DeepSWE v1.1: two runs of its 113 tasks, pooled.

Reason 1.0 on DeepSWE v1.1, 113 tasks, published September 14, 2026
HarnessPass rate95% intervalModelEffortAttemptsCost per attemptDateSource
Reason 1.073.0%66.9–78.4GPT-6 Astraxhigh226$5.32deepswe.json, Reason 1.0 post

165 of 226 attempts passed. The date is the day the result and its data file were published; the days of the two runs are not published.

This row is history, not a comparison with the 1.1 rows. Its pass rate is confirmed passes over all assigned attempts, its interval is a Wilson score interval over pooled attempts, and its cost is recorded model spend over all assigned attempts.

Method

  • Attempts. Reason ran three attempts on every task. A benchmark's pass rate averages each task's attempts, then averages the tasks, so every task counts the same. The Codex and Pi Terminal-Bench jobs are five attempts per task. The notes under the DeepSWE and Terminal-Bench tables name the two jobs whose counts are one attempt short.
  • Refusals and timeouts. An attempt that timed out, that the model provider refused, or that ended without a result scores 0. A refused request is tried again up to 10 times first: 11 of Reason's SWE-Atlas QnA attempts were refused on the first request and on all 10 retries, and score 0. An attempt the reward-hacking judge fails also scores 0, so a job's own score on Harbor Hub can be higher than the one here.
  • Intervals. In the Index table and the three tables per benchmark, each 95% interval comes from 2,000 bootstrap resamples of the tasks (seed 0), over every attempt in the linked Harbor jobs. The Index interval resamples all three benchmarks' tasks together. The other two tables say how theirs are made: Codex at max has the interval frontier.json gives, and Reason 1.0 a Wilson score interval over pooled attempts.
  • Cost. Cost per attempt is model cost: the mean over the attempts that have a recorded cost. The Index cost per attempt is the mean over all 303 tasks, each benchmark's cost weighted by its task count, which is how Artificial Analysis reports cost per task. Reason's October runs predate per-step token records; their cost and token totals per attempt match the provider's usage.
  • Dates. A row's run dates are the first and last day (UTC) of its Harbor job, as the job's Harbor Hub page gives them. An Index row spans every job it links.

The full rules are in METHODOLOGY.md, and the runner and scorer are in our benchmark repository. METHODOLOGY.md was at version 1.10.3 when this page last changed. The October jobs predate some of its later checks: a pinned harness release, token usage per response and trajectory fidelity.

Raw data

What this page does not show

  • The current Index. Every result here is on Coding Agent Index v1.5. Artificial Analysis's methodology page, read on Nov 21, 2026, lists v1.5 as September 2026 to October 2026, v2.0 as October 2026 to November 2026 and v2.1 as November 2026 to now. It gives the month of each move and no day. v2.0 replaced DeepSWE v1.1 with FrontierCode v1.1 and regraded SWE-Atlas QnA as SWE-Atlas-QnA-AA. v2.1 moved to FrontierCode-AA 2.0 and Terminal-Bench 4.1.
  • A score on v2.0 or v2.1. We have published no Reason result on either, so nothing here can be compared with Artificial Analysis's current leaderboard.
  • The hosted product. The Reason 1.0 runs measured the extracted coding harness, not the hosted product. For Reason 1.1, our methodology runs each attempt through Reason's Harbor adapter on the task's own container, in a new, empty workspace. Neither is a measure of Reason at work in a person's own repositories.
  • One set of conditions. The jobs ran on different days, from Aug 31, 2026 to Oct 9, 2026, and Reason ran three attempts per task on Terminal-Bench where the Codex and Pi jobs ran five.
  • An endorsement. Only the Reason rows are our runs. Artificial Analysis has not reviewed or endorsed our reproduction or this page.