Coding Agent Index v1.5
The Index is the equal-weight mean of the three benchmarks' pass rates. Reason's row is our own run. The Codex and Pi rows are public Harbor jobs on the same tasks that we did not run; frontier.json names who ran each. We scored every attempt in them by the same rules. Each row links its jobs, and gives its attempts per benchmark: DeepSWE v1.1, Terminal-Bench 4.0, SWE-Atlas QnA.
| Harness | Index | 95% interval | Model | Effort | Attempts | Cost per attempt | Run dates | Source |
|---|---|---|---|---|---|---|---|---|
| Reason 1.1 | 65.1 | 60.0–70.2 | GPT-6 Astra | xhigh | 339 + 198 + 372 | $3.96 | – | Harbor job 0ac70bbf |
| Codex | 62.5 | 57.5–67.5 | GPT-6 Astra | xhigh | 338 + 330 + 372 | $4.89 | – | Harbor job e20ecafd, Harbor job caf6e433, Harbor job 11a440fb |
| Pi | 61.4 | 56.3–66.3 | GPT-6 Astra | xhigh | 339 + 329 + 372 | $4.15 | – | Harbor job cd7d8de6, Harbor job 6ccd097a, Harbor job da4ac972 |
On the Index, Reason leads by 2.7 points over Codex (95% interval 0.1 to 5.2) and by 3.8 points over Pi (95% interval 1.1 to 6.6). A lead resamples the same tasks for both harnesses. On each single benchmark, the interval of Reason's lead includes zero; harbor-comparison.json has every one.
Results per benchmark
| Harness | Pass rate | 95% interval | Model | Effort | Attempts | Cost per attempt | Run dates | Source |
|---|---|---|---|---|---|---|---|---|
| Reason 1.1 | 72.3% | 64.3–79.4 | GPT-6 Astra | xhigh | 339 | $3.96 | – | Harbor job 0ac70bbf |
| Codex | 69.0% | 61.1–76.4 | GPT-6 Astra | xhigh | 338 | $4.88 | – | Harbor job e20ecafd |
| Pi | 69.6% | 62.0–77.0 | GPT-6 Astra | xhigh | 339 | $4.67 | – | Harbor job cd7d8de6 |
The two data files count Codex's job differently: harbor-comparison.json, which this table reads, has 338 attempts at $4.88 each, and frontier.json has 339 graded attempts, 234 of them passes, at $4.87 each. Both give a 69.0% pass rate.
| Harness | Pass rate | 95% interval | Model | Effort | Attempts | Cost per attempt | Run dates | Source |
|---|---|---|---|---|---|---|---|---|
| Reason 1.1 | 59.1% | 48.5–70.2 | GPT-6 Astra | xhigh | 198 | $6.09 | – | Harbor job 0ac70bbf |
| Codex | 57.9% | 47.3–68.5 | GPT-6 Astra | xhigh | 330 | $7.12 | – | Harbor job caf6e433 |
| Pi | 55.0% | 44.2–65.5 | GPT-6 Astra | xhigh | 329 | $5.53 | – | Harbor job 6ccd097a |
The Codex and Pi jobs are five attempts on each of the 66 tasks. Both data files count 330 attempts for Codex and 329 for Pi.
| Harness | Pass rate | 95% interval | Model | Effort | Attempts | Cost per attempt | Run dates | Source |
|---|---|---|---|---|---|---|---|---|
| Reason 1.1 | 64.0% | 55.9–71.5 | GPT-6 Astra | xhigh | 372 | $2.84† | – | Harbor job 0ac70bbf |
| Codex | 60.5% | 52.4–68.3 | GPT-6 Astra | xhigh | 372 | $3.71 | – | Harbor job 11a440fb |
| Pi | 59.4% | 51.6–66.9 | GPT-6 Astra | xhigh | 372 | $2.94 | – | Harbor job da4ac972 |
† Reason 1.1: 11 attempts the provider refused 1 + 10 times score 0; cost is over the 366 attempts with a recorded cost, as for the other rows.
Other people's published numbers
The homepage chart shows four harnesses on GPT-6 Astra. Its Reason and Pi rows, and Codex's DeepSWE value, are the xhigh rows above. Its other values are these: results other people published, at another effort or from another source. None of them is our run.
| Harness | Benchmark | Pass rate | 95% interval | Model | Effort | Attempts | Cost per attempt | Run dates | Source |
|---|---|---|---|---|---|---|---|---|---|
| Codex | Terminal-Bench 4.0 | 58.2% | 52.8–63.4† | GPT-6 Astra | max | 330 | $9.90 | – | Harbor job 0f01715e |
| Codex | SWE-Atlas QnA | 61.8% | — | GPT-6 Astra | max | 372 | — | — | Artificial Analysis |
| Devin Fusion CLI (Cognition) | DeepSWE v1.1 | 67.3% | — | GPT-6 Astra + SWE-2 Medium | xhigh | 339 | — | — | Artificial Analysis |
| Devin Fusion CLI (Cognition) | Terminal-Bench 4.0 | 50.0% | — | GPT-6 Astra + SWE-2 Medium | xhigh | 198 | — | — | Artificial Analysis |
| Devin Fusion CLI (Cognition) | SWE-Atlas QnA | 59.4% | — | GPT-6 Astra + SWE-2 Medium | xhigh | 372 | — | — | Artificial Analysis |
† Codex at max on Terminal-Bench 4.0 is the Terminal-Bench leaderboard's job: 192 of 330 attempts passed, five attempts per task. The interval is the one frontier.json gives for it, not the bootstrap of tasks used in the tables above.
A dash marks a value nobody has published. For Artificial Analysis's rows frontier.json records no run date and no interval. Artificial Analysis publishes one cost per task for a whole Index row, not one per benchmark: $7.47 for Codex on GPT-6 Astra at max and $4.54 for Devin Fusion CLI. frontier.json splits those into a cost per benchmark and marks each an estimate, so this table leaves them out. Artificial Analysis's numbers were retrieved on October 3, 2026, the date Introducing Reason 1.1 gives.
History: Reason 1.0 on DeepSWE v1.1
Reason 1.0's results cover one benchmark, DeepSWE v1.1: two runs of its 113 tasks, pooled.
| Harness | Pass rate | 95% interval | Model | Effort | Attempts | Cost per attempt | Date | Source |
|---|---|---|---|---|---|---|---|---|
| Reason 1.0 | 73.0% | 66.9–78.4 | GPT-6 Astra | xhigh | 226 | $5.32 | deepswe.json, Reason 1.0 post |
165 of 226 attempts passed. The date is the day the result and its data file were published; the days of the two runs are not published.
This row is history, not a comparison with the 1.1 rows. Its pass rate is confirmed passes over all assigned attempts, its interval is a Wilson score interval over pooled attempts, and its cost is recorded model spend over all assigned attempts.
Method
- Attempts. Reason ran three attempts on every task. A benchmark's pass rate averages each task's attempts, then averages the tasks, so every task counts the same. The Codex and Pi Terminal-Bench jobs are five attempts per task. The notes under the DeepSWE and Terminal-Bench tables name the two jobs whose counts are one attempt short.
- Refusals and timeouts. An attempt that timed out, that the model provider refused, or that ended without a result scores 0. A refused request is tried again up to 10 times first: 11 of Reason's SWE-Atlas QnA attempts were refused on the first request and on all 10 retries, and score 0. An attempt the reward-hacking judge fails also scores 0, so a job's own score on Harbor Hub can be higher than the one here.
- Intervals. In the Index table and the three tables per benchmark, each 95% interval comes from 2,000 bootstrap resamples of the tasks (seed 0), over every attempt in the linked Harbor jobs. The Index interval resamples all three benchmarks' tasks together. The other two tables say how theirs are made: Codex at max has the interval frontier.json gives, and Reason 1.0 a Wilson score interval over pooled attempts.
- Cost. Cost per attempt is model cost: the mean over the attempts that have a recorded cost. The Index cost per attempt is the mean over all 303 tasks, each benchmark's cost weighted by its task count, which is how Artificial Analysis reports cost per task. Reason's October runs predate per-step token records; their cost and token totals per attempt match the provider's usage.
- Dates. A row's run dates are the first and last day (UTC) of its Harbor job, as the job's Harbor Hub page gives them. An Index row spans every job it links.
The full rules are in METHODOLOGY.md, and the runner and scorer are in our benchmark repository. METHODOLOGY.md was at version 1.10.3 when this page last changed. The October jobs predate some of its later checks: a pinned harness release, token usage per response and trajectory fidelity.
Raw data
- harbor-comparison.json: Reason, Codex and Pi on GPT-6 Astra at xhigh, overall and per benchmark: pass rate, attempts, cost per attempt, 95% intervals and each row's Harbor job.
- frontier.json: 278 published results across harnesses, models and efforts, each with its source and who ran it, and its date where one is published.
- deepswe.json: Reason 1.0 on DeepSWE v1.1: both runs, every task's outcome and solving time, and the recorded model spend.
- Harbor Hub jobs, each with every trial and trajectory: Reason, all three benchmarks · Codex, DeepSWE v1.1 · Codex, Terminal-Bench 4.0 · Codex, SWE-Atlas QnA · Pi, DeepSWE v1.1 · Pi, Terminal-Bench 4.0 · Pi, SWE-Atlas QnA · Codex at max, Terminal-Bench 4.0.
- The posts these numbers were first published in: Introducing Reason 1.0 (September 14, 2026), Introducing Reason 1.1 (October 10, 2026) and Frontier pass rate at a fraction of the cost (November 19, 2026).
What this page does not show
- The current Index. Every result here is on Coding Agent Index v1.5. Artificial Analysis's methodology page, read on Nov 21, 2026, lists v1.5 as September 2026 to October 2026, v2.0 as October 2026 to November 2026 and v2.1 as November 2026 to now. It gives the month of each move and no day. v2.0 replaced DeepSWE v1.1 with FrontierCode v1.1 and regraded SWE-Atlas QnA as SWE-Atlas-QnA-AA. v2.1 moved to FrontierCode-AA 2.0 and Terminal-Bench 4.1.
- A score on v2.0 or v2.1. We have published no Reason result on either, so nothing here can be compared with Artificial Analysis's current leaderboard.
- The hosted product. The Reason 1.0 runs measured the extracted coding harness, not the hosted product. For Reason 1.1, our methodology runs each attempt through Reason's Harbor adapter on the task's own container, in a new, empty workspace. Neither is a measure of Reason at work in a person's own repositories.
- One set of conditions. The jobs ran on different days, from Aug 31, 2026 to Oct 9, 2026, and Reason ran three attempts per task on Terminal-Bench where the Codex and Pi jobs ran five.
- An endorsement. Only the Reason rows are our runs. Artificial Analysis has not reviewed or endorsed our reproduction or this page.