Run postmortem · Published 2026-10-07
Claude Haiku 5.5 at max completed a valid 6/100 run in 49m 14s, at an estimated API cost of $0.20.
One-hour track · max reasoning · provisional
Haiku explicitly selected a valid artifact and answered 6 of 100 hidden questions correctly. Both submitted candidates scored 4/25 on public validation. The agent finished normally in 49 minutes 14 seconds, with no failed or cancelled compute jobs. This is one valid run in the normal one-hour benchmark and remains provisional. ARI Bench grades the compact system the agent builds, rather than asking the controller to answer the hidden questions directly.
One valid autonomous run produced a provisional 6% hidden exact-match score.
1 of 3 required seeds · Hidden exact-match accuracy, not public calibration accuracy.
Haiku combined a trained byte-level model with task-aware copying and arithmetic logic. It generated synthetic training examples, weighted answer bytes during training, calibrated inference and tested packaged candidates.
A 60-step short test was followed by 33,428 steps of main training and a 24,154-step continuation. The main stages completed in 705 and 510 seconds. All three returned successfully.
The main model used eight layers, width 256 and eight attention heads. The continuation resumed the earlier weights with a lower learning rate and broader synthetic task examples.
The agent ran calibration and self-tests, submitted two candidates and explicitly finalized cand_b. The 12,389,112-byte gzip package fits the 16 MB cap. Its code, configuration and weights matched the saved selected candidate.
The initial attempt hit OpenCode 1.18.32’s default 32,000-token response ceiling three times, using the entire allowance for reasoning without starting training. That attempt was stopped and excluded. The completed max run used OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX=128000, matching Haiku’s published output limit.
After a short training test, Haiku trained the 8-layer model, calibrated it and submitted cand_a. The package was valid and achieved 16% public validation accuracy.
The continuation resumed cand_a and broadened synthetic examples. The resulting cand_b package was valid but also scored 4/25 on public validation.
Haiku finalized cand_b and exited normally. All 19 broker requests received responses, with no broker errors. The graded bundle matched the selected candidate.
All training and compute jobs completed without failures or cancellations. Haiku kept the selected package within the size cap and tested its inference path before finishing.
Both public validation scores were 4/25, and the final hidden score was 6/100. Continued training did not improve the measured public score in this run.
This single run does not establish a stable ranking. It used max reasoning on the standard Anthropic endpoint through OpenRouter, OpenCode 1.18.32 and a 128,000-token response allowance. That allowance is recorded so the runtime correction is visible.
Every archived run selected for this exact model-and-effort row is shown below. The narrative uses aggregate run evidence and final artifact structure; it does not expose hidden questions, answers, or raw private transcripts.
Swipe horizontally to see all run columns.
| Run | Score | Time | Cost at run | Artifact | Candidates | Grade |
|---|---|---|---|---|---|---|
| run-0135 | 6% | 49:14 | $0.20 | 12.39 MB | 2 | Valid |
318,936 reasoning tokens are included in the estimated cost. Response allowance: 128,000 tokens.
Autonomous runs are stochastic. Two more valid seeds are required before this can be treated as an official estimate.
The agent made 2 candidate submission actions across the published run set. Selected artifacts averaged 12.39 MB compressed.
The runs averaged 49:14 of wall time. 1 run explicitly finalized before the one-hour limit.
Retained totals: 21 agent messages · 2 candidate actions · 1 published run.
$0.20 estimated for this run, including reasoning.
The completed run’s estimated API cost is US$0.20, calculated from complete metadata for all 20 model responses: 175,902 uncached input tokens, 28,070 visible output tokens, 318,936 reasoning tokens and 866,166 cached input tokens. Rates were $0.10 per million input tokens, $0.50 per million output and reasoning tokens, and $0.01 per million cached input tokens. No prompt crossed the 100,000-token pricing threshold. The provider-billed amount has not been independently reconciled. The stopped initial attempt is excluded from this valid run’s chart cost; its recorded completed calls estimate to $0.0542, with its interrupted call not captured.
At current configured API-equivalent rates, the displayed run cost is $0.20 and value is 30.03 score points per dollar.
Estimated API cost, including reasoning / run · pricing snapshot 2026-10-07.
Adjacent rows provide score context without treating small one-seed differences as settled model rankings.
Compare Claude Haiku 5.5 with the complete ARI Bench field, including score, current cost, value per dollar, effort, and seed status.
View the leaderboard