Run postmortem · Published 2026-10-20
DeepSeek V4.1 Flash 1017 at max effort scored 20/100 in 55m 47s. An incompatible loader prevented its trained weights from loading; estimated API cost was $0.57.
One-hour track · max reasoning · provisional
DeepSeek V4.1 Flash 1017 finalized a valid artifact and answered 20 of 100 independent hidden questions correctly. Its public-validation score was 25/25. A post-run diagnostic confirmed that the submitted expert checkpoint was paired with incompatible starter loading code. The loader suppressed the resulting error, leaving the rule solver active and a uniform distribution for unmatched prefixes. This is the original autonomous submission and grade, with no operator repair or assisted regrade. One valid seed belongs to the normal one-hour track and remains provisional.
One valid autonomous run produced a provisional 20% hidden exact-match score.
1 of 3 required seeds · Hidden exact-match accuracy, not public calibration accuracy.
DeepSeek built a prefix-parsing rule engine and trained byte-level transformers from scratch on generated task examples mixed with the supplied corpus. The selected artifact contains trained weights, but its packaged loader cannot use them.
The inference code parses the visible prefix and applies rules for arithmetic, lists, sequences, records, tables, text transformations and extraction. Recognized cases receive solver-derived next-byte probabilities. Unmatched cases are intended to use a trained byte transformer.
All eight training jobs completed, totaling 17,530 optimizer steps. Six jobs trained expert byte models on synthetic examples mixed with the public corpus, following two short starter-model jobs. The final weights match the last 4,000-step checkpoint: four layers, four heads, width 256 and a 512-byte neural context. The expert architecture uses rotary positions, RMS normalization and query/key normalization.
The 11,959,643-byte gzip package fits the 16 MB cap. Its graded code, configuration and weights match the saved finalized candidate. However, the package contains the starter model.py rather than the architecture used to train the weights. A read-only diagnostic in the isolated grader image reproduced missing and unexpected checkpoint keys. infer.py catches that exception and returns a uniform 256-byte distribution for unmatched prefixes.
The first 18 candidate submissions failed with the grader's generic inference/distribution-contract error. This message also covers inference exceptions and does not establish that every failure was literally a probability-array shape problem. The first valid submission scored 24/25 public validation.
Later candidates passed all 25 public cases. The agent continued synthetic-data training, solver revisions and packaging checks. Of 242 compute calls, eight returned errors and one filesystem scan was cancelled after the normal 120-second tool timeout; its compute lease released the queue. All eight training jobs succeeded. A retained runtime-log snapshot captured 12 recovered provider stream errors.
There were 33 candidate submissions: 18 rejected and 15 valid. DeepSeek explicitly finalized best_v6 and exited after 55 minutes 47 seconds, before the one-hour cap. All 282 broker requests received responses. The final package passed 25/25 public validation.
The hidden grader evaluated the saved finalized artifact and recorded a valid 20/100. A subsequent isolated diagnostic used only a synthetic prefix and left the package unchanged. It confirmed the incompatible loader and uniform fallback. No repaired submission was substituted for the official grade.
The agent recovered from early submission failures, completed eight training jobs and saved a package under the size cap. Provenance checks confirm that the finalized artifact was the one graded.
An incompatible model loader silently disabled the trained fallback. Public validation still passed 25/25, but hidden accuracy was 20/100. The result shows why successful training and perfect public calibration cannot replace checking the actual packaged inference path.
Max was the highest supported reasoning effort in the retained launch catalog. The run used the normal OpenCode 1.18.32 controller, a 128,000-token response allowance and the Fireworks endpoint through OpenRouter. Returned reasoning was excluded by the frozen profile while internal reasoning usage was recorded. The cause and performance impact of the recovered provider errors are not established. One seed does not establish a stable ranking or the score a repaired package would earn.
Every archived run selected for this exact model-and-effort row is shown below. The narrative uses aggregate run evidence and final artifact structure; it does not expose hidden questions, answers, or raw private transcripts.
Swipe horizontally to see all run columns.
| Run | Score | Time | Cost at run | Artifact | Candidates | Grade |
|---|---|---|---|---|---|---|
| run-0141 | 20% | 55:47 | $0.57 | 11.96 MB | 33 | Valid |
291 completed model responses; 102,188 reasoning tokens are included in the estimate. Response allowance: 128,000 tokens.
Autonomous runs are stochastic. Two more valid seeds are required before this can be treated as an official estimate.
The agent made 33 candidate submission actions across the published run set. Selected artifacts averaged 11.96 MB compressed.
The runs averaged 55:47 of wall time. 1 run explicitly finalized before the one-hour limit.
Retained totals: 292 agent messages · 33 candidate actions · 1 published run.
$0.57 per displayed run at current configured rates.
The graphs use the complete primary-response estimate, US$0.567861672, displayed as $0.57. Metadata reconciles all 291 completed model responses: 256,207 uncached input tokens, 117,557 visible output tokens, 102,188 reasoning tokens and 37,884,262 cache-read tokens, with zero cache writes. Pinned Fireworks rates retained on October 20, 2026 were $0.30 per million input tokens, $1.20 per million output tokens including reasoning and $0.006 per million cache-read tokens. Reasoning is counted once as output. OpenRouter's key-usage counter initially lagged, then stabilized at a $0.569775822 increase across two follow-up readings. That counter may include auxiliary calls and delayed posting of the $0.0000339 pre-run connection probe; it is not an itemized run invoice. Both figures round to $0.57. The initial privacy-incompatible endpoint launch recorded zero observed usage; no account privacy settings were changed.
At current configured API-equivalent rates, the displayed run cost is $0.57 and value is 35.22 score points per dollar.
Current configured API-equivalent cost / run · pricing snapshot 2026-10-20.
Adjacent rows provide score context without treating small one-seed differences as settled model rankings.
Compare DeepSeek V4.1 Flash 1017 with the complete ARI Bench field, including score, current cost, value per dollar, effort, and seed status.
View the leaderboard