Choose models using evidence from your application’s inputs and output formats. Run identical tasks through OpenRouter and compare graded accuracy, billed cost, and request time. On our 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials across three repeats under the tested contracts and settings.
TL;DR
- Code checks measure correctness and output compliance.
Code checks measure correctness and output compliance.
- Sol passed 180/180 trials on our synthetic set; DeepSeek had the lowest response cost, with 157/180 passes.
Sol passed 180/180 trials on our synthetic set; DeepSeek had the lowest response cost, with 157/180 passes.
- Agreement with the public ranking was weak, with a Spearman correlation of 0.109.
Agreement with the public ranking was weak, with a Spearman correlation of 0.109.
- Before you trust a score, check how many times each task ran, how many answers came back, and whether each bill was verified.
Before you trust a score, check how many times each task ran, how many answers came back, and whether each bill was verified.
Define the behavior first, then its checks.
Step 1. Write tasks with answers a script can check
A task needs an input, expected answer, and output contract. For invoices, specify included rows, rounding, and acceptable currency suffixes.
We wrote all 40 baseline tasks and 20 hard tasks before the final run.
Compute references independently. Separate prompt-tuning cases from final evaluation. Refund cases test written decisions, without executing transactions.
Step 2. Grade with code, and check the grader
Use exact matching for APPROVE, and schema/value checks for JSON. Normalize only permitted differences. Accepting prose around a number hides a failure when your consumer requires only a number.
Test known-correct, wrong, and malformed responses. For a refund threshold, include exactly $200 and 30 days, then values just above each boundary.
Our development checker rejected valid Python because it omitted ord and type. We corrected it before the final study, specifying CPython 3.12.10/WASI with resource limits. Otherwise, checker failures would have counted against models.
The full-study runner isolates generated code and stops if sandbox setup is missing. The teaching example executes no generated code.
Step 3. Run the three-task Python example
Download the companion ZIP. Before extracting it, compare its SHA-256 checksum with:
PowerShell:
Linux or macOS:
If the values differ, stop and check the download. After they match, extract the ZIP. You need Python 3.12 or later. The standard-library teaching runner, JSON example, and comparisons passed offline checks on Windows x64 and Linux. macOS remains untested. The bundled full-study sandbox and native Ori setup support Windows x64 only.
From the extracted folder:
The check prints OFFLINE reference and grader checks passed; zero inference calls. The complete script follows in file order. Its companion module handles requests, persistence, and reporting.
Before using a score, check its reference and rejection rules. These offline fixtures cover boundary decisions, invalid formats, and truncated answers without calling a model.
Keep checking separate from spending. The command interface defaults to offline validation and requires an explicit live option and allowance before inference.
Configure and inspect the plan
Use the supplied Luna configuration to set the model, repeat count, token allowance and provider constraints:
Sol’s configuration preserves the repeat/token settings but uses its own price ceilings:
models selects candidates, repeats specifies requests per task, and max_tokens sets the completion allowance. Reasoning can consume that allowance before visible output. Offline checks make no API calls. Paid runs require the explicit —live flag. Length-truncated responses fail without retry.
Price ceilings use USD per million tokens; Luna’s prompt ceiling is approximately $0.10. If a provider no longer supports the saved settings, the runner must stop rather than silently change them. Provider fallback doesn’t swap models or resend inference.
Actual offline output:
A reservation is a local cost estimate, not an account hold or guaranteed cap. Plan before choosing an allowance.
Changing tasks, repeats, or tokens changes the estimate. If spending goes over the estimate, the script stops sending new request.
Make a small paid run
Create a key in API-key settings, add credit, and configure a suitable key limit. Set the variable locally in your terminal.
PowerShell:
Bash or Zsh:
Reuse an existing variable if configured. The package doesn’t load .env. For the shipped three-task Luna configuration:
Reconciliation performs read-only generation metadata lookups. Reports show response coverage, meaning completed responses divided by planned trials, plus passes/graded, failures, verified cost, and median request time. Until a bill is verified, the report keeps its cost unresolved. Request time excludes grading and metadata; it’s not time to first token.
Live results vary. Synthetic examples below illustrate reporting, not model measurements.
Step 4. Add your own task and grader
The supplied custom-tasks.json shows the task format:
python custom_task.py check independently computes its reference with Decimal and tests correct, wrong, and malformed answers. Custom money IDs use the two-decimal grader. Another contract needs its own grader and checks.
The complete json_extraction.py follows in three parts, in file order. It permits JSON whitespace and key-order differences, but rejects missing, extra, or duplicate keys and wrong types. A schema-valid object with a wrong value is incorrect. Filtered or abnormal finishes have separate classifications.
JSON syntax alone is insufficient. Reject duplicate keys and invalid number values, then validate the schema before comparing fields with the reference.
Check both acceptance and rejection. These fixtures distinguish valid but wrong values from malformed output, and verify that filtering or truncation can’t count as a pass.
The final main(…) call connects tasks, grading, and reference checks to the runner. Adapt the source facts and expected fields, then run:
Expected check output:
Use separate folders. These allowances apply only to the shipped single-case plans.
Set equal repeat counts before starting. Three repeats of one case test variability, not workload coverage. Add representative boundaries and held-out cases before drawing broader conclusions.
Step 5. Compare saved runs and handle missing answers
For your saved JSON runs:
This writes comparison.md and comparison.json without inference or regrading. It validates contracts, settings, and identities; incompatible runs require investigation rather than a silent ranking.
First rehearse it offline:
The table includes these synthetic fixture values, shown with abbreviated columns:
Sol’s fixture has a missing answer and an unverified bill. Its 5/5 score lacks complete coverage. These figures measure neither model.
Keep an uncertain trial and its reservation. Fix access or rate-limit problems, then resume with unchanged configuration, source, and allowance:
Resume continues only never-attempted trials. A known bill can’t recover missing text. If we can’t find a generation record for a request, the report keeps it marked as unresolved. Don’t discard the run or delete attempts. The README covers stale locks after crashes.
To compare cases answered by every participant:
The explicit intersection retains original coverage and lists exclusions. It can select easier cases and doesn’t repair missing bills. Never compare different subsets without disclosure.
The optional 60-task runner and runtime setup are in the companion. Its historical results provide a larger worked example.
Which model did best on our benchmark?
Each model received 40 baseline and 20 hard tasks three times, giving 120 baseline and 60 hard grades. These results describe the 60 synthetic tasks under our fixed settings, completed on 20 October 2026. AA denotes the Artificial Analysis Intelligence Index from our dated public snapshot.
Cost per 100 task responses is the verified cost of the 180 scored answers, scaled to 100. It excludes operational overhead and unresolved bills. Median seconds measures local request elapsed time.
Three findings guide the choice on this set.
- Highest pass rate. Sol passed 180/180 at about $0.2449 per 100 responses.
Highest pass rate. Sol passed 180/180 at about $0.2449 per 100 responses.
- Lowest response cost. DeepSeek cost about $0.00394 per 100, passing 157/180 with 14 truncations. Luna passed 162/180 at about $0.01848. Choose against your required pass rate and failure consequences.
Lowest response cost. DeepSeek cost about $0.00394 per 100, passing 157/180 with 14 truncations. Luna passed 162/180 at about $0.01848. Choose against your required pass rate and failure consequences.
- Output compliance affects the score. Sonnet’s 66 malformed responses cover 21 invoice, 30 date, and 15 refund trials. Raw answers include explanations where prompts required only a number or JSON. The score measures contract compliance as well as answer correctness, not pure reasoning ability.
Output compliance affects the score. Sonnet’s 66 malformed responses cover 21 invoice, 30 date, and 15 refund trials. Raw answers include explanations where prompts required only a number or JSON. The score measures contract compliance as well as answer correctness, not pure reasoning ability.
Each model row totals 180. The final row totals all 1,800 responses. Categories record failed checks, not their underlying causes.
All 53 truncated responses remained failures.
The study requested 8,192 tokens and low reasoning where supported. Sixteen trials needed replacement requests because their earlier attempts returned no answer. One replacement also returned no answer and required another dispatch, giving 17 replacement dispatches for 16 selected answers. No returned answer was retried. The companion preserves every attempt.
Known study bills were $5.1032, plus $0.5383 reserved for unresolved bills. Reader plans estimate $0.97 for Luna’s 180 study trials and $19.46 for Sol’s. Routing, dates, and original concurrent/resumed scheduling limit comparisons with the serial reader runner.
How do public LLM benchmarks compare with your own?
We retrieved an Artificial Analysis snapshot through our benchmarks API on 20 October 2026 at 10:03 UTC, matching exact canonical model versions.
Spearman correlation compares ranking order, from −1 for reversed rankings to +1 for matching rankings. Across ten models:
Agreement was weak in this sample. Different tasks and settings prevent conclusions about general benchmark reliability. Use public scores for shortlisting, then evaluate your workload.
How many times should you run each task?
In 68 of 600 model-task pairs, the model passed some repeats and failed others. Retain every repeat. These are repeated observations of 60 cases, not 180 distinct cases per model. Close rankings need more held-out cases and repeats.
FAQ
How many test cases do I need?
Start with a small set covering one workflow’s common inputs, boundaries, and known failures. Ten carefully reviewed examples can support initial exploration, but they don’t provide a statistical guarantee. Expand with representative held-out cases when differences are close or failures costly. Repeats assess consistency; they don’t replace coverage.
Can I use an LLM judge?
Yes, for properties such as tone or explanation quality that resist exact checks. Validate its rubric against human-labeled examples and inspect disagreements. Keep code checks for calculable answers and required schemas. Our LLM-as-a-judge guide explains that approach.
Next steps
Build a representative set from your application using our golden eval dataset guide. Compare candidates, then gate pull requests on the checks.
