Methodology and tasks

This is RS9's own benchmark: the tests we ran and the results we got. It is not a research paper. We only need it to be fair, consistent and openly stated, so you can see how each score was produced and rerun the tasks yourself.

How a run works

• Same start for everyone. Each agent starts clean with the same brief, the same files and the same time limit.
• One attempt per task, up to 30 minutes each. Every attempt runs in a fresh container. The agent never sees the tests or the answers.
• The time limit is in the prompt. From the next run on, each agent is told its time limit in its instructions, so it can plan its work around it. In this first run they were not told.
• Real work, made safe. Each task is a reconstructed version of work RS9 does, built from made-up data and small repositories, so no client material is used. Research tasks use a supplied pack of documents rather than browsing the web.
• Everything is written down. For every run we record the agent's client version and settings on the day. If an agent can't be run unattended, it is left out of that run and we say so.

How it is scored

• Pass or fail against a checklist. Each task has a written list of what must be right. An attempt passes only if every item does. Items passed still count towards partial credit, explained below.
• Code and data tasks are checked by automated tests.
• Writing and research tasks are checked by one AI judge working through the checklist. The judge comes from a provider that is not one of the agents being tested or, failing that, is never the same provider as the agent it is judging. It does not know which agent wrote what.
• CAD tasks are checked by building the model the agent wrote and measuring it: the overall dimensions, the holes and cutouts, and whether the parts fit together as the spec says.
• Front-end design tasks are rendered in a real browser at desktop and phone width and measured. A vision-capable AI judge then compares the screenshots against a checklist, without knowing which agent made them.
• Human spot-checks. Dale checks a handful of judge decisions and anything that looks wrong.
• Also recorded: the time each attempt took, and its cost where there is one. An attempt that runs out of time is shown as “Timed out” and counts as not passed. An “Error” means the agent itself failed to run properly.

How we keep it fair

Before publishing, every failure in this run was reviewed against the task's written brief. Where a check demanded more than the brief actually said, we fixed the check and re-graded the saved work. No agent was run again to improve its score. A second reviewer then checked that none of those fixes had bent toward the agents' answers.

How the score works

• The headline score is out of 100. It is the percentage of the tasks an agent passed outright: 15 of 20 tasks is a score of 75. A task only counts if every item on its checklist is right.
• Partial credit tells close results apart. It gives every task equal weight and credits it with the share of its individual checks that passed, including the judge's items on writing and research tasks. Two agents on the same score can differ here, and it breaks ties in the ranking.
• Infrastructure errors are left out. If an attempt failed because our set-up or the provider broke, not because the agent got it wrong, it is not counted for or against the agent, so a score can be out of fewer than 20.
• A few points either way means little. Each task is attempted once, so one task going the other way moves a score by about eight points.

How cost is calculated

• We count tokens, then price them. For every attempt we record the tokens the agent used: what it read, what it reused from its cache, what it wrote and how much it reasoned. We multiply those by the provider's public API price list, in US dollars. Cached reading is charged at the cheaper cached rate, and reasoning at the output rate.
• Subscription agents don't bill per task. Their cost here is what the same tokens would cost at the provider's public API price, not what anyone paid. It is a fair way to compare, not a receipt.
• OpenRouter models show a real charge. For the models we run through OpenRouter, the cost is what we were actually billed. That can be well below list price, because the host serving the model may charge a discounted rate. We keep the list-price figure too, and note it on each agent's card.
• Some costs are lower bounds. When an agent is stopped at the time limit it may not report its final usage, so its cost is slightly undercounted. If an attempt reports no usage at all, that agent's total is shown as “price unknown” rather than a partial figure.
• Cost per pass is the total cost divided by the tasks passed, so an agent that is cheap but fails often does not look better than it is. If we can't find a public price, the page says “price unknown”, never $0. An agent that passes nothing has no cost per pass.

How we track changes over time

• The same tasks, rerun on a schedule. The benchmark is rerun regularly, and each run keeps its own dated page. The “Over time” chart on the results page plots every agent's tasks passed, and its cost per pass, across those runs.
• Version changes are marked. We record each agent's model and client version every run. When either differs from that agent's previous run, the point gets a diamond and the tooltip says “version changed”, so a jump can be tied to an update rather than luck.
• Read small moves with care. With one attempt per task, an agent moving up or down by a task or two between runs is normal wobble. A steady drift, or a jump right after a version change, is what is worth noticing.

Caveats

• One attempt per task. Agents are not perfectly repeatable, so a difference of a task or two between agents means very little.
• These are RS9's tasks. They are not a general measure of which agent is best, and an agent that does well here may not do well on your work.
• A dated snapshot. Each result describes the versions and settings tested that day. Agents change often, so older runs are kept as dated pages rather than updated.
• Different set-ups. Most agents are run through their own apps on a subscription. One uses a model through OpenRouter. Compare times and costs with that in mind.
• Times are not like for like. Each time includes the provider's server speed and queueing, so a slow result can reflect a busy server rather than a slow model.

Run it yourself

Each run page has a download of its full results (a JSON file), so you can check our numbers. See the latest results.