RS9 SystemsRS9 Systems Pty Ltd, Sydney

Methodology

How we ran it, how it's scored, and where it falls short.

This is our own benchmark, not a research paper. Every agent gets the same tasks and the same checks. Where something wasn't fair, we say so.

What's in it

20 tasks drawn from the kind of work RS9 does, from software and data to documents, design and CAD. A few “frontier” tasks were added to stretch the top models. Each agent gets one attempt per task.

The tasks themselves stay private. Once a test is public it ends up in training data and the scores stop meaning much, so we publish the results and how they were scored, not the briefs, files or checks.

How a run works

  • Same start for everyone. Each agent starts clean, with the same brief, the same files and the same time limit.
  • One attempt per task. Every attempt runs in a fresh container. The agent never sees the checks or the answers.
  • Real work, made-up data. The tasks are rebuilt versions of work we do at RS9, using made-up data, so no client material is involved.
  • Everything is written down. We record each agent's client version and settings on the day. If an agent can't run unattended, we leave it out of that run and say so.

Time limits

Each task gets 60 minutes. Every agent is told the limit in its instructions, so it can plan around it. Work that's correct when the time runs out counts as a pass. An attempt that's still wrong at the limit is a fail.

The limit started at 30 minutes. When we moved it to 60, we re-ran every attempt that had timed out, for every agent, so all of these results are under the same rule.

How it's scored

  • Pass or fail against a checklist. Each task has a written list of what must be right. An attempt passes only if every item does.
  • Automatic checks first. Anything we can test with code is tested with code. The rest, mostly writing, goes to an AI judge that works through the checklist. It doesn't know which agent wrote what.
  • Judges from outside the field. GLM 5.3 judges the writing tasks and Gemma 4 does the visual checks. Both come from companies that aren't in the benchmark, so no agent is being marked by its own maker.
  • Spot-checks. I check a handful of judge decisions, and anything that looks off.
  • Also recorded. The time and cost of every attempt.

CAD and design checking

For CAD tasks we build the model the agent made and measure it against the brief. For front-end design tasks we load the page in a real browser at desktop and phone width and measure it, then Gemma 4 compares the screenshots with a checklist. It doesn't know which agent made which.

The score

  • Out of 100. It's the percentage of tasks passed. 15 of 20 is 75. A task only counts if every item on its checklist is right.
  • Partial credit. Every task has the same weight and earns the share of its checks that passed. Two agents on the same score can differ here, so it breaks ties in the ranking.
  • Our breakages don't count. If an attempt failed because our setup or the provider broke, not because the agent got it wrong, it's left out. So a score can be out of fewer than 20.
  • A few points means little. One task going the other way moves a score by 5 points.

The headline is each agent's average over its last few runs, because one run on its own is noisy. Every run is also published on its own page.

Keeping it fair

Before publishing, we audited every failure against the written brief. Where a check asked for more than the brief said, we fixed the check and re-graded the saved work. We didn't re-run any agent to improve its score. The only re-runs were attempts broken by our own setup, and the timed-out attempts when the limit went to 60 minutes. A second reviewer then checked that none of the fixes had leaned towards an agent's answer.

Cost

  • Tokens, then price. For every attempt we record the tokens used: read, reused from cache, written and reasoned. Then we price them at the provider's public API rates, in US dollars. Cached reads get the cheaper cached rate. Reasoning is charged at the output rate.
  • Subscription agents. They don't bill per task, so the cost is what the same tokens would cost at public API prices. It's not what anyone paid. It's a fair way to compare, not a receipt.
  • OpenRouter. For the model we run through OpenRouter, the cost is the real charge. That can be well below list price, because the host may discount it. We keep the list-price figure too and note it on the agent's card.
  • Some costs are lower bounds. An agent stopped at the limit may not report its final usage, so its cost is slightly under. If an attempt reports no usage at all, that agent's total shows as “price unknown” rather than a partial figure.
  • Cost per pass. Total cost divided by tasks passed. If we can't find a public price, the page says “price unknown”, never $0. An agent that passes nothing has no cost per pass.

How value is measured

Value is cost per task passed: total cost divided by tasks passed. The Best value board lists agents cheapest first, beside Best results. It never changes the score, which is quality only. Costs are on the basis above, so subscription agents show public API prices and OpenRouter shows the real charge. An agent with no known price goes last as “price unknown”. One that passed nothing shows “no tasks passed”. Bars are on a log scale so cheap and dear agents both fit. A longer bar always means cheaper.

Over time

  • Same tasks, rerun regularly. Each run keeps its own dated page. The “Over time” chart on the results page plots every agent's tasks passed, and its cost per pass, across them.
  • Version changes are marked. We record each agent's model and client version every run. If either changed since its last run, the point gets a diamond and the tooltip says “version changed”. That way a jump can be tied to an update rather than luck.
  • Small moves are wobble. With one attempt per task, a task or two either way is normal. A steady drift, or a jump right after a version change, is worth a look.

Caveats

  • One attempt per task. Agents aren't perfectly repeatable, so a task or two between agents means very little.
  • These are our tasks. It's not a general ranking. An agent that does well here may not do well on yours.
  • A dated snapshot. Each result is for the versions and settings we tested that day. Agents change often, so older runs stay as dated pages and aren't updated.
  • Different set-ups. Most agents run through their own apps on a subscription. One uses a model through OpenRouter. Keep that in mind when you compare times and costs.
  • Times aren't like for like. Each time includes the provider's server speed and queueing, so a slow result can be a busy server, not a slow model.

See the latest results.