How Testero works
Short version: we separate what we measured ourselves from what we merely know, and we show which is which next to every number.
The three evidence levels
We measured it
Our own test runs: fixed tasks, real API calls, timing and cost to the cent. Shown with date and full output.
Someone else measured it
Facts from a named public source, in our own words, with a link and a date. Never sold as our own test.
Plain facts
Prices, limits and capabilities from provider catalogs, refreshed every day, plus general background.
| Label | Meaning |
|---|---|
| Hero tested | Our own test. We ran it, we paid for it, we publish date, conditions and the full output. |
| Source | A fact or assessment from a public source that allows it. Summarized in our own words, linked and dated. Never presented as our own test. |
| General info | General knowledge about a product or category. Not tested and not researched in depth. Meant as orientation. |
| Catalog data | Machine-readable facts from provider catalogs: names, prices, limits, features, release dates. Refreshed once a day. |
Where the catalog data comes from
Names, prices, context sizes, supported features and release dates come from models.dev, an open database of AI models maintained by its community and published under the MIT License (© models.dev). We refresh our copy once a day and show the first-party list prices of the companies that make the models, plus a few specialised inference hosts.
We do not copy descriptions, images or third-party benchmark scores. Summaries on model pages are generated from the facts above. Prices are list prices in US dollars; what you pay can differ by provider, region, caching and batch discounts. An open dataset can contain mistakes or lag behind; if a number matters for a decision, confirm it on the provider’s own pricing page.
Hero tests
Each language model gets the same three tasks, once, with default settings and no system prompt: structured extraction, everyday reasoning, and building a dashboard. We record:
- Time to first token and tokens per second, measured on our server while the answer streams in.
- Real cost of the run as billed by the API.
- Result quality. The two JSON tasks are checked automatically against the correct answers. The dashboard task is checked for ten technical requirements and can additionally be rated by a human editor from 0 to 10; if so, the human rating counts.
Hero Score (0 to 100)
Score = 50% quality + 25% speed + 25% cost
- Quality: average result of the three tasks.
- Speed: average tokens per second on a logarithmic scale; 10 tok/s or less gives 0 points, 300 tok/s or more gives full points.
- Cost: total cost of the suite on a logarithmic scale; $1 or more gives 0 points, $0.0005 or less gives full points.
Limits you should know
- Three tasks cannot describe a model completely. The score is a practical first impression, not a verdict for every use case.
- Each published result is a single run. Models are not deterministic and API speed changes with load and time of day.
- We test through one API route (OpenRouter). Other routes to the same model can be faster or slower.
- Models without a score are not bad. They are simply not tested yet.
Money and independence
Some links on Testero are partner links, marked “Ad”. If you buy through them we may earn a commission at no cost to you. Partners cannot buy scores, rankings or test results. See the affiliate disclosure.
Corrections
Found a wrong number? Tell us via the contact details in the legal notice. We fix errors and say so.
