How we run these tests
Every number on this site comes from the process below. Where it is weak, it says so.
A benchmark is only worth as much as its method. This page describes exactly how a Benchivo challenge is produced, what the numbers mean, and — just as importantly — what they do not mean.
How prompts are chosen
Prompts are generated by a model that is not competing in the challenge. At launch that is Grok. This keeps prompt selection away from both contestants and away from us: we can reject a prompt as unusable, but we do not write the prompts the contestants receive.
Every prompt states its requirements explicitly and numerically, so that most of the scoring can be checked rather than judged. The full, unedited prompt is published on the challenge page.
How models are run
Each model is used through its own consumer web application, in a brand-new conversation, with no custom instructions, no project context, no memory and no retries. Both models receive byte-identical prompts.
The first response is the published response. If a model fails, refuses or produces something broken, that is what appears on the page. We do not fix outputs, and we do not quietly re-run a challenge because a result was disappointing.
What the timings mean
Two times are recorded: time to visible response, from submitting the prompt to the first visible token, and web app completion time, from submitting the prompt to the end of generation.
Both are measured inside the provider's own application. They therefore include network conditions, service load and interface behaviour. They are not model inference latency, and should not be quoted as such. Treat a difference of a few seconds on a single challenge as noise until it repeats across many.
How results are scored
Each result is scored on four components:
| Component | Weight | What it covers |
|---|---|---|
| Stated requirements met | 40% | Checked item by item against the prompt. |
| Automated checks | 20% | Validity, size, structure, rendering, errors. |
| Editorial review | 20% | Written up in full on every challenge page. |
| Community vote | 20% | Reported separately, never folded in silently. |
Weights vary by category — a game is judged more on whether it runs than on how it looks, an SVG illustration more on the reverse. Any category that departs from the default says so on its own page.
The technical score and the community vote are always shown as two numbers. When they disagree, that disagreement is the interesting part, and flattening it into a single figure would throw it away.
Community voting
Anyone can vote once per challenge round. Votes are deduplicated using a salted hash of IP address and user agent; no raw visitor data is stored and no account is required. This is enough to stop casual repeat voting and not enough to stop a determined effort — so votes inform the picture rather than decide it.
Model versions and re-testing
Models change under the same name. Every result records the exact model version and the surface it was tested on, alongside the date.
When a model is updated, we re-run challenges as a new round. Old rounds are never overwritten or deleted — the August result stays exactly as it was published, next to the December one. Over time this is what turns the site from a snapshot into a record of how these systems actually change.
What we get wrong
Single challenges are small samples. Web-app timings are noisy. Editorial review is a human judgement and carries human bias. Community votes reflect whoever showed up. None of these are solved problems, and the honest response is to publish the raw material — the prompt, the outputs, the measurements — so that anyone can disagree with our reading of it.