Paired comparisons, organic prompts, revealed preference.
Most model evaluations ask annotators to judge synthetic prompts. Verso does neither. The prompts come from people doing real work, and the preference is the choice they make when shown two answers. This page explains the design and what it controls for.
The paired design
Every data point in the Verso dataset has the same structure: one prompt, two models, one preference. The user asks a question on ChatGPT, Claude, Gemini, or Perplexity. Verso sends the same prompt to a second model and displays both answers side by side. The user picks the answer they prefer — or declares a draw.
The pairing is the key. Because both models answer the same prompt, prompt variance is eliminated. You don't need to wonder whether Model A got harder questions than Model B. They got the same questions. A preference difference between two models is a real difference, not an artifact of the prompt distribution.
Organic prompts vs. synthetic prompts
Synthetic prompt sets — whether hand-written by evaluation teams or generated by another model — have a known problem: they don't look like what people actually ask. The distribution is wrong. The phrasing is wrong. The follow-up patterns are missing entirely.
Verso prompts are organic. They come from people in the middle of their work — writing code, planning trips, comparing products, checking facts. The prompt distribution reflects real demand, not what an evaluation designer thought was representative. This matters because model performance is prompt-dependent: a model that wins on benchmark-style questions can lose on the messy, underspecified questions real users actually ask.
Revealed preference vs. stated preference
Surveys ask people which model they prefer. Verso doesn't ask — it observes. The user sees both answers and picks one. This is revealed preference: the choice is made in context, with both options visible, as part of actual work. Revealed preference is harder to collect than stated preference, but it's more reliable because the person has something at stake — they're choosing the answer they'll actually use.
What the pairing controls for
Prompt variance
Both models see the same prompt. Any difference in preference is attributable to the models, not to the questions they were asked.
Position bias
The user's original model always appears first (left). The Verso model appears second (right). This is a known bias — the first answer has an advantage. We measure and report it. In verticals where position effect is large, we flag it in the data.
Time-of-day effects
Both models answer at the same moment. Any performance differences tied to API load, model serving configuration, or time-dependent routing apply equally.
User difficulty variance
Because the comparison is within-user and within-prompt, we don't need to worry about whether some users ask harder questions than others. Each prompt is its own control.
What it does not control for
Transparency requires saying what we don't solve.
- Self-selection. Verso users are people who chose to install a Chrome extension that compares AI models. They are not a random sample of all AI users. The prompts skew toward power users and toward the platforms Verso supports.
- Position effect. The original model always appears on the left. We measure this bias but do not eliminate it. Buyers who need position-balanced data can request a panel where model order is randomized.
- Model identity. Users can see which models are being compared. This introduces a brand-loyalty effect. A user who chose Claude may prefer Claude's answer partly because they already trust it. We report preference rates with and without model identity visible where sample size permits.
Sample size and statistical power
The paired design is efficient. Because each prompt is its own control, you need fewer observations to detect a real difference than you would with an unpaired design. For a 5-percentage-point preference difference between two models on a given vertical, a few hundred paired comparisons are typically sufficient at conventional significance levels. We report cell sizes and confidence intervals alongside every rate we publish.
The current dataset contains 14,320 distinct prompts across 5 verticals and 14 model comparisons, collected since 2025-02.
Taxonomy
Every prompt is classified into a vertical (shopping, travel, code, research, general) and a sub-category within that vertical. Classification is automated and audited. The taxonomy is structural — it maps the shape of real demand — and it is one of the data products we sell. The categories are not arbitrary: they emerge from clustering real prompts, then stabilizing the boundaries once clusters are robust.
Longitudinal drift
Model versions change. A model that preferred Brand X in January may prefer Brand Y in March after a fine-tuning update. Verso maintains a fixed panel of prompts that are replayed against new model versions as they ship. This produces a time series of preference and brand-mention changes that is independent of organic prompt variation. The panel is versioned, and every replay is logged with the exact model version identifier.