What we cover, and what we don't.
Every cell in the table below represents a vertical–language pair. Cells with sufficient depth support full preference rates with confidence intervals. Thinner cells support model-vs-model deltas — enough to detect a winner, not enough to publish a stable rate. We say which is which.
Coverage table
Distinct prompts per cell. Updated as the dataset grows.
| Vertical | Language | Distinct prompts | Supports |
|---|---|---|---|
| General advice | EN | 4,200 | Full rates |
| Code & engineering | EN | 3,100 | Full rates |
| Product & brand choice | EN | 2,800 | Full rates |
| Product & brand choice | FR | 640 | Deltas only |
| Travel | EN | 1,900 | Full rates |
| Travel | FR | 380 | Deltas only |
| Research & analysis | EN | 1,400 | Deltas only |
Total: 14,320 distinct prompts across 14 model comparisons, since 2025-02.
What "distinct prompts" means
A distinct prompt is a unique user question, deduplicated by semantic similarity. If five users ask "best noise-cancelling headphones" in slightly different words, that counts as one distinct prompt with five paired observations. The metric that matters for statistical power is distinct prompts, not total comparisons — because paired observations on the same prompt are not independent.
Rates vs. deltas
Full rates
Cells with enough distinct prompts support a preference rate — "Model A is preferred X% of the time on shopping questions in English" — with a confidence interval narrow enough to be useful. This is the primary deliverable for AI labs running model evaluations.
Deltas only
Thinner cells can still detect which model wins on a given vertical, but the rate estimate is too wide to publish. These cells are useful for directional signal — "Model A outperforms Model B on French travel queries" — but not for reporting a stable number.
Growing a thin cell
If a cell you care about is thin — or missing entirely — we can recruit for it. Sponsored panels target consenting users in a specific vertical and language. The panel runs for a defined period, the data accumulates, and the cell moves from deltas-only to full rates. This is how we build depth in verticals that don't occur organically at high volume.