The setup
Each challenge publishes a brief. Human designers submit exactly one static image of one interface screen. An administrator adds three AI benchmarks: screens generated by current AI tools from the same brief, without a designer steering, refining or picking. Everything is re-encoded on upload so humans and machines are presented identically — same format, no metadata, same neutral grey stage.
“Better than AI” is therefore a narrow, testable claim: this human design was judged better than every AI benchmark, by this audience, with this much certainty.
Anonymous, blind voting
- Nobody — voters, jurors, even administrators reviewing live statistics — sees names. Accounts have none.
- During evaluation, nothing reveals whether a design is human or AI: no labels, no file names, no upload times. Model and tool names are revealed only when results are published.
- You never see your own work as an option. Jury members who also compete are excluded from their own entry in every method.
Pairwise comparisons
Participants of a challenge and jury members see two anonymous designs and choose the better one (or skip — up to 20 skips per round). Left/right placement is random. A voter can cast only one counted vote per pair per round; different voters may judge the same pair.
Model. We fit a Bradley–Terry model: the probability that design i beats design j is σ(θᵢ − θⱼ), where σ is the logistic function and θ is a latent strength. Each θ has a Gaussian prior N(0, 1.5²). The prior is what keeps sparse, lopsided or disconnected data from producing infinite or misleading strengths: a design that won its only two comparisons is not declared a champion. We take the posterior mode (Newton–Raphson) and a Laplace approximation of the posterior for intervals; rank ranges and “probability of beating every AI benchmark” come from sampling that posterior.
Which pairs you see. Pairs are assigned adaptively: designs with fewer comparisons first, pairs that connect otherwise separate parts of the comparison graph, pairs whose outcome is most uncertain, and similarly ranked designs (where a comparison is most informative). The algorithm also limits how often one voter sees the same design, so very active voters cannot monopolise any work. You can vote once or a hundred times; every valid comparison adds information.
Reported: rank, strength with a 95% interval (shown as the chance to beat an average entry), 90% rank interval, number of comparisons, unique voters, graph connectivity and the unweighted rank for transparency.
Like / Dislike
Every community member can rate designs one at a time — like or dislike, swipe or buttons. A like means approval, not “better than the previous card”. Each design is shown to each person once.
Ranking by raw likes would reward whatever was shown most; ranking by raw rate would crown a design with one like out of one. We use a Beta–Binomial model with an empirical-Bayes prior centred on the round’s average approval with a strength of 4 pseudo-votes. Designs with little data are pulled towards the average until evidence accumulates. We report likes, dislikes, unique voters, the raw rate and the posterior approval with a 95% interval.
Approval rates and Bradley–Terry strengths measure different things and are never placed on one scale.
Expert review
Jury members score each design on the challenge’s published criteria (by default Overall Design Quality, 1–7). Criteria are frozen for a round once it opens. A packet is valid only if it scores every design in the round (except the juror’s own) on every criterion — so nobody can boost a favourite by scoring only that one. Incomplete packets are kept as drafts and excluded.
Scores are normalised to 0–1 per criterion and combined with the criterion weights. Because valid packets cover the whole round, a lenient or harsh juror shifts every design equally and does not change the order. We report the mean with a t-based 95% interval, the number of valid experts and their agreement (Kendall’s W).
Statistical confidence
Every method gets a confidence label — insufficient, low, moderate, high — based on how much data there is, how evenly it covers the entries, whether the comparison graph is connected and how well adjacent ranks are actually separated. It is a description of the evidence, not a promise: no number of votes guarantees a decisive ordering, because two designs can genuinely be equally good. When participation is low, the challenge still runs; the results simply say so.
Reputation & voting reliability
Two separate things influence how much a single vote counts:
- Reputation — earned by taking part and doing well (e.g. +10 for an evaluated design, +15 for a top-10% placement in a method, +10 for credibly beating every AI benchmark, small capped amounts for voting), plus administrator-set starting values such as the jury grant.
- Voting reliability — inferred from hidden consistency checks. Occasionally (about 8% of requests, never within 8 comparisons of the original) you are shown a pair you already judged, with sides swapped. These repeats never count towards any ranking. Disagreeing with yourself is normal on close pairs, so a mismatch on a near-tie carries little weight, and one mismatch barely moves anything; only a sustained pattern does.
A vote’s weight is reliability factor × reputation factor × activity factor, always ≤ 1. Reputation contributes 0.7–1.0, reliability 0.25–1.0 (new voters start near 0.81), and the activity factor keeps one tireless account from dominating: beyond 40 votes in a round, each additional vote counts progressively less, so total influence grows only with the square root of activity. Weights are frozen when a round closes, which makes published results reproducible. The full formulas are in EVALUATION.md in the source repository.
Why methods disagree
Participants comparing side by side notice details; a quick yes/no reflects first impressions from a broader audience; jurors apply explicit criteria. Each is a legitimate lens. The Compare methods view shows rank correlations (Spearman’s ρ and Kendall’s τ, with sample sizes and, where meaningful, intervals) and highlights designs the methods place far apart. A high correlation means the audiences agree, not that they are right.
What results don’t mean
- There is no objectively correct measurement of design quality. We measure the judgements of particular people under particular conditions.
- A rank without its interval is half a fact. Overlapping intervals mean the order is not settled.
- The three methods are never averaged. Any “overall winner” you compute yourself is your own model, not ours.
- Beating three AI benchmarks says something about those benchmarks, generated at a point in time with specific tools — not about AI in general.
Questions about the method are welcome; so is being proven wrong. See it applied