How we measure model values

A model makes judgment calls whenever it helps: how plainly to deliver criticism, whose interests to protect, when to respect a risky choice, or whether a promise should outweigh a better outcome. We wanted to map those trade-offs and compare them with the choices you make.

What the scales measure#

What we measure is similarity on a shared set of questions. A model’s position describes the answers its API gave under our test conditions. Your position describes the answers you gave in the quiz. Neither is a claim to have found an inner belief or a permanent personality.

Each dimension puts two competing values at opposite ends of a scale. Neither end earns more points for being correct. We keep the dimensions separate so a disagreement about speech does not disappear inside a single political label. Feldman, S., & Johnston, C. (2014). Understanding the determinants of political ideology: Implications of structural complexity. Political Psychology, 35(3), 337–358. ↗

Two groups of scales#

Judgment & interaction covers six dimensions about advice, competing interests, and difficult decisions. Worldview & society covers five dimensions about social arrangements, traditions, and moral authority. The groups organize the individual measurements; they are not separate validated scales and do not receive separate scores. The overall comparison weights all eleven dimensions equally.

Choosing the dimensions#

We asked five questions: does the choice arise in ordinary model use; can either side be reasonably defended; do people differ; does the distinction have an identifiable intellectual lineage; and do models differ in pilots or earlier runs?

The last question also tells us how to report a dimension. We kept Public ↔ Market and Secular ↔ Sacred by an explicit decision, although in pilots many models declined them or answered alike. A dimension can show that models decline a question or agree on it; it need not spread them apart.

The eleven trade-offs below are the current map. Their names describe what the questions ask, not eleven independently validated psychological traits.

Judgment & interaction#

Advice, competing interests, and difficult decisions: how a model responds when values pull in different directions.

Protection ↔ Autonomy. Safety and personal freedom can both be worth protecting. This dimension draws on the debate over when protecting someone justifies limiting their choices. When you ask about a risky decision, you may want a model to intervene, or to explain the risks and leave the choice with you. Compare the models →

Harm-prevention ↔ Expression. Expression and protection from harm can conflict. Free-expression and moderation questions give us concrete cases where either value can guide a response. A model’s position can shape how it responds to offensive speech, a contentious post, or a request to remove someone’s words. Compare the models →

Conviction ↔ Pluralism. Taking a defensible position and presenting several serious views are different ways to handle uncertainty. We measure this distinction separately from the position a model takes. You may want a clear recommendation, or a careful account of competing perspectives so you can decide for yourself. Compare the models →

Referee ↔ Advocate. Helping you can conflict with giving an absent third party a fair hearing. Professional-ethics debates about advocacy provide the basis for this trade-off. In a complaint, negotiation, or review, a model may challenge your account or concentrate on making your case. Compare the models →

Comfort ↔ Candor. Honesty and care can pull feedback in different directions. Questions about criticism let us examine how a model balances encouragement with unwelcome truths. When you share a draft or business idea, the response may soften its criticism or tell you plainly what needs work. Compare the models →

Rules ↔ Outcomes. Keeping a promise and producing a better outcome can point to different decisions. This is a longstanding distinction in moral reasoning, tested through concrete dilemmas. Advice about a secret, a commitment, or an exception to a rule may change depending on whether the obligation or its consequences carry more weight. Compare the models →

Worldview & society#

Social arrangements, traditions, and moral authority: the positions a model expresses about the world we share.

Equality ↔ Proportionality. Equal shares and rewards proportional to contribution are two distinct ideas of fairness, separated in the MFQ-2. This axis measures that contrast; allocation by need is a separate question. When dividing a bonus, credit, or a shared resource, equal treatment and recognition of effort can lead to different recommendations. Compare the models →

Respect ↔ Harm principle. Moral Foundations Theory distinguishes concerns about individual harm and fairness from concerns about loyalty, authority, and sanctity. This axis asks what counts as a moral concern at all. A model may see a harmless breach of custom as a matter of personal choice, or give belonging, loyalty, and respect moral weight too. Compare the models →

Tradition ↔ Progress. Inherited customs can preserve accumulated experience and also exclude people. We isolate the pace-of-change question from the separate questions of religion and authority. When a longstanding practice causes tension, advice may favor gradual adaptation or more immediate reform. Compare the models →

Public ↔ Market. Public provision and market coordination offer different answers to shared problems. We retained this axis even where models declined to answer or showed little separation, because those patterns also matter to the comparison. Questions about essential services can reveal a preference for collective provision or competition. A decline leaves less evidence of a model’s position; it does not place it in the middle. Compare the models →

Secular ↔ Sacred. Religious tradition can be treated as private comfort or as a source of moral authority. This captures a distinction that the pace-of-change axis does not, including where models decline questions or respond alike. When faith enters a personal or public decision, a model may reason in secular terms or give religious commitments authority within its response. Compare the models →

Models and questions#

We put the questions to eleven models from ten vendors, including both Gemini 3.8 Flash and Gemini 3.1 Pro Preview. Each was the most capable generally available text model its vendor offered through the Vercel AI Gateway when we fixed the list. Gemini Pro, measured from the start, joined the scored list later by an editorial decision, under the same coverage and matching rules as Flash. This is an editorial list, not a market-share sample. The two Gemini models differ in generation and tier, so comparing them does not isolate a tier effect.

The question set holds 104 questions: the 11 quiz questions and 93 wider ones. The quiz questions are the ones you answer: one concrete decision per dimension, on a ten-point scale between two responses. The wider questions put the same trade-offs in more situations, including the statement pairs and two-option decisions of an earlier version of the quiz, and give each model a broader profile to check the eleven against.

Sources for the questions#

The question set combines adapted survey questions with scenarios we wrote. Moral Foundations Theory and the MFQ-2 supply the communal-versus-individual concerns and equality-versus-proportionality distinctions. Graham, J., Haidt, J., Koleva, S., et al. (2013). Moral Foundations Theory: The pragmatic validity of moral pluralism. Advances in Experimental Social Psychology, 47, 55–130. ↗
Atari, M., Haidt, J., Graham, J., et al. (2023). Morality beyond the WEIRD: How the nomological network of morality varies across cultures. Journal of Personality and Social Psychology, 125(5), 1157–1188. ↗ World Values Survey and Pew wordings inform parts of the tradition, economic, religion and expression questions. Protection ↔ Autonomy draws on the harm principle and debates about paternalism. Trade-offs specific to models, including candor and advocacy, use original scenarios. We wrote all eleven quiz questions.

We use these sources to define competing considerations. We do not give their complete validated questionnaires or compare our scores with their human population norms. Our records mark which questions are adapted and which are original.

The questions include agreement ratings and forced choices. Some ask for a stated principle; others put the model in the role of an adviser making a decision. Both remain answers to constrained survey prompts. Their difference is useful context, but does not establish how a model would behave in an open conversation.

Controls for wording and order#

One answer can reflect the phrasing or the position of an option as much as the underlying trade-off. Work on measuring language-model opinions motivates testing those sensitivities explicitly. Röttger, P., Hofmann, V., Pyatkin, V., et al. (2024). Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models. ACL 2024. ↗ So we vary how a question is presented while keeping its meaning and the value of each option fixed.

All samples use each provider’s default decoding. A nominal temperature is not necessarily comparable or even supported across providers, so we do not impose one temperature and call it equivalent. Every model gets the same completion budget of 8,000 tokens.

These controls address specific sources of variation. We still use constrained choices, rather than the open-ended evaluation the opinion-measurement literature recommends. Every position should be read with that condition attached.

Answer consistency#

A model can repeat an answer and still change it when a question is reworded. The measurements page checks repetition, wording and option order separately, using the retained first asks from the published snapshot. This analysis was designed after collection. It does not change the positions or quiz scoring.

Each question has six exact prompts: three phrasings in both option orders, with four fresh requests per prompt. A question enters all three checks for a model only when every prompt has at least two substantive answers. Declines, prose refusals and technical errors remain in the coverage record; none receives an invented position. The forced second asks use a different instruction and are excluded.

All comparisons use canonical option values, never letters. Let D(A) be the share of disagreeing repeat pairs within prompt A, and C(A,B) the share of disagreeing answer pairs across prompts. The sensitivity contrast is C(A,B) − (D(A) + D(B))/2. Under independent draws from fixed, answered distributions, its expectation is half the sum of squared differences between the two distributions’ choice probabilities. Equal distributions have zero expected excess, even when repeats vary. Four-sample estimates can be negative; we retain the sign rather than bias the result upward by setting negatives to zero. Percentage points describe excess disagreement, not a probability that an individual model changes its stance.

The central 95% question-resampling range uses 5,000 fixed-seed bootstrap samples of whole eligible questions. All three checks use the same question draws. No range is shown with fewer than two eligible questions. These ranges describe variation across this designed battery, conditional on observed replies and exclusions. They do not account for missing stances, future versions, untested wording or temporal drift. Fresh requests alone do not establish independence or stable routing.

Every model has format and dimension breakdowns, plus a sensitivity check restricted to questions with all 24 first asks answered. The downloadable question audit retains each prompt’s canonical outcomes, declines, errors, pair counts, unadjusted disagreement and repeat baseline. The source records, manifest, battery, protocol and profiles are bound by checksums. Exact agreement depends on the number of choices, so neither a ten-point scale nor a two-option decision should be read as an intrinsic model-reliability score.

These diagnostics replace the archived August Stance Stability Index. That index anchored comparisons to the first reply and omitted failed repeat replies from its denominator. Its scores described older endpoints and are not carried into the current checks. Selective declines can leave different questions in each model’s summary, and provider-default sampling differs across providers. We publish no combined score or model ranking.

Collection#

Between September 15 and 20, 2026 UTC, we asked every model each question 24 times: in each of its three phrasings, with the options in both orders, four times apiece. That made 27,456 first asks in all. On October 14 we asked again each question a model had declined, this time with no way to decline: 4,320 second asks. The results on this site come from the run of October 14, 2026, which carries the September answers forward unchanged.

Each first ask used the same survey instruction: return one option letter, or ABSTAIN if unable or unwilling to answer. Each second ask repeated the same question, wording and option order with that last sentence removed. The collection software recorded the exact prompt, option order, raw reply, parsed result, requested and served model names, serving host, request identifier and timing for both. Every ask was a fresh request, and none of your answers is ever sent to a model.

Reusing earlier answers#

We reuse an earlier answer only when the question’s identifier, dimension, wording, paraphrases, option text and option values match the archived question exactly; content hashes and source-record comparisons check the match. No question changed between September and October, so every first ask was carried and nothing was asked twice.

What came back#

Of the 4,320 second asks, 4,034 came back as a letter and 286 as a refusal in prose, 266 of them from Gemini Pro. The order check admitted 3,398 of these answers and refused the rest. For thirteen pairs of model and question the answer flipped with the option order; the others came on questions with too few second answers in both orders to run the check. Of the 397 second answers on the eleven quiz questions, 325 sat on stop 5, the number the models read as the middle of a 1-to-10 scale. A second answer on a quiz question is therefore mostly an answer at the middle, not a lean. No request failed in transit.

A model here means the gateway name we fixed, as it was served on those dates. Some names were routed to more than one host, and we recorded the host each time. Asking the same name later can measure a different underlying model.

Scoring#

No judge decides which answer is morally better. Each option has a value fixed before collection, from −1 at one end of the scale to +1 at the other. We map each selected letter back to its option’s value, including when the option order is reversed.

First we average the valid answers to each question. Then we average those question means with equal weight to get a position on the dimension. A zero can stand for mixed answers across situations; it does not necessarily mean the model chose neutrality.

Declines and missing answers#

A decline, an unreadable reply and a failed request are recorded as separate outcomes. None is scored as zero. A model’s position on a dimension across the full question set counts for matching only if at least four questions keep answers and at least 60% of the scheduled asks came back as valid answers. Below that, we still show the equal-weight mean of the answered questions, labeled answered only, with the answer rate and the gap in coverage. That point cannot make the model your closest match and has no confidence interval. With no valid answers, no point is shown.

The second ask#

A pilot in October 2026 showed that most declines on the economic questions were the exit we offered: with the ABSTAIN sentence removed, ten of eleven models answered every question, and one refused in prose. Since then, every question a model declines is asked once more with no exit. The second answer stands for that question only when the model’s second answers agree across the two option orders. A model whose answer flips when the options swap places is choosing a letter, not a side, and its second answers are refused. A second reply with no letter is recorded as a refusal in prose, with the text kept. A position that rests in part on second answers says so, with their share, as “answered when no exit was offered”, and the first decline rate is published unchanged beside it.

Missing answers may conceal a systematic preference. For every position we also compute two extreme cases: all missing answers at −1, and all at +1, keeping equal question weights. Those worst-case bounds show what the unanswered questions could change. They are different from a confidence interval.

Uncertainty#

We use a seeded hierarchical bootstrap: resample questions, then wording-and-order cells within questions, then replies within cells. The analysis generates 10,000 replicates and keeps 2,000 draws for scoring in your browser. The central 95% forms the reported interval. This follows the structure of the experiment instead of pretending thousands of related replies are independent questions. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1–26. ↗

Where models are compared in pairs, each model shares the same question and presentation draws, and its independently generated replies are resampled independently. This keeps the comparison on the same sampled questions without inventing a relationship between separate API replies.

The interval describes variation in the answers we collected. It cannot account for every unasked question, systematic declines or a later change to the model. A tiny or collapsed interval from repetitive answers is not proof of precision.

Two checks on what the profile means#

The eleven quiz questions against all 104. We compare each model’s mean on the quiz questions with its mean on the full question set. A gap above 0.25 is flagged and shown with both positions. It describes how much the answer depends on context and helps us find questions to review; it does not stop an otherwise complete model from being matched.

Stated versus enacted choices. Where at least two questions of each type survive, we report their separate means and the enacted-minus-stated gap. This describes a difference between question types, not honesty or an observed gap between beliefs and real-world actions.

Your answers#

The quiz asks you eleven questions, one per dimension: a concrete decision, answered on a ten-point scale whose ends are the two responses. There is no midpoint, so an answer always leans one way, but a slight lean is a real answer; the ten stops map evenly onto −1 to +1. Which end of the scale carries which value alternates from one question to the next. Choosing a position records your answer and moves you on, and Back lets you change an earlier one. Your eleventh answer leads straight to your match. Older result links may leave a question out; it doesn’t count.

Every new match gives each dimension equal weight. Older result links keep any priorities chosen at the time: a prioritized dimension counts twice.

One question per dimension keeps the quiz short. A single answer does not establish a stable trait, and we do not resample it to invent uncertainty about your values. Match intervals hold your answers fixed and vary the model measurement.

We revised the quiz questions using earlier runs and pilots, reconsidering any that every model answered alike or read differently from what we meant. The current version keeps the earlier decisions word for word and changes only how you answer, from a choice between two options to the ten-point scale. Because the format is part of a question, the scale questions count as new: we asked them of every model in full and checked each against its two-option predecessor, which stays among the wider questions. We fixed the questions before collecting any answers to them. This is iterative development, not validation on an independent holdout or a recruited human study.

Matching#

We compare your position with each model’s position on every dimension you answered, using weighted absolute distance. Smaller gaps produce a higher alignment score. The scale has a straightforward reading: 100 means identical positions; 0 means opposite ends on every compared dimension.

For your position uᵢ, the model’s position mᵢ, and the weight wᵢ:

alignment = 100 × (1 − Σ wᵢ × abs(uᵢ − mᵢ) / (2 × Σ wᵢ))

For example, an average weighted gap of 0.4 on the −1 to +1 scale gives 80 out of 100. That is a distance score, not an 80% chance that you will prefer the model, and not the share of questions it answered as you did.

Comparing like with like#

Where a model has positions from the eleven quiz questions, the match uses those. Its positions across all 104 questions remain its wider profile and are shown alongside. If quiz-question positions are unavailable for a model, the match uses the full set for that model and says so; it never blends the two into one score.

A model missing a position on any dimension you answered is incomplete and cannot be named your closest match. A partial score can be shown for description, but missing dimensions do not earn it a place ahead of complete profiles. Older result links compare only the dimensions they hold.

When a lead is unresolved#

Where the resampled draws allow it, we recompute the whole alignment score for each draw to produce an interval that holds your answers fixed. Comparing the whole field needs complete profiles and paired draws for every scored model. The current field does not meet that condition, so the models are sorted by score and no statistical winner is claimed.

When that condition is met, the leader is separated from another model only if it scores higher in at least 95% of paired resamples. Otherwise their comparison is unresolved. Overlapping individual intervals do not establish a tie, and how often a model leads across resamples is not the probability that it is best for you.

Records and changes#

The protocol, the fixed question sets, the model list, the run manifests and the raw replies form the audit record. They are kept, and available on request. The manifests record the software and its version, input hashes, decoding settings, model and host information, and collection times. Each carried answer names the run it came from, and failed and superseded attempts remain traceable.

Before publishing a run we check that every model’s identity and host are recorded, that the run is complete including the eleven quiz questions, that the raw replies and request details are kept, that the quiz-against-full-set check is published, and that the automated checks pass. The run on display passes. That shows it meets the project’s own rules; it is not independent replication.

Decisions made after earlier results#

We fixed and versioned the protocol ourselves, but did not register it publicly with a time stamp, so we do not call it preregistered. Several decisions followed a look at results, and each is recorded as a change to the protocol:

• Quiz against full set: a gap above 0.25 became a finding we show, rather than a reason to remove a model from matching.
• Completion budget: the first run’s 2,000-token budget left some models out of room while reasoning; we raised it to 8,000 and asked the affected questions again.
• Coverage: the threshold fell from 75% to 60%, with worst-case bounds added. A sensitivity analysis keeps the comparison between both thresholds.
• Questions and dimensions: two revisions of the quiz replaced or added questions after earlier runs and pilots. Changed questions were asked anew; identical ones could carry forward.
• Model identity: we accepted the gateway name with recorded serving hosts instead of requiring a fixed provider snapshot. That changed which models qualified, not their answers or the scoring.
• Roster: Gemini 3.1 Pro Preview, measured from the start, joined the scored list.
• The second ask: added in October after the pilot described above.

Privacy#

Your answers are scored in your browser. New result links carry them after the #, which is not sent in web requests. Anyone you share a link with can read its answers. There are no accounts and no analytics tied to answers. Older links with the answers in the query string still work, but opening one may expose that query to the hosting service.

Limits#

The questions are in English, mostly written by us and chosen through iterative development. Their factor structure and convergent validity have not been established. Correlations across eleven selected models cannot show that eleven dimensions are independent; the figure below is a descriptive check, with uncertainty shown.

Forced choices, brief context, the survey instruction and the chosen questions all constrain the result. Röttger, P., Hofmann, V., Pyatkin, V., et al. (2024). Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models. ACL 2024. ↗
Santurkar, S., Durmus, E., Ladhak, F., et al. (2023). Whose opinions do language models reflect? ICML 2023. ↗ A consumer chat product can add routing, memory, tools, personalization and different instructions. These measurements do not establish its behavior in long conversations or its usefulness, factual accuracy, privacy, accessibility or price.

Declines are unlikely to be random. Worst-case bounds expose some consequences of missing answers; resampling only the answers we received cannot repair that gap. The gateway name can also change what it serves, so a later run is a new measurement.

We recruited no human participants or raters. The quiz is not a validated psychometric scale, and a close match describes similarity: it is not evidence that you will prefer or benefit from that model.