Research explainer

An AI grader can change its verdict when the answer order changes

Before using an automated score to declare one answer better, check whether the preference survives a change in presentation. Research on AI judges shows why the scoring method deserves its own review.

Suppose a team compares two answers to the same buyer question. An AI grader prefers the first, so the team treats that answer as the stronger version. But if the grader changes its choice when the answers trade places, the result is harder to interpret: the answer text stayed the same while the assessment changed.

This is a separate issue from whether an assistant retrieved the right source or cited a brand. It concerns the system used to judge the answers afterward. Teams using automated evaluations for content, support responses or AI visibility reporting need to inspect that judging step as well as the generated text.

What the research found

In their 2023 study, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Lianmin Zheng and colleagues examined language models as evaluators of assistant responses. They documented position bias alongside other limitations, while also finding substantial agreement with human preferences in their evaluations. Their work supports using an AI judge with an understood method, rather than assuming either perfect reliability or no value. Read the paper

Peiyi Wang and colleagues examined the issue in Large Language Models are not Fair Evaluators, published at ACL in 2024. In their experiments, changing the order of candidate responses could change the evaluation. They studied calibration methods, including evaluating candidates in both positions, and compared results with human judgments. These are historical experiments with particular models, prompts and tasks; they do not establish today's error rate for every AI grader. Read the ACL paper

The practical distinction is between the answer and its position on the grader's input. A preference for whichever response appears first is not the same as a stable preference for a particular response.

Two hypothetical grading passes preserve the same question and answer texts but reverse their presentation order; the output records answer identity separately from position.

Proposed diagnostic, not a result from a new test. “Draft Cedar” and “Draft Pine” are stable answer identifiers; first and second describe where they are presented. No winner is asserted in this diagram.

Preserve answer identity when you compare judgments

The smallest useful record keeps the original question, both complete answers, the evaluation rubric and the judge configuration together. Give each answer a stable identifier that does not change when it moves. If a system returns “A wins,” map that display label back to the answer it represented in that pass before counting a win.

For a local review, first decide what “better” means. An answer can be easier to read but less complete, or more persuasive but factually unsupported. A rubric should distinguish the criteria the team actually needs. If accuracy is part of the decision, preserve the reference evidence needed to check it rather than expecting a preference label to establish truth.

To examine order sensitivity, preserve the question, answer text and rubric while presenting each answer in both positions. Save the judge's full output and configuration for each pass, including the model version and settings available from the provider. Rewriting an answer between passes would introduce another explanation for a changed verdict.

Report disagreement instead of hiding it in a winner

Scroll horizontally to read all columns. Use the arrow keys when the table is focused.

What the two passes sayWhat you can recordWhat remains unresolved
Both prefer the same answer identityAgreement across these two presentation orders.Whether the preferred answer is correct, and whether the judgment is stable across other runs.
Each prefers whichever answer appears firstAn order-sensitive outcome in this pair of passes.How much is due to position rather than other variation.
The verdict changes without a consistent position preferenceDisagreement between the passes.Its cause; do not automatically label it position bias.
One pass fails or produces an unusable judgmentAn incomplete comparison.The missing judgment, not a tie or a loss.

Two calls alone cannot isolate every source of variation. Keep the discrepancy visible and use a human review of the disputed criterion when it matters to the decision. Do not pick the convenient order or count an unavailable judgment as evidence against an answer.

The two papers also illustrate different reporting choices. Zheng and colleagues describe a conservative approach requiring the same answer to win in both orders, treating inconsistent results as ties. Wang and colleagues describe averaging scores across balanced positions as one part of their calibration framework. Neither rule should be copied without stating what the resulting score represents. Consistency under a chosen rule is not proof of factual accuracy.

Use the score for the decision it can support

If a report says one answer improved, ask what rubric was used, whether it stayed the same, which answer records were compared and how disagreements were handled. A useful handoff includes the text and the disputed criterion, so a reviewer can decide whether the difference is meaningful.

For citation analysis, keep the additional question of source support separate. An eloquent answer can still contain a claim its linked page does not support. Our citation-support explainer describes that check. The judge's preference, the citation record and the factual review answer different questions.

Sources and scope

Gist publishes this explainer and sells AI visibility software. We reviewed the 2023 Zheng study and the final 2024 ACL paper on November 14, 2026. This article reports their methodological findings and proposes a review procedure; we ran no new model evaluations and make no current model ranking or Gist evaluator-performance claim. The diagram and table are explanatory examples, not measured win rates.