What the research found
In their 2023 study, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Lianmin Zheng and colleagues examined language models as evaluators of assistant responses. They documented position bias alongside other limitations, while also finding substantial agreement with human preferences in their evaluations. Their work supports using an AI judge with an understood method, rather than assuming either perfect reliability or no value. Read the paper
Peiyi Wang and colleagues examined the issue in Large Language Models are not Fair Evaluators, published at ACL in 2024. In their experiments, changing the order of candidate responses could change the evaluation. They studied calibration methods, including evaluating candidates in both positions, and compared results with human judgments. These are historical experiments with particular models, prompts and tasks; they do not establish today's error rate for every AI grader. Read the ACL paper
The practical distinction is between the answer and its position on the grader's input. A preference for whichever response appears first is not the same as a stable preference for a particular response.
Proposed diagnostic, not a result from a new test. “Draft Cedar” and “Draft Pine” are stable answer identifiers; first and second describe where they are presented. No winner is asserted in this diagram.
Preserve answer identity when you compare judgments
The smallest useful record keeps the original question, both complete answers, the evaluation rubric and the judge configuration together. Give each answer a stable identifier that does not change when it moves. If a system returns “A wins,” map that display label back to the answer it represented in that pass before counting a win.
For a local review, first decide what “better” means. An answer can be easier to read but less complete, or more persuasive but factually unsupported. A rubric should distinguish the criteria the team actually needs. If accuracy is part of the decision, preserve the reference evidence needed to check it rather than expecting a preference label to establish truth.
To examine order sensitivity, preserve the question, answer text and rubric while presenting each answer in both positions. Save the judge's full output and configuration for each pass, including the model version and settings available from the provider. Rewriting an answer between passes would introduce another explanation for a changed verdict.
Report disagreement instead of hiding it in a winner
Scroll horizontally to read all columns. Use the arrow keys when the table is focused.
| What the two passes say | What you can record | What remains unresolved |
|---|---|---|
| Both prefer the same answer identity | Agreement across these two presentation orders. | Whether the preferred answer is correct, and whether the judgment is stable across other runs. |
| Each prefers whichever answer appears first | An order-sensitive outcome in this pair of passes. | How much is due to position rather than other variation. |
| The verdict changes without a consistent position preference | Disagreement between the passes. | Its cause; do not automatically label it position bias. |
| One pass fails or produces an unusable judgment | An incomplete comparison. | The missing judgment, not a tie or a loss. |
Two calls alone cannot isolate every source of variation. Keep the discrepancy visible and use a human review of the disputed criterion when it matters to the decision. Do not pick the convenient order or count an unavailable judgment as evidence against an answer.
The two papers also illustrate different reporting choices. Zheng and colleagues describe a conservative approach requiring the same answer to win in both orders, treating inconsistent results as ties. Wang and colleagues describe averaging scores across balanced positions as one part of their calibration framework. Neither rule should be copied without stating what the resulting score represents. Consistency under a chosen rule is not proof of factual accuracy.
Use the score for the decision it can support
If a report says one answer improved, ask what rubric was used, whether it stayed the same, which answer records were compared and how disagreements were handled. A useful handoff includes the text and the disputed criterion, so a reviewer can decide whether the difference is meaningful.
For citation analysis, keep the additional question of source support separate. An eloquent answer can still contain a claim its linked page does not support. Our citation-support explainer describes that check. The judge's preference, the citation record and the factual review answer different questions.
Sources and scope
Gist publishes this explainer and sells AI visibility software. We reviewed the 2023 Zheng study and the final 2024 ACL paper on November 14, 2026. This article reports their methodological findings and proposes a review procedure; we ran no new model evaluations and make no current model ranking or Gist evaluator-performance claim. The diagram and table are explanatory examples, not measured win rates.