AI Systematic Biases · evaluation
An LLM evaluator can change its preference when the same candidate answers are shown in a different order.
Human analogy: primacy and position effects. This is a behavioural analogy, not a claim that the model has the same mental mechanism.
Why it matters
If an LLM judge selects releases, scores agents or filters generated content, answer order can become an invisible evaluation variable.
Evidence status: strong for LLM-as-a-judge settings; varies across judges, tasks and quality gaps
Evidence level: peer reviewed large scale multi model
Signals
Self-test
Evaluate A vs B and B vs A with identical rubric and fresh context. Treat disagreement as a reliability signal instead of choosing one order as canonical.
Record the exact model identifier, surface, system prompt, relevant settings and run date. A single surprising answer is an anecdote, not a bias measurement.
Mitigation
Applies to
llm as a judge coding and automation agents
Dated model evidence
These findings describe the model or model set tested at the stated time. A newer release needs new evidence.
A study of more than 150,000 evaluation instances found non-random position bias with substantial variation by judge, candidate and task.
Sources
