AI Systematic Biases · prompt and context
A model's multiple-choice accuracy or selected option can change when answer choices are reordered without changing their meaning.
Human analogy: order and position effects. This is a behavioural analogy, not a claim that the model has the same mental mechanism.
Why it matters
Benchmarks, classification prompts and structured agent decisions can produce different results because of layout rather than reasoning quality.
Evidence status: strong benchmark evidence; older model snapshot should be retested on new generations
Evidence level: peer reviewed multi model
Signals
Self-test
Permute answer choices across otherwise identical fresh runs. Compare both accuracy and the distribution of selected positions.
Record the exact model identifier, surface, system prompt, relevant settings and run date. A single surprising answer is an anecdote, not a bias measurement.
Mitigation
Applies to
benchmarks and multiple choice decision support coding and automation agents
Dated model evidence
These findings describe the model or model set tested at the stated time. A newer release needs new evidence.
The paper reported large accuracy gaps across option reorderings on several benchmarks, with the size depending on model and task.
Sources
