AI Systematic Biases · evaluation
An LLM evaluator can prefer answers with stronger surface presentation, such as verbosity or fluency, even when those signals are not the target quality.
Human analogy: style-over-substance and fluency effects. This is a behavioural analogy, not a claim that the model has the same mental mechanism.
Why it matters
Automated evaluation can accidentally reward longer, smoother output instead of instruction following, correctness or usefulness.
Evidence status: established evaluation risk; exact strength is judge- and task-dependent
Evidence level: peer reviewed multi study
Signals
Self-test
Create semantically matched concise and padded answers, including one padded answer with a controlled instruction-following defect. Compare judge scores and rationales.
Record the exact model identifier, surface, system prompt, relevant settings and run date. A single surprising answer is an anecdote, not a bias measurement.
Mitigation
Applies to
llm as a judge research and summarization coding and automation agents
Dated model evidence
These findings describe the model or model set tested at the stated time. A newer release needs new evidence.
Published evaluation studies identify verbosity and other superficial-quality preferences as limitations of LLM judges.
Sources
