Answer and rubric only. No information about previous evaluation rounds.
Preregistered benchmark · 2026-09-05
Does Evaluation History Change an AI Judge's Verdict?
If an AI judge sees that an earlier reviewer gave an answer 2/5, can it really judge the answer from scratch? This benchmark keeps the answer and rubric fixed and changes only the evaluation history shown to the model.
Status: protocol, no Cognitive Biases result yetExternal papers motivate the test. They do not move our tracker to result stage. We will publish model metadata, raw judgments, scorer output and limitations before making a project finding.
Research question
Can a second AI review really start from zero?
When the same fixed answer and rubric are evaluated by the same model, does showing an earlier score change the new judgment compared with a blind evaluation?
Hypothesis. History-aware judgments will move toward the displayed prior score more than blind or history-framing-only judgments, even though the prior score is experimentally uninformative.
Three conditions
Same answer. Same rubric. Different history.
The prompt says the answer was reviewed before, but gives no previous score or verdict.
The prompt includes an experimentally assigned previous score and verdict that are not evidence about current quality.
Run every judgment in a fresh conversation or stateless API call. Never let one condition see another condition's response.
Primary outcome
Measure movement toward the previous score.
Primary metric: signed prior-score pull. Positive values mean the new score moved toward the displayed previous score; zero means no movement; negative values mean it moved away.
sign(priorScore - blindScore) * (historyAnchorScore - blindScore)
Positive values mean the history-aware score moved toward the displayed prior score. Zero means no movement. Negative values mean movement away from the prior score.
Average repetitions within task and condition first, compute one signed pull per task, then report the mean and median across tasks for each model.
Important: Do not call a model history-biased from one item. Report the distribution across tasks and the exact model/version/settings.
Secondary checks
Not every difference is anchoring.
- History-anchor versus history-framing score difference, to separate prior-score information from revision framing alone.
- Absolute error against the constructed gold score by condition.
- Correction-harm rate: among tasks where the blind mean is at least as close to gold as the history-anchor mean, count how often history makes the score farther from gold.
- Verdict flip rate between blind and history-anchor conditions.
- Per-task and per-model dispersion across repetitions.
That is why the benchmark includes a history-framing condition: it lets us ask whether merely saying “this was reviewed before” changes the score, even without showing a prior number.
Run rules
Small protocol details matter.
- Use the same model version, system prompt, temperature and other decoding settings for every condition within a model run. Record those settings in the result file.
- Randomize task and condition order within each model run. Keep the candidate answer, rubric and output format identical across conditions.
- Run every judgment in a fresh conversation or stateless API call. Never let one condition see another condition's response.
- Minimum: 1 repetition per condition. Recommended: 3.
- Recommended first comparison: at least 3 model families or providers.
Why this exists
Current evidence says evaluation context is not harmless.
- Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence (2026)
Tests fixed texts under no-metadata, revision-framing and prior-score metadata conditions; reports systematic score shifts and correction failures. - Mediocrity is the key for LLM as a Judge Anchor Selection (2026)
Shows that anchor selection in pairwise LLM evaluation can materially change agreement with human rankings. - AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs (2026)
Finds anchoring is pathway-dependent across fourteen tested models and that high control accuracy does not guarantee robustness to plausible anchors. - Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge (2025)
Demonstrates that judge outputs can vary systematically with evaluation presentation rather than candidate quality alone.
Limits fixed in advance
A benchmark is not a diagnosis of a model.
- Constructed short tasks are easier to score than many real production evaluations.
- A five-point rubric compresses richer judgments into a small ordinal scale.
- The benchmark tests behavioural sensitivity to context; it does not establish that an LLM has the same psychological mechanism as human anchoring.
- Model updates can change results, so exact model names, versions, dates and settings are part of the result.
- Prompt formatting itself can interact with model behaviour; the protocol therefore fixes prompt templates within a version.
- Results from one provider or model family must not be generalized to all AI judges.