AI-era research protocol
Do not invent a bias. Design a test.
A useful new label should survive comparison with established concepts and produce a prediction that can fail. This protocol turns an observation into a small reproducible human-AI experiment.
Research rules
Make the claim smaller before making it stronger.
Core design
Human → AI → Human is the unit of study.
Record what the person believed before AI input, what the model produced, and what changed afterward. That separation helps distinguish a model failure from a human reliance effect or a feedback loop between both.
Starter experiments
Four tests you can reproduce or extend.
ai-anchoring-loop
AI Advice Order Test
Does seeing an AI estimate before making your own estimate pull the final judgment toward the model?
Conditions:
- Independent estimate first, then AI advice
- AI advice first, then human estimate
Primary measure: Absolute shift toward the AI estimate
Secondary measures: confidence change, decision time, distance from a reference-class estimate.
Minimum report: Report the model/version, task, anchor value, sample size, mean shift by condition and uncertainty.
evaluation-history-contamination
Blind vs History-Aware AI Judging
Does an earlier evaluation score change a later AI judge's score when the candidate answer and rubric stay fixed?
Conditions:
- Blind evaluation with no history
- Prior-review framing without a score
- Prior-review framing with an experimentally assigned score and verdict
Primary measure: Signed movement of the history-aware score toward the displayed previous score relative to the blind score
Secondary measures: history-anchor versus history-framing difference, absolute error against constructed gold score, correction-harm rate, verdict flip rate, repeat dispersion.
Minimum report: Use fresh contexts, keep model/version/settings fixed within a run, report every task-condition score, and publish model-level results before pooling.
sycophancy-reinforcement-loop
AI Agreement Pressure Test
Does signalling a preferred answer change model agreement and then increase human confidence in that preferred answer?
Conditions:
- Neutral question
- Question containing a preferred conclusion
- Preferred conclusion plus explicit request for counterevidence
Primary measure: Change in model agreement rate and user confidence
Secondary measures: factual accuracy, counterargument quality, answer revision rate.
Minimum report: Keep the underlying task identical across conditions and report both model behaviour and human confidence separately.
source-memory-blur
Human-AI Source Memory Test
Do mixed human-AI workflows make it harder to remember who produced an idea or sentence?
Conditions:
- Human-only creation
- AI-only suggestion
- Mixed human-AI creation
Primary measure: Correct source-attribution rate after a delay
Secondary measures: confidence in attribution, verbatim recognition, idea ownership judgment.
Minimum report: Predefine the delay, keep contribution labels hidden during the test, and distinguish idea-source from wording-source errors.
cognitive-offloading-debt
AI Offloading & Retention Test
Can AI improve immediate completion while weakening unaided retention or transfer?
Conditions:
- Unaided work
- AI-assisted work
- AI-assisted work plus retrieval/explanation step
Primary measure: Delayed unaided retention or transfer performance
Secondary measures: immediate task quality, time on task, self-rated effort, confidence.
Minimum report: Do not infer long-term learning from immediate task quality. Include a delayed or transfer measure.
structured-output-authority
Structured Output Authority Test
Does formatting the same AI answer as a table or scorecard increase trust or reduce verification compared with equivalent prose?
Conditions:
- Equivalent answer in plain prose
- Equivalent answer in a structured table or scorecard
- Structured answer plus a note that formatting does not increase evidence quality
Primary measure: Trust rating and verification rate
Secondary measures: error acceptance, perceived objectivity, decision confidence, time to first verification action.
Minimum report: Keep claims, wording, sources, model identity and planted error rate constant. Report readability separately so a fluency advantage is not silently treated as authority.
dialogue-momentum
Dialogue Momentum Test
After a multi-turn AI conversation converges on a plan, are people less likely to reopen an early assumption when new counterevidence appears?
Conditions:
- Continuous conversation with prior convergence
- Fresh context containing the same plan and new counterevidence
- Continuous conversation plus an explicit instruction to reopen the original assumption
Primary measure: Rate of revising the challenged assumption
Secondary measures: final decision quality, confidence change, amount of counterevidence discussed, time to revision.
Minimum report: Keep the plan, counterevidence and decision task equivalent across conditions. Report whether differences come from human revision, model response, or both.
citation-density-halo
Citation Density Halo Test
Do repeated citation markers make an AI answer look better supported when the number and quality of unique sources do not change?
Conditions:
- Two unique sources cited once each
- The same two sources repeated through many citation markers
- Repeated markers plus explicit disclosure that only two unique sources are used
Primary measure: Perceived evidence strength
Secondary measures: trust, verification intent, estimate of unique source count, error acceptance.
Minimum report: Hold answer text, source quality and unique source count constant. Record whether participants correctly understand source independence rather than assuming every marker represents new evidence.
Starting evidence
Use research as a constraint, not decoration.
- How was my performance? Exploring the role of anchoring bias in AI-assisted decision making (2025)
- Anchoring bias in large language models: an experimental study (2025)
- Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence (2026)
- Mediocrity is the key for LLM as a Judge Anchor Selection (2026)
- Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy (2026)
- Too Nice to Tell the Truth: Quantifying Agreeableness-Driven Sycophancy in Role-Playing Language Models (2026)
- Large Language Models are overconfident and amplify human bias (2025)
- The AI Memory Gap: Users Misremember What They Created With AI or Without (2025)
- ChatGPT as a cognitive crutch: Evidence from a randomized controlled trial on knowledge retention (2025)
These sources support parts of the research space. They do not validate every working label in the AI-era Bias Lab.