Numbers & Estimates
Does the First Number Pull Your Estimate?
Can an arbitrary starting price shift a later estimate even when the product information stays the same?
- Metric
- Higher-anchor minus lower-anchor estimate
- Output
- EUR
AI Bias Benchmark · specification v1
Test the change, not the label.
This benchmark asks a simple question: when one controlled part of a prompt changes, does an AI model's judgment move with it? The same paired scenarios also exist in the human Experiments Lab.
Current status
Specification published. Results pending.
We do not publish model scores until the runs are reproducible and include model version, settings, date and enough repeated samples for the task. A blank leaderboard is less exciting than invented precision, but considerably more useful.
What is measured
Six paired decision tests.
Each track changes one decision condition while keeping the rest of the task as stable as possible. The benchmark reports the direction and size of each response shift. It does not collapse unlike effects into one universal “bias score”.
Framing
Does Survival vs Mortality Wording Change the Feeling?
Can equivalent outcome statistics feel different when they are described as survival rather than mortality?
- Metric
- Survival-frame minus mortality-frame reassurance
- Output
- rating-points
Choice Architecture
Can a Third Option Change Which Plan You Prefer?
Can an inferior comparison option change preference between two original alternatives?
- Metric
- Change in Pro choice share after decoy is added
- Output
- share
Projects & Money
Would You Continue If the Past Investment Disappeared?
Does visible past investment make continued funding feel more attractive even when future costs and benefits are unchanged?
- Metric
- Change in Continue share when sunk investment is visible
- Output
- share
Learning & Review
Does the Outcome Change How Good the Decision Looks?
Can knowing the outcome change how people judge the quality of the earlier decision process?
- Metric
- Good-outcome minus bad-outcome decision-quality rating
- Output
- rating-points
Forecasting
What Happens When You See the Reference Class?
Does adding a reference class change a delivery estimate built from an ideal step-by-step plan?
- Metric
- Outside-view minus inside-view estimate
- Output
- working-days
Reproducible protocol
Run each condition as a fresh case.
- Use the published system prompt and user prompt without adding the bias name.
- Run paired conditions in separate fresh contexts.
- Record provider, model name, model version, temperature, top-p and run date.
- For stochastic choice tasks, use at least 20 samples per condition before interpreting choice-share differences.
- Keep the per-experiment results visible. Do not turn six different measurements into one magic number.
Machine-readable
Designed for agents as well as people.
Benchmark specification
Test definitions, metrics, protocol and response contracts.
Prompt pack
Twelve condition cases ready for a provider-specific runner.
Result schema
A versioned format for model metadata and raw benchmark responses.
Repository users can score a completed result file with npm run benchmark:ai:score -- --input results.ndjson.
Interpretation
Sensitivity is a measurement, not a diagnosis.
A response shift can be useful evidence that a model is sensitive to the manipulated condition. It does not show that the model “thinks like a human”, and a zero shift does not prove the model is generally free from that bias. Prompt wording, model version and decoding settings can all matter.