04 / How It Goes Wrong
Bias
Systematic unfairness in AI output, inherited from the training data or introduced by design, that treats some people or groups worse than others.
How you’ll hear it
“The screening tool keeps favouring one kind of candidate.”
“Bias. We test the outcomes across groups before we use it.”
Avoid
Assuming a model is neutral because it is a machine.
The longer version
Models learn patterns from data that reflects existing inequalities, and product choices can add more. Bias matters most where AI affects access to jobs, credit, services or care. It is found by testing outcomes across groups, and it is a legal concern as well as an ethical one.
In depth
Bias in AI is systematic unfairness: results that are consistently worse for some groups of people than others. It usually comes from training data that reflects existing inequalities, and it can be reinforced by how a system is designed and used.
Language models can reproduce stereotypes in the text they write, rank candidates differently based on names, or perform worse in some languages and dialects. Predictive systems can learn patterns from historical decisions that were themselves unfair.
Bias is found by measuring outcomes across groups, not by inspecting the model. That means deciding in advance which groups matter for your use case, testing the system’s results for each, and repeating the test after changes.
Where AI affects access to jobs, credit, services or care, bias is a legal risk as well as an ethical one, and high-risk uses under the EU AI Act carry explicit duties on data quality.
What to do
- Decide which groups your system must treat fairly, and measure outcomes for each before launch.
- Keep a person accountable for decisions that affect people’s access to jobs, money or services.
- Re-test for bias whenever the model or the data changes.