What is AI bias?
By Nick Jang, founder of MINERVA Studio ·
The short answer
AI bias is when an AI system produces results that are systematically worse or unfair for certain people or groups, for example lower approval rates, higher error rates or worse service for one group than another with no good reason. It usually comes from the data a model learned from, the way the problem was framed, or how people use the output. NIST sorts it into three kinds: systemic, statistical and computational, and human. It can be measured and reduced, but rarely eliminated, so the job is to test for it, decide what level is acceptable, and keep watching.
Where it comes from
NIST’s Special Publication 1270 (March 2022) makes the point that bias isn’t only a data problem. It describes three categories:
| Category | What it is | Example |
|---|---|---|
| Systemic | Bias in the institutions and practices that produced the data and the decisions. | Historical lending records that reflect past discrimination. |
| Statistical and computational | Errors from samples that don’t represent the population, or from modelling choices. | A model trained mostly on urban customers that performs worse in rural areas. |
| Human | Systematic errors in how people build, interpret and use AI. | Staff accepting a model’s score without question, or overriding it more often for some groups. |
A common route is a proxy: a variable that looks neutral but tracks a protected characteristic. Postal code can stand in for ethnicity; years of continuous employment can stand in for gender or disability. Removing the protected characteristic from the data doesn’t remove the bias if a proxy is still there.
Well-documented examples
- Hiring. Reuters reported in 2018 that Amazon dropped an experimental recruiting tool after finding it penalised CVs containing the word “women’s”, as in “women’s chess club captain”. It had learned from ten years of mostly male hires.
- Health care. A 2019 study in Science by Obermeyer and colleagues found that a widely used algorithm for directing extra care used past health spending as a stand-in for health need. Because less had been spent on Black patients with the same needs, they were scored as healthier than they were. Fixing the target would have raised the share of Black patients flagged for extra help from 17.7% to 46.5%.
- Face analysis. The 2018 Gender Shades study found commercial gender classification systems misclassified darker-skinned women up to 34.7% of the time, against at most 0.8% for lighter-skinned men.
- Credit. After complaints in 2019 that the Apple Card gave women lower limits, New York’s Department of Financial Services investigated and in 2021 found no unlawful discrimination, but criticised how little customers were told about how limits were set. Perceived bias with no explanation is its own problem.
In each case nobody set out to discriminate. The model did exactly what it was trained to do.
How bias is measured
Testing compares outcomes or errors across groups. Common measures:
| Measure | Asks |
|---|---|
| Approval or selection rate | Does each group get the positive outcome at a similar rate? |
| Error rates | Are false declines and false approvals similar across groups? |
| Calibration | Does a score of 0.7 mean the same risk whatever the group? |
| Adverse impact ratio | Is one group’s selection rate below a set fraction, often four-fifths, of the highest group’s? |
Here’s the uncomfortable part: researchers have shown that, except in special cases, several of these can’t all be satisfied at once. Equalising approval rates can unbalance error rates, and vice versa. So reducing bias involves a choice about which kind of fairness matters for this decision, and that choice should be made and recorded by people with the authority to make it, not left to whoever tuned the model.
Reducing it
- Frame the problem carefully. The health care case was a framing failure: spending was the wrong target.
- Check the data for representation, missing groups, historical decisions and proxies.
- Test before launch on each relevant group, and for proxies of protected characteristics.
- Mitigate by rebalancing data, adjusting the model or setting group-aware thresholds where the law allows.
- Keep a person in the right place: review of declines, an appeal route, and monitoring of overrides.
- Monitor after launch. Populations change, and a fair model can drift into an unfair one.
- Explain decisions where people are affected; explainable AI methods help find the drivers.
In a Canadian bank or insurer
Human rights law prohibits discrimination on protected grounds whether a person or a model makes the decision. OSFI’s E-23 expects bias to be managed in the data and in AI models specifically, and Quebec’s AMF guideline on AI expects fair treatment of clients. In practice that means bias testing is part of a model’s validation, its results and the fairness choice are written down, and an open bias finding blocks approval until someone deals with it.
MINERVA records fairness tests as validation results on the model, opens a finding when one fails, and won’t let the model be approved while a high-severity finding stays open. See AI governance software.
Related questions
What is a simple definition of AI bias?
When an AI system gives systematically worse or unfair results for some people or groups than for others, without a legitimate reason.
Can AI bias be eliminated?
Rarely completely. It can be measured and reduced, but several definitions of fairness conflict with each other, and data reflects the past. The aim is to test, choose an acceptable level openly and monitor it.
What causes AI bias?
Biased or unrepresentative training data, proxies for protected characteristics, the way the problem is framed, modelling choices, and how people use the output. NIST groups these as systemic, statistical and computational, and human biases.
What is an example of AI bias?
A 2019 study found a US health care algorithm under-identified Black patients for extra care because it used past spending as a stand-in for need. Amazon also dropped a recruiting tool that penalised CVs mentioning women’s activities.
Is AI bias illegal in Canada?
Discrimination on protected grounds is prohibited under human rights law whether a person or a system makes the decision. Financial regulators such as OSFI and the AMF also expect institutions to manage bias in their models.
Sources
- NIST — SP 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (March 2022)
- Obermeyer et al. — Dissecting racial bias in an algorithm used to manage the health of populations, Science (2019)
- Buolamwini and Gebru — Gender Shades (2018)
- New York DFS — Report on Apple Card investigation (March 2021)
New answers are posted on MINERVA on LinkedIn. Follow along, or send the question you’d like answered next to minerva@minerva.studio.
A plain-language explanation, not legal advice. The regulator’s own text is the source; read it, and take advice on how it applies to your institution.