Unfairness enters through the data, the categories, the objective and the deployment — and average accuracy can hide all four.
Do now: Take one process and ask who is most likely to be misjudged by it, and what they could do about it.
If nobody can appeal, it is not a fair system. It is a fast one.Unfair outcomes rarely announce themselves. They arrive as an average that looks fine, applied to a group nobody measured separately, in a process nobody can appeal. Naming the mechanism is what lets an ordinary person spot it before it does harm.
Four doors unfairness comes through
Data. The examples the system learned from under-represent some people, or record historical decisions that were themselves unfair. A hiring tool learning from a decade of past hiring learns the past, including its exclusions, and reproduces them at speed.
Categories. The way the world is chopped up decides who is visible. Forms with no option for informal employment, name fields that break on how many people are actually named, language options that stop at the official languages — each of these makes a real population harder to serve before any model is involved.
Objective. What the system was optimised for may not be what matters to the person affected. Optimising for repayment rate is not the same as optimising for fair access to credit, and a system can be excellent at the first while being indefensible on the second.
Deployment. A tool that works acceptably in one setting is applied in another where the population, the language, the connectivity or the stakes are different. Nothing about the model changed; everything about the fairness did.
The model is only one of four doors, and usually not the widest.
Why the average hides the harm
Overall accuracy is a weighted average, and weights follow group size. A system can be ninety-four per cent accurate overall while being close to useless for a minority group, because that group barely moves the headline number.
So the question is never 'how accurate is it'. It is 'how accurate is it for whom'. Break results down by whatever axis matters in your context — language, region, gender, age, device, income, disability — as far as it is lawful and ethical to measure. Where you cannot measure directly, look for proxies and be honest that you are estimating.
Also ask what kind of error each group receives. Being wrongly rejected and being wrongly accepted are not equivalent harms, and they often fall on different people. A system that mostly errs toward rejection for one group is doing something quite specific to that group.
Appeal is the part that makes it fair
Every process that materially affects someone needs three things: a human who can review the decision, a route the affected person can actually reach, and a record that makes review possible. Missing any one of them turns a mistake into a permanent fact about someone's life.
'Reachable' is a real constraint, not a formality. An appeal that requires an email address, a smartphone, fluent written English and office-hours availability excludes exactly the people most likely to have been misjudged in the first place.
And appeals are data. A group appealing at three times the rate of others is telling you where your system is failing, well before any audit would find it. Organisations that log appeals by outcome learn faster than organisations that merely permit them.
An appeal route nobody can reach is a fairness claim, not a fairness control.