AINUA AI Academy
F4.7 · Foundations

Beginner · responsible practice · 19 min core lesson · 4 min quick note

High-stakes decisions need qualified humans

Match review strength to possible harm.

Ethics, safety and citizenshipHuman agencyAI understanding
Nia and Kio introducing High-stakes decisions need qualified humans
Cold open

The most valuable thing an AI system can do, in some situations, is decline — and the second most valuable thing is for you to notice that it did not.

By the end, you can

sort a real task into green, amber or red, and design an escalation route with a named qualified person for anything in the red tier.

Quick note · 4 min

Green is routine drafting. Amber involves personal data, money, employment or reputation. Red involves health, law, credit, safety, children or anything irreversible.

Do now: Sort three of your real tasks into the three tiers and name who must review each.

A disclaimer is not oversight. Oversight has a name attached.
Why it matters

Most AI harm does not come from dramatic misuse. It comes from an ordinary person using an ordinary tool for something that quietly needed a professional, because nothing in the interface suggested otherwise.

01

Three tiers, and where the line actually falls

Green: low-stakes drafting, formatting, brainstorming, summarising material you already understand. Routine checks are enough. Most daily use lives here and should.

Amber: anything touching personal data, meaningful money, employment, or someone's reputation. Customer communication, business advice, pricing, hiring materials, public content. A named human reviews and approves before it leaves.

Red: diagnosis and treatment, legal judgement, lending and credit decisions, child welfare, public safety, and anything that cannot be undone. These need a qualified person with actual authority, a documented process, and a route to challenge the outcome. AI may assist with preparation; it does not make the decision.

The line is not about how complex the task feels. It is about what happens to a person if it goes wrong, and whether that can be reversed.

When a task sits between two tiers, treat it as the higher one. The cost of over-caution is a delay; the cost of under-caution is a person.
02

Why the interface will not warn you

The same box that helps you write an email will also produce something that reads like medical advice, legal advice or a credit assessment. The tone does not change. Many systems add a disclaimer, and disclaimers are almost entirely ineffective at the moment they matter, because the answer above them is specific, fluent and apparently addressed to your situation.

So the classification has to happen before you ask, not after you read. Decide the tier from the task, and set the rule while you are calm about it, rather than while you are looking at an answer you would like to be able to use.

Be especially careful where a task is red for the other person and green for you. Drafting a summary of someone's symptoms feels like writing. For them, it may be the thing that decides whether they seek care.

03

Escalation that actually functions

An escalation route needs four things written down: what triggers it, who specifically it goes to, what they receive, and what happens while everyone waits. Routes without the fourth are the ones that fail — people improvise when there is no defined holding action, and improvising is exactly what escalation was meant to prevent.

Name a role and a person, not a department. 'Refer to legal' is not a route. 'Send to the named compliance officer, with the file and the specific question, and take no action until they respond' is.

And give the person doing the work permission to escalate. If escalating is treated as an admission of incompetence, people will stop, and the failures will simply happen further downstream where they are harder to see.

Escalation fails on the holding action. Decide what happens while everyone waits.