08 / Trust & Governance
Explainability
Being able to say why an AI system produced a particular result, in terms the affected person or a reviewer can understand.
How you’ll hear it
“Why was this claim rejected?”
“If we cannot answer that, we cannot use the system for this decision.”
Avoid
Accepting “the model decided” as the explanation for a decision about a person.
The longer version
Explainability matters most when AI influences decisions about people. Language models cannot fully explain their inner workings, so practical explainability comes from design: grounded sources, recorded reasoning, clear rules and a person who can account for the outcome.
In depth
Explainability is the ability to say why an AI system produced a particular result, in terms the affected person, a reviewer or a regulator can understand.
It matters most when AI influences decisions about people: hiring, lending, benefits, healthcare. People have a reasonable expectation of understanding, and sometimes a legal right to meaningful information about decisions that affect them.
Large language models cannot fully explain their inner workings. Asking the model to explain itself produces a plausible story, not a guaranteed account of what happened. Practical explainability therefore comes from how the system is designed: answers grounded in cited sources, decision rules written down, reasoning and inputs recorded, and a person accountable for the outcome.
If an outcome cannot be explained to the person it affects, the system is not ready for that decision.
What to do
- For any decision about a person, write down who can explain it and how.
- Prefer designs that cite sources and record inputs over ones that only produce a verdict.
- Test the explanation with someone outside the team. If they cannot follow it, neither will the person affected.