What is explainable AI?
By Nick Jang, founder of MINERVA Studio ·
The short answer
Explainable AI (XAI) is the set of methods that show why a model produced a particular result, in terms a person can check. Global explanations describe how the model behaves overall (which inputs matter most); local explanations describe one decision (why this applicant was declined). The two most used tools are SHAP, which splits a prediction into each input’s contribution, and LIME, which fits a simple model around one prediction. Banks need explanations for validation, for customers and for regulators.
Why explanations matter
A logistic regression scorecard explains itself: each variable has a weight you can read. A gradient-boosted model with 400 features, or a neural network, doesn’t. It may be more accurate, but nobody can say from looking at it why it said what it said.
That matters to four people. The validator needs to know whether the model is right for the right reasons, not leaning on a proxy for age or postcode. The customer who was declined is often owed a reason. The business owner needs to trust it enough to use it. And the examiner wants to see that someone understood it before it was approved.
Global and local explanations
| Kind | Answers | Typical methods |
|---|---|---|
| Global | How does the model behave overall? Which inputs matter most? | Feature importance, partial dependence plots, mean SHAP values, surrogate models. |
| Local | Why did the model produce this result for this case? | SHAP values for one prediction, LIME, counterfactuals ("approved if income were $8,000 higher"). |
SHAP and LIME, briefly
SHAP (SHapley Additive exPlanations, Lundberg and Lee, 2017) borrows Shapley values from game theory. It treats each input as a player and asks how much each one moved this prediction away from the average. The pieces add up exactly to the prediction, which is why validators like it. It can be slow on big models, and correlated inputs can split credit in odd ways.
LIME (Local Interpretable Model-agnostic Explanations, Ribeiro, Singh and Guestrin, 2016) nudges the input many times, watches how the prediction changes, and fits a small simple model to those nearby points. It works on almost anything, text and images included, but two runs can give slightly different answers.
Neither proves a model is fair or right. They show what the model is using. A validator still has to judge whether that makes sense.
NIST’s four principles
NIST’s 2021 report on explainable AI (NISTIR 8312) set four principles that hold up well:
- Explanation: the system gives a reason for its output.
- Meaningful: the reason makes sense to the person it’s for, which differs for a data scientist and a customer.
- Explanation accuracy: the reason truly reflects how the system reached its output.
- Knowledge limits: the system only operates where it was designed to, and says when it isn’t confident.
What the rules expect
OSFI’s E-23 builds explainability into each stage. Development should set explainability requirements, which can vary with the model’s purpose and level of autonomy. The review should evaluate how explainable the model is for its intended use, AI and machine-learning models included, and confirm its outputs are appropriately explainable. Deployment should make sure explanations reach the people who need them. A model nobody can explain is hard to rate, hard to validate and hard to approve.
In Quebec, Law 25 requires a business that makes a decision about a person based exclusively on automated processing to tell them, and on request to explain the personal information used and the main factors and parameters that led to the decision. In the United States, the CFPB has said lenders using complex algorithms must still give specific, accurate reasons when they deny credit.
MINERVA records each model’s explainability approach, the reviewer’s findings and the approver’s decision on the same record. See model validation and AI model risk management.
Related questions
What is explainable AI in simple terms?
Methods that show why an AI model produced a particular result, in a way a person can understand and check.
What is the difference between SHAP and LIME?
SHAP splits one prediction into each input’s exact contribution using Shapley values from game theory. LIME fits a small simple model around one prediction by perturbing the input. SHAP is more consistent; LIME is quicker to apply to any model.
What are global and local explanations?
A global explanation describes how a model behaves overall, such as which inputs matter most. A local explanation describes why it produced one specific result.
Is explainable AI required by law in Canada?
There is no general AI law, but Quebec’s Law 25 gives people the right to be told about decisions made exclusively by automated processing and, on request, the main factors behind them. OSFI’s E-23 expects banks to understand and review their AI models.
Can large language models be explained?
Only partly. Methods exist for attributing an answer to parts of the prompt or source documents, but full explanations of how an LLM reasons are not yet reliable, so controls lean more on testing, grounding and monitoring.
Sources
- NIST — Four Principles of Explainable Artificial Intelligence (NISTIR 8312, 2021)
- Lundberg and Lee — A Unified Approach to Interpreting Model Predictions (NeurIPS 2017)
- Ribeiro, Singh and Guestrin — “Why Should I Trust You?” Explaining the Predictions of Any Classifier (2016)
- Québec — Act respecting the protection of personal information in the private sector (Law 25), s. 12.1
- CFPB — Circular 2022-03, adverse action notices and complex algorithms
- OSFI — Guideline E-23, Model Risk Management (2027)
New answers are posted on MINERVA on LinkedIn. Follow along, or send the question you’d like answered next to minerva@minerva.studio.
A plain-language explanation, not legal advice. The regulator’s own text is the source; read it, and take advice on how it applies to your institution.