How do you validate an AI model?
By Nick Jang, founder of MINERVA Studio ·
The short answer
AI model validation is the independent check that a machine learning or generative AI model is sound for its intended use: that it was built on suitable data and a defensible method, performs as expected, fails in ways that are understood, and stays within limits once it's live. The pillars are the same as for any model validation (conceptual soundness, outcomes testing and ongoing monitoring), but AI adds work: explainability, bias across groups, robustness to unusual or hostile inputs, drift, and for generative models, output quality that can't be checked against a single right answer. OSFI's E-23, the US SR 26-2 and the UK's SS1/23 all treat AI models as models that need it.
What changes with AI
| Area | Traditional model | AI model |
|---|---|---|
| Conceptual soundness | Review the theory and the equations. | Review the method choice, features and training process; explain the drivers with tools such as SHAP. |
| Data | Representative, clean, complete. | All that, plus labelling quality, leakage between training and test sets, proxies for protected characteristics, and synthetic data. |
| Performance | Accuracy and stability on a holdout sample. | Also across segments, under shifted data, and for generative models on task-specific evaluation sets and human review. |
| Fairness | Sometimes. | Routinely, by group, with the chosen fairness measure recorded. |
| Robustness | Sensitivity to inputs. | Adversarial inputs, prompt injection, jailbreaks and edge cases; see red teaming. |
| Reproducibility | Same inputs, same output. | Seeds, versions and, for generative models, variation across repeated runs. |
| Monitoring | Periodic performance checks. | Drift in inputs and outputs, more often, with retraining triggers. |
Validating machine learning models
- Benchmark the model against a simpler one. If a logistic regression nearly matches a gradient-boosted model, the complexity may not be worth its opacity.
- Explain the global drivers and a sample of individual decisions, and check they make business sense. See explainable AI.
- Test by segment, not only overall: new customers, thin files, regions, product lines.
- Stress the inputs: missing values, outliers, shifted distributions.
- Test for bias where people are affected; see AI bias.
- Check overfitting: performance gap between training, validation and out-of-time samples.
Validating generative AI
A language model rarely has one correct answer, so validation shifts from a single accuracy number to evidence across many cases:
- A task-specific evaluation set: real questions with expected answers or graded criteria, built with the business.
- Groundedness: for systems that answer from documents, whether each answer is supported by the source.
- Confabulation rate: how often it invents facts, fees, policies or citations.
- Safety and policy adherence: refusals where it should refuse, and no advice it mustn't give.
- Security: prompt injection and data leakage attempts.
- Consistency: the same question several times, and small rewordings.
- Human review of a sample by people who know the domain.
NIST's generative AI profile lists the risks the tests should cover.
Vendor and foundation models
Most banks can't see inside a vendor's model or a foundation model, but that doesn't remove the duty. The UK's SS1/23 puts it plainly: firms should satisfy themselves that vendor models have been validated to the same standard as their own, check the vendor's data and assumptions, and monitor performance using their own outcomes. In practice that means asking for the vendor's validation evidence, testing the model on your own data and use cases, and watching what changes when the vendor updates it. See third-party AI.
Independence and the record
E-23 expects review to be independent of development, at an intensity set by the model's risk rating. The validator's findings go to the approver, who decides with them in hand. MINERVA records validation results against the model and the approval they support, opens a finding when a test fails, and won't let the model be approved by its own developer or while a high-severity finding is open. See OSFI E-23 software.
Related questions
What is the difference between AI model validation and traditional model validation?
The pillars are the same, but AI validation adds explainability work, bias testing across groups, robustness to adversarial inputs, closer drift monitoring and, for generative models, evaluation sets and human review instead of a single accuracy measure.
How do you validate a large language model?
With a task-specific evaluation set, checks for groundedness and confabulation, safety and policy adherence tests, prompt injection and data leakage attempts, consistency across repeated and reworded questions, and expert human review of samples.
Do vendor AI models need validation?
Yes. Regulators such as OSFI and the PRA expect institutions to be satisfied that vendor models are validated to the same standard as their own and to test and monitor them on their own data and use.
Who should validate AI models?
People independent of the developers, with both statistical and AI skills and enough business knowledge to judge whether outputs make sense. Some institutions use objective third parties for this.
Sources
- OSFI — Guideline E-23, Model Risk Management (2027)
- Bank of England PRA — SS1/23, Model risk management principles for banks
- Federal Reserve — SR 26-2, Revised Guidance on Model Risk Management
- NIST — AI 600-1, Generative AI Profile
New answers are posted on MINERVA on LinkedIn. Follow along, or send the question you’d like answered next to minerva@minerva.studio.
A plain-language explanation, not legal advice. The regulator’s own text is the source; read it, and take advice on how it applies to your institution.