What is model validation in banking?
By Nick Jang, founder of MINERVA Studio ·
The short answer
Model validation is an independent review of whether a model is sound and fit for the purpose it is used for. A validator, separate from the people who built the model, examines its design and assumptions, the data behind it, how it was implemented and how it performs, then records findings and a recommendation for the person who approves its use. OSFI’s E-23 calls this model review and scales its depth to the model’s risk rating.
What validation is, and what it is not
Developers test their own models, and they should. Validation is different: it’s a second pair of hands with nothing riding on the answer. The point is effective challenge. Does the method fit the problem? Does the data hold it up? Does the code in production do what the documentation claims? Does the model still work on data it never saw? A good validator asks the awkward question the developer was hoping nobody would.
It ends in a written report with findings, each rated by severity, and a recommendation. Under E-23 the review can confirm or challenge the model’s risk rating, and the decision to use the model belongs to an approver, not to the validator.
What a validator reviews
| Area | The question | Typical evidence |
|---|---|---|
| Conceptual soundness | Is the method right for this problem, and are its assumptions reasonable? | Literature, alternatives considered, assumption tests, expert judgement documented. |
| Data | Is the data accurate, relevant, representative and traceable? | Data quality checks, lineage, treatment of missing values, proxy and synthetic data controls, bias checks. |
| Implementation | Does the production code match the documented model? | Code review, replication of outputs, change control. |
| Performance | Does it predict or classify well, in and out of sample? | Discrimination, calibration, stability and backtesting results against thresholds. |
| Use and limits | Is it used as intended, and are its limits known to its users? | Approved uses, limitations, overlays and exceptions on record. |
| Monitoring plan | Will a problem be caught after go-live? | Metrics, thresholds, frequency and escalation. |
Common validation techniques
Which tests apply depends on the model. These are the ones that come up most in banking:
- Backtesting: comparing predictions with what actually happened, such as default rates against predicted probabilities, or value-at-risk breaches against the expected count (the Basel traffic-light test).
- Discriminatory power: how well a model ranks good from bad, measured by AUC, the Gini coefficient or the Kolmogorov–Smirnov statistic.
- Calibration: whether predicted levels match observed levels, with the binomial test or Hosmer–Lemeshow.
- Stability: whether the population has shifted since development, with the population stability index (PSI; above 0.25 usually signals a major shift) and characteristic stability for each input.
- Benchmarking: building a challenger model or using an industry benchmark and comparing results.
- Sensitivity and stress testing: moving inputs and assumptions to see how outputs respond and where the model breaks.
- Out-of-time and out-of-sample testing: checking performance on data the model never saw.
- Fairness testing: comparing outcomes and error rates across groups, where the use calls for it.
- For generative AI: evaluation sets that measure accuracy and hallucination, robustness to prompt injection, toxicity and leakage of sensitive data.
A validator also reads what the numbers can’t show: an unstated assumption, a population the model was never built for, or a business that now uses the output in a way nobody approved.
How often a model is validated
Frequency follows the risk rating. Many institutions review their highest-rated models every year, medium ones every two years and low ones every three, with a full review on any material change. E-23 doesn’t fix the cycle; it asks for the scope, scale and intensity of review to match the model’s risk, and for monitoring in between. A material change to a model, its data or its use triggers a new review whatever the calendar says.
Independence, and why it is the point
E-23 says plainly that review must be independent of development. It can be done in-house or by an objective third party, but a model’s developer can’t validate it, and in most banks can’t approve it either. Independence is easy to write into a policy and easy to lose in practice, when a small team wears two hats or a deadline arrives. The separation has to be visible in the record: who built it, who reviewed it, who approved it, as three different names.
MINERVA records validation results from tests run in the institution’s own environment, through a Python SDK, against the approval they support, and refuses an approval by the model’s developer or one that lacks the evidence its rating calls for. See the E-23 platform.
Related questions
What is the difference between model validation and model monitoring?
Validation is a periodic, independent, in-depth review. Monitoring is the ongoing tracking of performance and inputs against thresholds between reviews, usually by the model owner.
Can a vendor validate its own model?
A vendor’s own testing is useful evidence, but it is not independent review for the institution. E-23 expects vendor models to be reviewed by the institution or an objective third party, to the extent the vendor allows.
What is a model validation report?
The written result of a review: scope, the tests performed, findings with their severity, the validator’s conclusion and a recommendation to the approver.
What does E-23 call model validation?
Model review. It describes an independent process that assesses a model’s conceptual soundness and performance, at an intensity set by its risk rating.
What is effective challenge?
Critical review by people who are independent, competent and have the standing to make the model owner act on what they find.
Sources
- OSFI — Guideline E-23, Model Risk Management (2027)
- Federal Reserve — SR 26-2, Revised Guidance on Model Risk Management
- Basel Committee — Supervisory framework for the use of backtesting (1996)
New answers are posted on MINERVA on LinkedIn. Follow along, or send the question you’d like answered next to minerva@minerva.studio.
A plain-language explanation, not legal advice. The regulator’s own text is the source; read it, and take advice on how it applies to your institution.