What is model monitoring?
By Nick Jang, founder of MINERVA Studio ·
The short answer
Model monitoring is the ongoing check that a model in production still does what it was approved to do. It tracks the model’s performance, the stability of its inputs and scores, data quality, how it’s being used and how often people override it, against thresholds agreed before the model went live. When a threshold is breached, monitoring triggers a decision: watch, fix, restrict or retire. OSFI’s E-23 makes defined monitoring standards an expectation for every model from 1 May 2027.
Why validation isn’t enough
Validation checks a model at a point in time. Monitoring is everything after. A model that passed validation in March can be quietly wrong by October if its customers, its data feed or the economy shifted. That’s model drift, and monitoring is how you see it.
Monitoring also catches things that have nothing to do with drift: a broken data pipeline filling a field with zeros, a team using the model for a product it wasn’t approved for, or staff overriding it so often that it’s no longer really making the decision.
What to monitor
| Area | Typical metrics |
|---|---|
| Performance | Accuracy, Gini or AUC, precision and recall, error rates, once outcomes are known. |
| Calibration | Predicted versus actual rates, by score band. |
| Stability | Population stability index on the score; characteristic stability on key inputs. |
| Data quality | Missing values, out-of-range values, a field that stopped arriving, schema changes. |
| Usage | Volumes, the products and segments it’s used for, whether that matches its approved uses. |
| Overrides | How often people overrule the model, and whether overrides turn out right. |
| Fairness | Outcome differences across groups, where the use calls for it. |
| Operational | Latency, failures, the version actually running versus the version approved. |
For generative AI, add: the share of answers that fail an automated or human quality check, refusals, flagged harmful content, prompt injection attempts and the model version behind any third-party API.
Setting thresholds
A threshold turns a number into a decision. Agree them before go-live, in the monitoring plan, so nobody negotiates them after a bad month. Most teams use a traffic light:
- green: within tolerance, keep going;
- amber: outside tolerance, investigate and report to the model owner;
- red: a breach, escalate to the model risk function with a decision due by a date.
Tighter thresholds and more frequent checks go with higher risk ratings. A fraud model scoring millions of transactions may be watched daily; a quarterly capital model, quarterly.
What OSFI E-23 expects
Principle 3.6 of OSFI’s E-23 says institutions should have defined standards for model monitoring and decommissioning. Monitoring is there to keep models fit for purpose and to detect performance issues or breaches. The standards should set frequency, scope and evaluation criteria by risk rating and model type, define thresholds for breaches and for what counts as a material model change, and track changes in model performance, model usage and input data. The guideline flags the higher potential for drift in AI and machine-learning models.
The model inventory then carries each model’s monitoring status, with exceptions. That’s the piece examiners look for: not that monitoring exists, but that a breach was seen, owned and decided.
MINERVA records thresholds and results against each model, raises a breach to its owner and keeps the decision with the evidence. See OSFI E-23 software.
Related questions
What is the difference between model validation and model monitoring?
Validation is an independent check of a model at a point in time, usually before use and then periodically. Monitoring is the continuous tracking of the model in production between validations.
What metrics are used in model monitoring?
Performance (accuracy, Gini or AUC), calibration, stability (PSI and CSI), data quality, usage, overrides, fairness where relevant, and operational measures such as failures and versions.
How often should a model be monitored?
In proportion to its risk rating and how quickly its environment changes: daily or monthly for high-volume or high-risk models, quarterly or less often for lower-risk ones.
What happens when a monitoring threshold is breached?
The breach is escalated to the model owner and the model risk function, investigated, and resolved by a recorded decision: accept, fix, restrict use, or retire the model.
Does OSFI require model monitoring?
Yes. E-23 Principle 3.6 expects defined standards for model monitoring and decommissioning, scaled to each model’s risk rating, from 1 May 2027.
Sources
- OSFI — Guideline E-23, Model Risk Management (2027)
- Federal Reserve — SR 26-2, Revised Guidance on Model Risk Management
- NIST — AI Risk Management Framework: Generative AI Profile (NIST AI 600-1)
New answers are posted on MINERVA on LinkedIn. Follow along, or send the question you’d like answered next to minerva@minerva.studio.
A plain-language explanation, not legal advice. The regulator’s own text is the source; read it, and take advice on how it applies to your institution.