What is model drift?
By Nick Jang, founder of MINERVA Studio ·
The short answer
Model drift is a model’s performance decaying over time because the world has moved away from the data it learned from. It comes in two main kinds: data drift, where the inputs change (a new customer mix, a new product), and concept drift, where the relationship between inputs and outcomes changes (the same income now means more risk because rates went up). You catch it by comparing today’s inputs and outcomes with the ones the model was built on, using measures like the population stability index, and acting when they cross a threshold set in advance.
Why models drift
A model is a snapshot. It learned from a period of history and assumes the future will look enough like it. That holds until it doesn’t: an interest rate shock, a pandemic, a competitor’s launch, a new fraud scheme, a change in how a field is filled in on the application form.
Nothing in the model breaks when this happens. It keeps producing scores with the same confidence. They just get less right, slowly, and nobody notices unless someone is measuring.
Data drift, concept drift and the rest
| Kind | What changes | Example |
|---|---|---|
| Data drift (covariate shift) | The inputs: who or what the model sees. | A credit card model built on existing customers starts seeing mostly students after a campus promotion. |
| Concept drift | The link between inputs and the outcome. | The same debt-to-income ratio predicts more defaults after rates rise. |
| Prior or label drift | How often the outcome happens. | Fraud rates jump during a holiday season, so the base rate the model assumed is wrong. |
| Upstream data change | A field’s meaning or format, often by accident. | A source system starts recording income monthly instead of yearly. |
People often say "data drift vs model drift". Strictly, data drift is one cause and model drift is the effect: the drop in performance. You can have data drift without much harm, if the model copes with the new inputs, and concept drift with no visible data drift at all.
How drift is measured
| Measure | What it compares | Rule of thumb |
|---|---|---|
| Population stability index (PSI) | The distribution of a score or input now versus at development, bucket by bucket. | Under 0.10 stable; 0.10 to 0.25 watch; over 0.25 investigate. A convention, not a law. |
| Characteristic stability index (CSI) | The same as PSI, for each input variable. | Points to which input moved. |
| Kolmogorov-Smirnov test | The largest gap between two distributions. | Sensitive with large samples; read with care. |
| Jensen-Shannon distance | How different two distributions are, on a 0 to 1 scale. | Useful for text and category data too. |
| Performance metrics | Accuracy, Gini or AUC, calibration, once outcomes are known. | The real test, but outcomes can take months to arrive. |
That last row is why input measures matter. A mortgage default may take two years to show up. Input drift tells you something has changed long before the outcomes confirm it.
A worked example
A lender’s application score was built on 2021–2022 data. In 2025 its PSI comes in at 0.31. The validator breaks it down by input and finds the self-employed share of applicants has doubled after a broker partnership. The model was never tuned for that group.
The options: recalibrate on recent data, add an overlay for self-employed applicants while a rebuild is done, or restrict the model’s use for that segment. Whatever is chosen, the decision, the evidence and the person who made it get written down. That’s the part an examiner asks about.
What OSFI expects
OSFI’s E-23, in effect from 1 May 2027, asks institutions to set standards for model monitoring: how often, what is tracked and the thresholds that count as a breach. It names changes in model performance, model usage and input data, and calls out the elevated potential for drift in AI and machine-learning models. Generative models drift too, when the provider updates the model behind an API without telling you.
MINERVA records each model’s monitoring thresholds and results against the model in the inventory, so a breach is visible, owned and decided. See OSFI E-23 software.
Related questions
What is the difference between data drift and concept drift?
Data drift is a change in the inputs the model sees. Concept drift is a change in the relationship between those inputs and the outcome. Both can make a model less accurate.
What PSI value indicates drift?
A common convention is under 0.10 stable, 0.10 to 0.25 worth watching, and over 0.25 significant enough to investigate. Institutions set their own thresholds by model and risk rating.
How often should model drift be checked?
In proportion to the model’s risk and how fast its environment changes: monthly or even daily for fraud and high-volume models, quarterly for slower ones.
Can large language models drift?
Yes. Their inputs change with how people use them, and a provider can update the underlying model, which changes behaviour without any change on your side.
How do you fix model drift?
Depending on the cause: recalibrate or retrain on recent data, add a temporary overlay, restrict the model’s use, or rebuild it, and record the decision and evidence.
Sources
- OSFI — Guideline E-23, Model Risk Management (2027)
- Federal Reserve — SR 26-2, Revised Guidance on Model Risk Management
- NIST — AI Risk Management Framework
New answers are posted on MINERVA on LinkedIn. Follow along, or send the question you’d like answered next to minerva@minerva.studio.
A plain-language explanation, not legal advice. The regulator’s own text is the source; read it, and take advice on how it applies to your institution.