A model that performs well at deployment does not stay that way. The world changes, the data changes, and the model — frozen at training time — silently gets less accurate. Nothing errors; predictions keep coming, just worse ones. This silent decay is the defining risk of ML in production, and monitoring across three layers is the only way to catch it before it shows up as a business problem.
Operational health
The first layer is the same as any service: is the model serving? Watch latency, throughput, error rates, and resource use. This catches the infrastructure failures — a serving endpoint down, predictions timing out — and it is necessary but nowhere near sufficient, because a perfectly healthy endpoint can be serving increasingly wrong predictions. Operational monitoring tells you the model is running, not that it is right.
Data and prediction drift
The layer that catches silent decay is drift monitoring. Because ground-truth labels often arrive late or never, you monitor the model's inputs and outputs as a proxy for its health:
- Data drift — the distribution of incoming features shifts away from the training data; the model is seeing inputs it was not trained for.
- Prediction drift — the distribution of the model's outputs shifts, which can signal the input world has changed.
- Feature integrity — inputs that are null, out of range, or malformed, which quietly corrupt predictions.
Drift is your early warning. It does not prove the model got worse, but it flags that the conditions it was trained under no longer hold — the signal to investigate or retrain before quality visibly craters.
Real model quality
When you can eventually obtain ground-truth labels — a loan defaulted or did not, a flagged transaction was or was not fraud — you can measure the model's actual accuracy in production against reality. This is the truest signal but usually the most delayed, because labels take time to arrive. Where you can get it, close the loop and track real performance over time; where you cannot, drift monitoring is your stand-in.
Unlike software that throws an error when it breaks, a decayed model just returns worse answers while looking perfectly healthy. Operational monitoring will show green the entire time. That is why drift and quality monitoring are not optional extras — they are the only things that see the failure mode unique to ML, and without them you learn about model decay from a business metric going the wrong way.
Monitoring ML models in production means three layers: operational health that any service needs, drift monitoring on inputs and outputs as the early warning for silent decay, and real quality measurement against ground truth when labels arrive. The operational layer alone will report a healthy system while the model quietly rots — so it is the drift and quality layers that actually protect you from the failure mode that makes ML different.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.