Your click-through rate model is well-calibrated on validation at launch. Six months later it's significantly overestimating CTR across the board. What are the likely causes and how do you fix it?
tldr
CTR overestimation 6 months post-launch: most likely concept drift (user behavior changed), covariate shift (content distribution changed), or feedback loop tightening (training on position-biased clicks inflates CTR estimates). Diagnose by checking feature distributions at serving vs training, slicing calibration by content type and time. Fix: (1) recalibrate on recent data (Platt scaling, fast); (2) retrain on rolling window; (3) fix position bias in training data. Monitor calibration weekly in production — don't wait for users to notice.
follow-up
- How do you distinguish concept drift from covariate shift, and why does the distinction matter for your fix?
- If you retrain daily on the last 90 days of data, what happens to your model's behavior during a major external event (e.g., a holiday, a viral news story)?
- What is population stability index (PSI) and how do you use it to detect feature drift before it hurts model performance?