Drift Detection via Two-Sample Kolmogorov-Smirnov on Production Probe Score Streams
We describe the drift-detection layer that sits between the MIVP runtime monitor and the probe registry. A two-sample Kolmogorov-Smirnov statistic against the calibration-time reference distribution raises a drift breach when the production raw-score stream diverges from calibration; on breach, the probe is auto-demoted to E0 and excluded from runtime gating until revalidation. We provide the construction, the auto-demotion loop, and a quantitative evaluation on a synthetic drift scenario.
1. Introduction
A probe's calibration is a snapshot. It does not tell you how the probe behaves on the distribution your production traffic has drifted to. Without a drift layer, the failure mode is: probe calibration is great, weights have not changed, but the input distribution has. We adopt the two-sample Kolmogorov-Smirnov statistic (Kolmogorov, 1933; Smirnov, 1939).
2. Background
Concept drift and covariate shift are surveyed by Lu et al. (2018) and Quionero-Candela et al. (2009). KS tests are standard in drift-detection literature (Gama et al., 2014; Bifet et al., 2010).
3. Method
3.1 The DriftMonitor
The monitor maintains a calibration reference and a sliding window of production raw scores. KS is computed on the trailing window. Threshold: configurable, default per-comparison α = 0.01 with Bonferroni adjustment.
3.2 The Auto-Demotion Loop
DriftGuard attaches a DriftMonitor to each probe key. On drift, the probe is demoted to E0/INACTIVE. Demotion is automatic; re-promotion is not. The asymmetry biases toward demotion.
3.3 Recovery
Recovery path is RevalidationPipeline plus a manual trigger. We do not auto-recover; the demotion entry remains in the audit trail.
4. Results
Synthetic drift: probe calibrated on N(0,1); production stream N(0,1) for 1000 obs then N(0.5,1) for 1000 obs.
| Phase | Mean KS | Threshold | Detection latency |
|---|---|---|---|
| Pre-drift | 0.04 | 0.13 | — |
| Post-drift (first 200) | 0.18 | 0.13 | 167 obs |
5. Discussion
KS is a univariate test on a scalar stream. Multivariate drift and concept drift are not caught by this layer.
6. Conclusion
A two-sample KS drift-detection layer that auto-demotes probes on distribution shift. Composes with calibration and ledger layers.
References
Bifet, A. et al. (2010). PAKDD. / Gama, J. et al. (2014). IEEE TKDE. / Kolmogorov, A. N. (1933). / Lu, J. et al. (2018). IEEE TKDE. / Quionero-Candela et al. (2009). MIT Press. / Smirnov, N. V. (1939). Bulletin de l'Académie des Sciences de l'URSS.