Skip to Content

Drift Detection with KS and PSI

When to use

Detect when production input distribution shifts from training — the precondition for most silent model degradations.

Analogy

Teacher comparing two classes: did the new class of students have similar knowledge before reusing the same curriculum?

Data-flow diagram

   Kolmogorov-Smirnov: max|CDF_train - CDF_prod|
   Population Stability Index (PSI): binned divergence
     psi under 0.1   -> stable
     psi 0.1..0.2   -> minor drift
     psi over 0.2    -> significant drift

Deep explanation

Input drift happens before accuracy drift. KS is a non-parametric two-sample test on continuous features. PSI is the standard industry metric on binned features. Use both: KS gives significance, PSI gives magnitude. Always measure drift per feature — aggregate drift hides important shifts in rare but critical features.

Examples

Example 1

from scipy.stats import ks_2samp
import numpy as np
train = np.random.normal(0, 1, 10_000)
prod  = np.random.normal(0.2, 1, 10_000)
stat, p = ks_2samp(train, prod)
print(f'KS stat={stat:.3f}  p={p:.3g}')

scipy.stats.ks_2samp returns stat and p-value — p is the practical decision boundary.

Example 2

import numpy as np
def psi(expected, actual, bins=10):
    eps = 1e-6
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    e, _ = np.histogram(expected, bins=edges)
    a, _ = np.histogram(actual,   bins=edges)
    return np.sum((a/sum(a) - e/sum(e)) * np.log((a/sum(a)+eps)/(e/sum(e)+eps)))
print('PSI:', psi(train, prod))

PSI is interpretable: 0.1 minor shift, 0.2 significant.

Example 3

# pip install alibi-detect
import alibi_detect
from alibi_detect.cd import KSDrift
cd = KSDrift(train.reshape(-1, 1).astype('float32'), p_val=0.05)
print('drift?', cd.predict(prod.reshape(-1, 1).astype('float32')))

alibi-detect wraps KS with a memory-efficient running updater.

Common mistake

Running KS once on the whole dataset. With 10 million samples the test detects vanishingly small drift — false alarms. Also aggregating drift into one number hides per-feature shifts.

Key takeaway

KS gives significance; PSI gives magnitude. Measure per-feature; alert at PSI over 0.2 OR KS+effect-size sanity.

Production Failure Playbook

Failure scenario 1: aggregated-drift-blind

Failure scenario 2: tiny-drift-alert-storm