Monitoring & Drift Detection
20 min
A deployed model's input data can silently shift over time — user behavior changes, an upstream system starts sending slightly different values, a seasonal pattern kicks in. Data drift monitoring compares a live batch of feature values against a trusted reference batch (often the training data) to catch this before it quietly degrades predictions. A simple, effective drift signal: how many reference standard deviations apart are the two batches' means?
def feature_drift(reference_values, current_values):
ref_mean = sum(reference_values) / len(reference_values)
cur_mean = sum(current_values) / len(current_values)
ref_std = (sum((x - ref_mean) ** 2 for x in reference_values) / len(reference_values)) ** 0.5
if ref_std == 0:
return 0.0
return round(abs(cur_mean - ref_mean) / ref_std, 4)
A score around 0.1 is well within normal sampling noise for this reference distribution; a score of 8+ is a huge, unmistakable shift -- exactly the kind of jump that should page someone or trigger a retraining check, which is exactly what the next module covers.
Write `feature_drift(reference_values, current_values)`: compute the difference between the two sample means, divided by the REFERENCE sample's standard deviation (a standardized drift score). Return `0.0` if the reference standard deviation is `0`. Round to 4 decimal places.
Why standardize the drift score by dividing by the reference standard deviation, instead of just comparing raw mean values?
You can compute a standardized data drift score for a feature, putting shifts of very different magnitudes on a comparable scale.