Model Versioning & Canary Rollouts
20 min
Deploying a new model version instantly to 100% of traffic means any regression — a bug, a subtle accuracy drop, a latency spike — hits every single user immediately. A canary rollout instead ramps up gradually: start the new version on a small slice of traffic (say 5%), watch its error rate, and only increase that slice if things look healthy. Any regression only affects a small fraction of traffic while it's being caught, and can be rolled back to 0% instantly.
def next_canary_stage(current_pct, error_rate, error_threshold=0.05, cap=100):
if error_rate > error_threshold:
return 0
next_pct = min(current_pct * 2, cap) if current_pct > 0 else 5
return next_pct
The rollback branch always wins regardless of current_pct -- a canary at 80% traffic gets rolled all the way back to 0% just as fast as one at 5%, because the whole point of the pattern is that bad behavior gets caught and reversed quickly at ANY stage.
Write `next_canary_stage(current_pct, error_rate, error_threshold=0.05, cap=100)`: if `error_rate` exceeds `error_threshold`, return `0` (full rollback). Otherwise, if `current_pct` is `0`, return `5` (start the canary at 5%); else return `min(current_pct * 2, cap)` (double the traffic percentage, capped).
What problem does a CANARY rollout (gradually increasing a new model version's traffic share: 5% -> 10% -> 25% -> 100%) solve that an instant 100% cutover doesn't?
You can implement a canary rollout's stage-progression logic — gradual ramp-up on healthy metrics, instant rollback on a regression.