Outcome Drift Detection: Catching Silent Regressions After a Rule Release

A release that returns 500 errors is caught within seconds. A release that keeps answering 200 OK at the same p95 latency and starts producing a different distribution of decisions — more DECLINE, fewer APPROVE — is caught by nothing in standard APM.

The Decision Server records 'runtime.outcome_mix' — an hourly count per outcome shape, with numbers collapsed to sign plus magnitude and strings hashed. 'GET /monitoring/drift' compares two versions and returns PSI (Population Stability Index), total variation, shape movers, and a level verdict (stable / minor / moderate / significant / insufficient).

**PSI thresholds**: < 0.1 stable, 0.1–0.2 minor, > 0.2 significant (hold rollout), > 0.25 consider rollback.

**The AI step** is asked only when drift is measured: both sides sampled, verdict other than insufficient, at least one recorded shape shift. Asked to explain a drift that was never measured, a model will invent one. The panel prevents this at the button level.

**The caveat always shown**: "two live populations are being compared — an outcome shift is not by itself proof the version caused it." Drift detection is evidence, not verdict.