Label Shift MonitorML & Forecasting

I’m inspecting the existing detector, config, and incident notes first so I can identify where prevalence shifts are leaking into the drift signal.

Executed exec_command call_wzvN2MBBx4IgXyrfwbOc69PP

Executed exec_command call_EnBYAh0kNoW15YtxFkNbScAu

Executed exec_command call_rjohRPaZhaCw3sdnWSQbek4S

Executed exec_command call_65v6gjR9CPj3IYP5g9EZHqxc

The current detector is purely marginal: it tests `P(X)` feature-by-feature and ignores the label. I’m checking the public data generator next to confirm the intended failure mode and choose a label-conditioned test that still catches real within-class drift.

Executed exec_command call_VXNauEjRzRsT3xNiMFGDChh8

Executed exec_command call_BiWVcbcy92CfKVQWoohoBOyx

Executed exec_command call_DJwxhXlrvsPr8625qO1ZNWWw

The failure mode is confirmed on the public sample: the current script alerts exactly the fraud-correlated features even though the generator only changes the fraud base rate. I’m prototyping a label-stratified test now and will verify it suppresses those alerts without losing sensitivity to within-label drift.

Executed update_plan call_JHqLDhNOQBwhnYhjuC9TTlvy

Executed exec_command call_qgXXv7rkPRRdBPxAUcpd5Cv7

Executed exec_command call_WwvGSTCKeYAneQwiyIiI3fQt

Executed exec_command call_2SadzbGFWDlkflH0D5byWxcf

Executed exec_command call_xHuOVKZWBVFrX2otyShdpimw

Executed exec_command call_jx2WJbv5q6d4cCpXAF3kSOyY

Executed exec_command call_K7STQVh6ITvTT7biGYjiXm7C

The asymptotic stratified tests still produce a couple of noisy alerts on the public sample, mostly from the small fraud stratum. I’m switching the prototype to a label-stratified permutation test so the p-values are calibrated under changing fraud prevalence rather than under the old marginal assumptions.

Executed exec_command call_cvQX56pY5rQMcycSSqbwWTNP

Executed write_stdin call_eu1uerMGZcA1QHF2nDwXKBzF

Executed exec_command call_nmpNbCtLr3W4yu9lXMyzCOiL

Executed write_stdin call_48lbgmjo7R7SBO056CNk5T8k

I have a detector shape that behaves correctly on the public prevalence-shift case: it standardizes feature distributions to a common fraud mix and uses a label-stratified permutation test, so base-rate changes alone no longer trip the alarm. I’m writing the artifact now, then I’ll rerun it on the public sample and a few synthetic within-class drift cases.

Executed exec_command call_wZbN7M7gRrfvJrGIvx0xR15E

I’m adding the artifact files under `/logs/artifacts` now. The detector will load its colocated config, standardize both windows to the same label mix, and then run a stratified permutation test per feature.

Executed apply_patch call_AGLsrzDYmElyiaw366tgfqcM

The artifact is in place. I’m running the submitted script exactly through its verifier interface on the public sample, then I’ll hit it with a few synthetic within-label drifts to make sure it still fires when it should.

Executed exec_command call_gfSeNC784cqqPgdu0ONcVkzG

Executed exec_command call_M07Ex8nD7qXwIowY4GZuyEx2

Executed exec_command call_GzysEPoEgsPrrwQM5OucMxdd

The public replay is clean: no retraining alert even though the fraud rate jumps from about 10% to 31%. I’m finishing with synthetic checks where only the within-class feature behavior changes, to confirm those still alert.

Executed write_stdin call_CI8aRyiuxVGwsDeOL10EuDIY

Executed update_plan call_H6VowHtTnaTlpw7atAt0UyQz

Created [monitor.py](/logs/artifacts/monitor.py), [config.json](/logs/artifacts/config.json), and [REPORT.md](/logs/artifacts/REPORT.md). The fix replaces the marginal `P(X)` drift test with a label-conditioned detector: both windows are standardized to the same fraud mix, then each feature is tested with a label-stratified permutation test. That removes alerts caused only by fraud-rate spikes while still catching real changes in `P(X | fraud)` or `P(X | not fraud)`. Continuous features use a weighted KS distance; categorical features use weighted total variation. Verification is clean on the public sample: the artifact returns `retrain: false` even though fraud prevalence rises from `10.45%` to `30.7%`. Synthetic checks with actual within-class drift still alert the changed feature, including fraud-only `amount` drift and non-fraud `velocity` drift.