Label Shift MonitorML & Forecasting

Human baseline3h

TaxonomyML & ForecastingML & Forecasting

Task prompt

# Fraud Drift Monitor The drift-monitoring pipeline in `/app` compares a reference CSV to a current production CSV and emits a retraining trigger when it believes feature distributions have changed. The on-call team has noticed that during fraud-rate surges the monitor fires retraining alerts on several features that are correlated with fraud. They suspect many of these alerts are false positives driven by the surge itself, not by genuine changes in the feature behavior. Your goal is to make the monitor robust to fraud-rate spikes: it should not waste retraining cycles when the only thing that changed is how common fraud is, but it must still detect real changes in the feature behavior. ## What you are given - `/app/monitor.py` — the current detector. - `/app/config.json` — feature metadata (label column, continuous/categorical lists). - `/app/reference.csv` and `/app/current.csv` — public examples for local debugging. - `/app/INCIDENT_RUNBOOK.md` — notes from the last alert spike. ## What you must produce Write the following under `/logs/artifacts/`: - `/logs/artifacts/monitor.py` — a self-contained fixed detector. - `/logs/artifacts/config.json` — the configuration used by your detector. - `/logs/artifacts/REPORT.md` — a concise explanation of the bug and the fix. During verification, the submitted monitor may be executed on hidden streams and produce intermediate JSON outputs under `/logs/artifacts/`; these intermediate files are not part of your graded deliverables. ## Detector interface The verifier will invoke your artifact as: ```bash python /logs/artifacts/monitor.py <reference.csv> <current.csv> <output.json> --label <label_column> ``` The script must accept positional arguments for the reference CSV, current CSV, and output JSON path, plus a `--label` option. It must write a JSON object to `<output.json>` containing at least: ```json { "retrain": true, "alerts": ["feature_3", "feature_7"], "p_values": { ... } } ``` - `alerts` is the list of feature names whose distribution has genuinely shifted. - `retrain` is `true` when `alerts` is non-empty. The example feature names above are placeholders; do not hard-code them. You may add extra fields, but `alerts` and `retrain` must be present. ## Notes - The public data in `/app` is only for local debugging; the hidden verifier uses separate streams. - Do not simply raise thresholds or apply generic multiple-testing corrections to silence alerts. - A correct solution must separate prevalence-driven alerts from genuine feature drift.

What it tests

The task is hard because the visible failure (many alerts during a fraud-rate surge) is accompanied by an authoritative-looking runbook that recommends the wrong fix (multiple-testing correction + threshold tuning). The real issue is a confounding prevalence shift: the marginal feature distributions move because P(Y) changed, not because P(X|Y) changed. A solver must infer this from the synthetic data, reject the incorrect runbook, and implement a label-conditional, type-aware detector. The data is synthetically generated using seeded numpy RNGs in generate_public_data.py and generate_fixtures.py, but it mimics a realistic production monitoring scenario with rare fraud labels, correlated features, and mixed continuous/categorical variables. This creates the three-way entanglement of (1) separating prevalence from conditional drift, (2) using the right test for continuous vs. categorical features per label stratum, and (3) preserving recall for genuine covariate drift while silencing prevalence-driven alerts. In the real world, an MLOps engineer or data scientist maintaining a production fraud-monitoring pipeline would be expected to recognize this distinction.