Model drift and data updates

1) Why it matters

In iGaming, the distribution of traffic, payments and gaming behavior is changing rapidly (seasonality, providers, stocks, regulatory rules). Without systemic drift control, the expected cost of errors is growing: losses in Net Revenue, false RG/AML interventions, and an increase in bonus abuse. The goal is to detect drift early, diagnose the cause, and safely update data and models.

2) Drift taxonomy (what can "float")

Covariate drift (X): features have changed (for example, the share of mobile/ASN, mix of providers).
Label/Outcome drift (Y): the event base (chargeback-rate, click/convert-rate) has changed.

Concept drift (P(Y	X)): previous signs are otherwise associated with the target (new fraud schemes, other reactions to promotions).
Feature drift: displacement of statistics of a specific feature (mean/var/rate/missing).
Schema drift: evolution of schemes/categories/ID of providers, new values.
Operational drift: lag feature, source errors, missing/timeout growth, hot keys.
Fairness drift: deterioration of quality/calibration in slices (market, device, provider).

3) Signals and thresholds (landmarks)

PSI (Population Stability Index): 0. 10–0. 20 warning,> 0. 20 - drift event.
KL-divergence/JS: growth to thresholds by key features/speed.
KS for rates (under labels): increasing CDF discrepancy.
ECE (calibration):> 0. 05 warning,> 0. 07 - action.
Expected-cost @ threshold: X% growth to base model.
Coverage and Missing-rate: <99% coverage drop, missing/timeout> threshold growth.
Slice metrics: PR-AUC/ECE/expected-cost by market/device; drop triggers> Y%.

Window selection: operational - 1h/6h/24h (sliding), reporting - D + 1/D + 7 (accounting for delayed labels).

4) Windows and labels (delays)

Fast proxy labels: click, deposit 7d, completed RG case - for early valuation.
Retained labels: chargeback (45-90d), churn/LTV - for retrospective validation and threshold/calibration adjustment.

As-of discipline: neither in features, nor in labels - "events from the future."

5) Root cause diagnostics (RCA checklist)

1. Data: PSI/KL by top features, missing/lag, schema diffs, new categories.
2. Source: provider alerts (PSP/game provider), API/timeout errors.
3. Seasonality/promotions: peak missions/tournaments, RTP/catalog changes.
4. Region/Residency: Traffic shift across markets, new KYC/RG rules.
5. Technique: degradation of the feature cache, slow CDC, small files storm.
6. Reference books/currency rates/calendars: out of date? (FX/holidays).
7. Fairness: quality failure in specific slices.

6) Playbook of drift actions

6. 1 Covariate/Feature drift

Quick: update calibration (Platt/Isotonic D + 1), adjust the threshold according to expected-cost.
Medium-term: partial refeature (stable aggregates/windows), update TE/WOE with time-aware CV.
Long-term: retrain with new selections/windows, if necessary, revise the architecture of the feature.

6. 2 Label/Outcome drift

Recalculate thresholds by fresh labels (D + 1, D + 7), re-calibrate.
Fix guardrails (limit aggressive actions to stabilization).

6. 3 Concept drift

Shadow training of the new version on the latest data → canary → full rollout.
Consider domain decomposition (separate models by segment/market).

6. 4 Schema/Operational drift

Enable dual v1/v2 schema entry, online/offline equivalence test.
Fix sources, enable cache/folbacks, compensate for backfill gaps.

6. 5 Fairness drift

Temporary slide thresholds/calibration, targeted retrain/rebalance feature, proxy variable audit.

6. 6 Escalations

Kill-switch (guardrails breach) → safe fallback/previous version.
Rollback one-click with growth 5xx/latency/expected-cost.

7) Data update policy

7. 1 Increments and CDC

Stable watermarks/log replication; idempotent MERGE/UPSERT.

7. 2 Backfill/Reprocessing

Backfill: Dogon for range (with quotas and windows).
Reprocessing: recalculating when logic changes/errors are corrected.
Метки: `logic_version`, `reprocessed_at`, `reason`; impact report (metrics/cost).

7. 3 Time-travel/WORM

ACID tables (Delta/Iceberg/Hudi), WORM archives of reports/releases.
"As it was on the date": reproducibility of regulatory reporting.

7. 4 Directories/FX/Calendars

Auto-updates, signature, versions and freshness check (SLO on latency and age).

8) Metrics and alerts (minimum set)

Data: PSI/KL by top features, missing-rate, feature-fetch latency.
Quality: PR-AUC/KS (on proxy labels), ECE, expected-cost @ thr.
Operations: p95/p99 latency, 5xx, coverage, autoscaling, cost/request.
Fairness: slice panels (markets/devices/providers).
Contracts: schema-violations, online/offline equivalence test.

9) Model update procedures

1. Shadow: new model on query copies, latency/quality/cost comparison.
2. Canary: 5-10% → 25% → 50% → 100% at green SLOs.
3. Threshold/calibration: calculate D + 1; thresholds - config in the registry.
4. Documentation: model card (data, windows, metrics, risks, fairness).
5. Archive: WORM release (weights, calibration, test logs, drift reports).

10) Examples (fragments)

10. 1 PSI in SQL ideas (bining prepared in advance)

sql
-- ref_dist(bin, p_ref), prod_dist(bin, p_prod) для фичи amount_base
SELECT SUM((p_prod - p_ref) LN((p_prod + 1e-9)/(p_ref + 1e-9))) AS psi
FROM (
SELECT bin, COUNT()/SUM(COUNT()) OVER() AS p_prod
FROM prod_binned GROUP BY bin
) p
JOIN (
SELECT bin, COUNT()/SUM(COUNT()) OVER() AS p_ref
FROM ref_binned GROUP BY bin
) r USING (bin);

10. 2 Expected-cost threshold adjustment (pseudocode)

python thr_grid = np. linspace(0. 01, 0. 99, 99)
costs = [expected_cost(y_true, y_prob >= t, c_fp, c_fn) for t in thr_grid]
thr_best = thr_grid[int(np. argmin(costs))]

10. 3 Equivalence online/offline feature

python diff = np. abs(f_online. values - f_offline. values)
assert np. quantile(diff, 0. 95) < MAX_ABS_DIFF_95

10. 4 Backfill with load limiting (orchestration idea)

yaml job: backfill_gold_ggr limits: {concurrency: 1, max_partitions_per_run: 4}
guards:
- window: "02:00-06:00"
- markets: ["EEA","UK"]
- budget: "compute_hours<=50"

11) Tests and change control

Contracts schemes/feature: consumer-driven tests, double entry v1/v2.
Regression tests of metrics: do not worsen PR-AUC/ECE/expected-cost beyond tolerance.
Online/offline equivalence tests: on a reference sample.
Chaos tests: feature cache failure, source timeouts, burst traffic.

12) Fairness and Compliance

Slice reports (disparate impact, equalized odds), slice thresholds/calibration.
PII-minimization, residency (EEA/UK/BR), DSAR/RTBF, Legal Hold.
Audit decisions: 'policy _ id', 'threshold', reasons for interventions, WORM logs.

13) Cost and performance

Cost dashboards: cost/request, cost/feature, state-size стрима, IO/scan/GB для batch.
Optimization: materialization of heavy offline features, hot window cache, INT8/FP16 with equal quality.
Quotas: backfill/replay limits, retrain budget by market/team.

14) RACI

R (Responsible): MLOps (monitoring/registry/kickouts), Data Eng (data/CDC/backfill/contracts), Data Science (diagnostics/calibration/retrain/fairness).
A (Accountable): Head of Data / CDO.
C (Consulted): Compliance/DPO (PII/RG/AML/DSAR), Security (KMS/Audit), SRE (SLO/Value), Finance (Budgets/ROI).
I (Informed): Product/Marketing/Operations/Support.

15) Roadmap

MVP (2-4 weeks):

1. PSI/KL by top features and score, ECE, expected-cost on proxy labels.

2. Dashboards coverage/missing/feature-lag, alerts and runbook 'and.

3. Procedures for recalibration (D + 1) and thresholds; shadow path for new models.

4. Scheme contracts/feature and online/offline equivalence test.

Phase 2 (4-8 weeks):

RCA panel, slice/fairness monitoring, backfill plan with quotas.
Auto-roll calibration/thresholds, threshold simulator (what-if).
WORM archive of drift reports and releases.

Phase 3 (8-12 weeks):

Auto-retrain on drift events (canary), multi-regional policy.
Cost-quotas/chargeback, chaos-/DR-exercises, autogen documentation.

16) Pre-sale checklist

SLI/SLO and alerts are configured (PSI/ECE/expected-cost/coverage/latency/5xx).
Schema contracts/feature and v1/v2 double entry are green.
Recalibration/threshold-update procedures are documented and automated.
Shadow/canary with one-click rollback checked.
Backfill/reprocessing with quotas and windows - ready; WORM archive is enabled.
Slice/fairness panels and segment owners are assigned.
PII/DSAR/RTBF/Legal Hold policies enforced; audit enabled.
Cost under control (cost/request, cost/feature), cache/TTL configured.

17) Anti-patterns and risks

We saw a high PSI - immediately retrain "blindly," without RCA and calibration.
There is no accounting for delayed labels → false conclusions, "sawing thresholds" every day.

Lack of an online/offline equivalence test → "double reality."

Ignore fairness: Market/device failures remain invisible.
Backfill without quotas/windows → a hit to costs and SLAs.
The threshold is fixed "forever" → expected-cost growth with seasonality.

18) The bottom line

Drift management is not a one-time retrain, but a process: observability → diagnostics → minimal-risk action (calibration/threshold) → safe retrain/refichering → cost control and auditing. With this discipline, models remain accurate, ethical and compliant, even as player, provider and market behaviour changes.

Model drift and data updates

Get in Touch

Quick Contact

The video will be updated soon

We are currently very busy with projects