Deploying ML Models

1) Role and goals

Deployment = Deliver inference reliably to product and operations while meeting RG/AML/Legal and budgets.
Objectives: low latency, high availability, reproducibility, safety and rapid rollback.

2) Serving architectures

2. 1 Patterns

Online (real-time): REST/gRPC, p95 50-150 ms for personalization; ≤2 -5 s for RG/AML alerts.
Near-real-time: micro-matches 1-5 min (OLAP-showcases).
Batch/offline: Gold night showcases, WORM exports for the regulator.
In-process scoring: embedding the light model into the service (low latency).
Serverless: cold start features for rare tasks.

2. 2 Topologies

Single Model Service → simple and fast.
Ensemble/Graph (router → preprocess → model → postprocess) → complex pipelines.
Sidecar Feature Fetcher → pulls online features from caches (Redis/Scylla).

3) Model register and versions

Registry: 'model _ id', 'version', 'stage = {Staging, Production, Archived}', artifacts (weights, preprocessor, calibration), requirements (CPU/GPU/memory), model card (data, metrics, risks, fairness).
Immutable artifacts: content-hash; WORM copies of releases.
Output policy: through registry and declarative manifests only.

4) Containerization and packaging

dockerfile
FROM python:3. 11-slim
ENV PYTHONUNBUFFERED=1
WORKDIR /app
COPY requirements. txt.
RUN pip install -r requirements. txt --no-cache-dir
COPY artifacts/./artifacts/
COPY src/./src/
CMD ["python", "src/serve. py"]

python serve. py from fastapi import FastAPI import joblib, time app = FastAPI()
model = joblib. load("artifacts/model. joblib")
scaler = joblib. load("artifacts/scaler. joblib")

@app. post("/score")
def score(payload: dict):
t0 = time. time()
x = preprocess(payload, scaler)
y = float(model. predict_proba([x])[0,1])
return {"score": y, "latency_ms": int((time. time()-t0)1000), "model_version": "1. 8. 3"}

5) Kubernetes/Helm and Autoscaling

yaml apiVersion: apps/v1 kind: Deployment metadata: {name: ml-score, labels: {app: ml-score}}
spec:
replicas: 3 selector: {matchLabels: {app: ml-score}}
template:
metadata: {labels: {app: ml-score}}
spec:
containers:
- name: api image: registry/ml-score:1. 8. 3@sha256:...
ports: [{containerPort: 8080}]
resources:
requests: {cpu: "500m", memory: "512Mi"}
limits:  {cpu: "2",  memory: "2Gi"}
envFrom: [{secretRef: {name: ml-secrets}}]
readinessProbe: {httpGet: {path: /healthz, port: 8080}, periodSeconds: 5}
livenessProbe: {httpGet: {path: /livez,  port: 8080}, periodSeconds: 10}

yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: {name: ml-score-hpa}
spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: ml-score}
minReplicas: 3 maxReplicas: 50 metrics:
- type: Pods pods:
metric: {name: requests_per_second}
target: {type: AverageValue, averageValue: "15"}
- type: Resource resource: {name: cpu, target: {type: Utilization, averageUtilization: 70}}

6) Roll-out strategies

Shadow: the new model processes copies of requests, responses are ignored; comparison of metrics/latency/cost.
Canary: 1-10% of traffic → 25% → 50% → 100% with green SLOs; automatic rollback on degradation.
Blue-Green: parallel stacks; instant routing switch.
Gradual Feature Flag: Market/Tenant/Device Routing.
A/B/n: online experiments with sequential testing.

yaml routes:
- match: {tenant: "EEA"} # feature-flags/rules to: {model: "1. 8. 3", weight: 20}
- match: {tenant: "EEA"}
to: {model: "1. 7. 9", weight: 80}

7) Feature online/offline: equivalence

Unified transformation library and Feature Store (online + offline).
Equivalence test: MAE/MAPE between online features and offline reference on a reference sample.
Cache feature: Redis/Scylla, TTL for window features; timeouts and fallbacks.

8) Observability, SLO and alerting

8. 1 SLI/SLO benchmarks

Latency: p95 ≤ 150 ms (personalization), p99 ≤ 300 ms; RG/AML alerts ≤ 5 with end-to-end.
Availability: ≥ 99. 9%.
Error inference: ≤ 0. 5% 5xx; coverage ≥ 99%.
Drift: PSI characteristic/rate <threshold, ECE (calibration) is stable.
Бизнес: uplift Net Revenue, fraud saved, time-to-intervene.

8. 2 Metrics (Prometheus)

yaml
- http_request_duration_seconds{quantile="0. 95"}
- http_requests_total{code=~"5.."}
- model_inference_latency_ms_bucket
- feature_fetch_latency_ms_bucket
- model_score_distribution_bucket
- psi_feature_{name}
- expected_cost_live

yaml
- alert: HighP95Latency expr: histogram_quantile(0. 95, sum(rate(model_inference_latency_ms_bucket[5m])) by (le)) > 0. 15 for: 10m
- alert: DriftDetected expr: psi_feature_amount_base > 0. 25 for: 15m

Трейсинг: OpenTelemetry — span’ы `feature_fetch`, `score`, `postprocess`, `guardrail`.

9) RG/AML guardrails and security policy

Pre-/Post-filter: masks of prohibited actions (frequency of impressions, cooldown, prohibition of aggressive offers).
Policy Shielding: speed above RG threshold → soft intervention/pause.
Audit: logging 'policy _ id', 'propensity', 'mask', 'decision', 'reason'.
PII and residency: tokens instead of ID, separate encryption and cluster keys on EEA/UK/BR; banning cross-region joins without justification.
Secrets: KMS/CMK, Secret Manager; no PII in logs/tracks.

10) Calibration, thresholds and decision policy

Calibration (Platt/Isotonic) as an artifact.
Threshold by expected cost; configurable in the registry/feature flag.
Safety caps: upper/lower action limits, manual override for compliance.

11) Rollbacks, degradation and DR

One-click rollback - Switches the route to the previous' model _ version '.

Runbook: scripts "latentnost↑," "5xx↑ errors," "drift/calibration broke," "external feature provider unavailable."

Fault isolation: circuit breaker, retry/backoff, cache of the last valid solution.
DR: backups of artifacts/registry, replication to a "warm" region, exercises.

python try:
features = fetch_features(timeout=30)
except TimeoutError:
features = last_known_good(user_id) # fallback

12) Cost engineering and performance

Path profiling: features (30-60%), model (20-40%), network/IO (10-30%).
Cost reduction: caching hot features, replay quotas, lightweight models, INT8/FP16 (if appropriate), lazy-postprocess.
HPA on RPS/CPU/latency, state-size restriction on stream feature.
Chargeback: cost/request, cost/feature; budgets for markets/teams.

13) Secure releases and compliance

Product readiness criteria: SLO green on shadow, no drift/leak, model card is full, fairness slices are normal.
Regulator: WORM-archive of the release (weights, calibration, thresholds, metrics, test logs), unchangeable export reports.
DSAR/RTBF: procedures for removing traces of a specific user from caches/logs/features.

14) QA before rolling out

Integration tests: API contract, feature diagrams, empty/extreme cases.
Load: p99, distribution tails, burst traffic.
Online/offline equivalence test.
Chaos tests: turning off the feature cache/base, timeouts of external services.

15) Configuration examples

yaml
- match: [headers: {x-exp: "canary"}]
route:
- destination: {host: ml-score-v1-8-3, weight: 20}
- destination: {host: ml-score-v1-7-9, weight: 80}

python
@app. get("/healthz")
def health():
return {"ok": True, "model_version": "1. 8. 3", "registry_sig_ok": verify_signature()}

16) Processes and RACI

R (Responsible): MLOps (serving/orchestration/observability), Data Eng (features/caches/contracts), Data Science (model cards/calibration/thresholds).
A (Accountable): Head of Data / CDO.
C (Consulted): Compliance/DPO (PII/RG/AML/DSAR), Security (KMS/Secrets/Audit), SRE (SLO/Incidents), Finance (ROI/Budgets).
I (Informed): Product/Marketing/Operations/Support.

17) Implementation Roadmap

1. Model registry and immutable artifacts; FastAPI/gRPC service + K8s/Helm.

2. Shadow launch with p95/5xx/drift monitoring, feature equivalence test.

3. Canary 10% → 50% → 100% with auto-script rollback and alerts.

4. Model card, calibration, expected-cost threshold and Guardrails v1.

• Feature cache, circuit breakers, runbooks/DR exercises.
• Autoscaling on RPS/latency, cost-dashboard and chargeback.
• Fairness slice monitoring, WORM release archive.

• Graph of serving (candidates → re-rank), slate-logging propensity.
• Multi-region, residency (EEA/UK/BR) with individual keys.
• Auto-roll/overdrive, autogene quality/calibration reports.

18) Delivery checklist

• Model card is full; data/features/calibration/thresholds are versioned.
• Contracts feature and online/offline equivalence test are green.
• SLO: p95, 5xx, coverage - green on shadow and 10% canary ≥ 24 h.
• Alerts and dashboards (latency/errors/drift/expected-cost) are included.
• Guardrails RG/AML and solution audits are active; PII/residency met.
• One-click rollback and runbook incidents tested.
• The cost fits into the budget; HPA and cache are configured.

19) Anti-patterns and risks

Manual rollouts without registry and immutable artifacts.
Inconsistent online/offline features → discrepancies in sales.
No shadow/canary → hidden regressions.
The threshold is not expected cost, there is no calibration.
Synchronous external lookups without timeouts/cache.
No DR/rollback, no WORM release archive.

20) The bottom line

ML production is not a "model challenge," but an engineering platform: registry and versions, safe rollouts (shadow/canary/blue-green), feature and calibration discipline, observability and SLO, guardrails for RG/AML, and a clear rollback plan. By following this playbook, you get a fast, reliable and compliant inference that consistently delivers business value at a controlled cost.