Deploying ML Models

(Section: Technology and Infrastructure)

Brief Summary

Reliable ML production deployment is a collection of: repeatable artifacts (model/tokenizer/config), standardized surfing (Triton/KServe/vLLM), secure release process (canary/blue green/shadow), observability (latency, quality, drift) and runbooks on incidents. Low latency (anti-fraud/personalization), strict SLO, PII/compliance and cost control are critical for iGaming.

1) Deployment modes

Batch (offline): night/hour tasks (scoring retrospective, updating segments). Cheap, predictable.
Online (synchronous API): anti-fraud, personalization, recommendations, LLM tips. Requires p95 SLA (for example, ≤ 100-300 ms).
Stream (near-real-time): 1-60 sec windows (Flink/Spark/Kafka Streams) for CRM signals and triggers.
Hybrid: online fast rough score + offline recalculation/calibration.

2) Artifacts and packaging

Model artifacts: weights, tokenizer, preprocessing/postprocessing configs, dataset/code version.
Formats: PyTorch/TF SavedModel, ONNX for compatibility, TensorRT-engine for acceleration, GGUF/awq/gptq for LLM quantization.
Containers: Docker OCI images with pinned dependencies; multi-platform tags (CPU/GPU).
Immutable releases: tagging 'model: fraud-v3. 2. 1`, `image: fraud:3. 2. 1`.

3) Serving platform

Triton Inference Server: multi-model, dynamic butching, ensemble-pipelines.
KServe (K8s-native): auto-scale (HPA/KPA), canary/shadow, own runtime.
vLLM/TGI (LLM): continuous batching, KV cache, speculative decoding.
Fichestor: online (ms-SLA) + offline for feature parity.

yaml apiVersion: serving. kserve. io/v1beta1 kind: InferenceService metadata: { name: fraud }
spec:
predictor:
canaryTrafficPercent: 15 model:
modelFormat: { name: triton }
storageUri: s3://models/fraud/v3. 2. 1/
resources: { limits: { nvidia. com/gpu: "1" } }

4) Release strategies

Blue-Green: two identical stacks, instant traffic switching, simple rollback.
Canary: incrementally increasing traffic (1% → 5% → 25% → 100%) across SLO/quality gates.
Shadow: The new model gets a copy of the traffic, the responses don't affect anything - a safe estimate.
A/B tests: we measure business metrics (conversion, retention), statistical significance.

map $request_id $route {
default old;
"~ canary" new; # 5-15% by flag/cook/feature-toggle
}

5) SLOs and operating budgets

Online anti-fraud/personalization: p95 ≤ 100-150 ms, p99 ≤ 250-400 ms.
LLM hints (128-512 tokens): p95 ≤ 300-800 ms generation of the first tokens, tokens/s ≥ target.
Availability: ≥ 99. 9% for critical pathways.
Quality: AUC/PR-AUC/Top-K @ N ≥ threshold;% toxic/incorrect responses ≤ X.
Cost: $/1k requests or $/1k tokens - within budget.

6) CI/CD for models

1. Train/finetune → model in the registry (metadata: data/code/metrics/licenses).

2. Pack & Validate: unit tests pre/post, API compatibility, load tests (latency/tokens/s).

3. Canary Deploy: 1-5% traffic; observability (SLO/quality/cost).

4. Promote/Rollback by criteria gates.

yaml jobs:
build-serve:
steps:
- run: make export_onnx && make docker_build
- run: pytest tests/serve --maxfail=1
- run: python perf_check. py --p95 120 --fail-on-regress
- run: kubectl apply -f kserve-canary. yaml

7) Delay and throughput optimization

Triton/vLLM, request concurrency, pre-/post-processing on CPU.
Quantization (INT8/FP8/INT4) with calibration; TensorRT/ONNX Runtime compilation.
Caching: feature (online-feature/Redis), results and KV cache for LLM.
Warmup: warming up the scales/caches during dumping; "warm" autoscale hearths.
Time budget: early stop, token limit/beam, temperature adaptation.

8) Observability: telemetry, drift, quality

SRE metrics: RPS, p50/p95/p99, errors (5xx/4xx), GPU/CPU util, memory, queue, batch-fill.
ML metrics: AUC/PR-AUC, calibration error, coverage, tokens/s, response length, cache hit.
Drift: PSI/JS divergence by inputs/features, distribution shift monitoring; alerts.
Online Quality: Gold Test Cases, Answer Sampling, Automatic RAG-score/LLM Toxicity.
Logging: prompt/response (anonymized), trace_id, model version.

inference_latency_ms_bucket{model="fraud-v3. 2. 1",le="100"} 12345 inference_qps{model="fraud-v3. 2. 1"} 450 tokens_per_second{model="llm-help-v1"} 210

9) Feature management and consistency

Feature-parity: the same offline/online transformations; version features as code.
Online fichestor: ms-SLA, TTL, upsert, idempotency; cache closer to surfing.
Backfill/refresh: A plan to keep online scoring from diverging from offline metrics.

10) Security, PII and Licenses

PII: tokenization/masking, segmentation by region (EU/TR/LATAM), encryption at rest/in transit.
Secrets/keys: KMS/Secrets Manager, no secrets in images.
LLM policies: content filters, safe stoppers, red-teaming.
Licenses: check conditions for datasets/weights, prohibitions on redistribution/commerce.
Isolation: namespace-RBAC, quotas, tains/tolerations for GPU pools.

11) Autoscale and QoS

Autoscaling: by RPS/queue/latency/GPU-util; min-ready-pods for hotlines.
QoS classes: online critical (anti-fraud)> LLM chat> experiments. Preemption in favor of the critical.
Multi-region: latency-based routing, warmed-up weight caches, feature replication.

12) Runbooks and incidents

p99 growth: check batch-fill, queue, GPU-util, cache miss; enable aggressive butching/lower beam/tokens.
Quality fell: rollback to the previous version, turn on shadow, fix the sources of drift.
The cost is growing: enable quantization/TensorRT, increase batch, optimize feature/cache, reduce the frequency of LLM generations via RAG/result cache.
PII incident: immediate stop-the-line, artifact recall, access audit, procedure report to regulator.

13) Sample templates

text dynamic_batching { preferred_batch_size: [4, 8, 16, 32]
max_queue_delay_microseconds: 2000 }
instance_group { kind: KIND_GPU count: 2 }

--tensor-parallel-size 2
--max-num-seqs 512
--gpu-memory-utilization 0. 9

python resp = client. score({"features": f}) # v3. 2. 1 assert set(resp. keys()) >= {"score","version","latency_ms"}

14) Implementation checklist

1. Define SLO/SLA (latency/availability/quality/cost).
2. Standardize artifacts and model registry (versions, metadata).
3. Choose a serving stack (Triton/KServe/vLLM) and a fichester.
4. Set up canaries/blue green/shadow and automatic gates.
5. Build CI/CD: compatibility tests, perf regression, safe promotion.
6. Include observability (SRE + ML metrics), drift monitoring and alerts.
7. Provide PII/security/licenses and auditing.
8. Configure autoscale/QoS and multi-region policies.
9. Prepare the runbook and have a game-day.
10. Enter cost management: butching, quantization, cache, RAG.

15) Antipatterns

Deploy "as is" without canary/observability → unexpected incidents.
Inconsistent offline/online features → metric discrepancy.

The absence of perf tests and limits → p99 "floats."

Logging prompts/responses without anonymization → PII risk.
One common GPU pool for everything without QoS → critical online suffers.
There is no rollback and snapshots of artifacts → long downtime.

Summary

Successful deployment of ML models are containerized artifacts, standardized serving, a secure release process (canary/blue-green/shadow), hard SLOs, and quality/drift/cost observability. Add fichestore, CI/CD with perf gates, PII hygiene, autoscale and QoS - and your anti-fraud/personalization/LLM services will consistently keep peak iGaming loads while remaining predictable in p99 and budget.