ML modellerini ýerleşdirmek

1) Roly we maksatlary

Ýerleşdirmek = RG/AML/Legal we býudjetler berjaý edilende önüme we amallara infensi ygtybarly eltip bermek.
Maksatlar: pes gizlinlik, ýokary elýeterlilik, köpelmek ukyby, howpsuzlyk we çalt gaýdyp gelmek (rollback).

2) Serwingiň arhitekturasy

2. 1 Nagyşlar

Online (real-time): REST/gRPC, p95 şahsylaşdyrmak üçin 50-150 ms; RG/AML-alertler üçin 2-5 s ≤.
Near-real-time: mikrobatçi 1-5 minut (OLAP-vitrinalar).
Batch/offline: Gold gijeki penjireler, düzgünleşdiriji üçin WORM eksporty.
In-process scoring: ýagtylyk modelini hyzmatda goýmak (pes gijikdirme).
Serverless: seýrek meseleler üçin sowuk başlangyç funksiýalary.

2. 2 Topologiýalar

Single Model Service → aňsat we çalt.
Ensemble/Graph (router → preprocess → model → postprocess) → çylşyrymly paýlaýynlar.
Sidecar Feature Fetcher → kesişden (Redis/Scylla) onlaýn-çeňňekleri çykarýar.

3) Model sanawy we wersiýalary

Registry: 'model _ id', 'version', 'stage = {Staging, Production, Archived}', artefaktlar (agram, processor, kalibrlemek), talaplar (CPU/GPU/ýat), model kartoçkasy (maglumatlar, metrikler, töwekgelçilikler, fairness).
Imutable artefaktlar: content-hash; WORM göçürmeleri.
Çykyş syýasaty: diňe reýestr we deklaratiw manifestler arkaly.

4) Gaplamak we gaplamak

dockerfile
FROM python:3. 11-slim
ENV PYTHONUNBUFFERED=1
WORKDIR /app
COPY requirements. txt.
RUN pip install -r requirements. txt --no-cache-dir
COPY artifacts/./artifacts/
COPY src/./src/
CMD ["python", "src/serve. py"]

python serve. py from fastapi import FastAPI import joblib, time app = FastAPI()
model = joblib. load("artifacts/model. joblib")
scaler = joblib. load("artifacts/scaler. joblib")

@app. post("/score")
def score(payload: dict):
t0 = time. time()
x = preprocess(payload, scaler)
y = float(model. predict_proba([x])[0,1])
return {"score": y, "latency_ms": int((time. time()-t0)1000), "model_version": "1. 8. 3"}

5) Kubernetes/Helm we awtoskeyling

yaml apiVersion: apps/v1 kind: Deployment metadata: {name: ml-score, labels: {app: ml-score}}
spec:
replicas: 3 selector: {matchLabels: {app: ml-score}}
template:
metadata: {labels: {app: ml-score}}
spec:
containers:
- name: api image: registry/ml-score:1. 8. 3@sha256:...
ports: [{containerPort: 8080}]
resources:
requests: {cpu: "500m", memory: "512Mi"}
limits:  {cpu: "2",  memory: "2Gi"}
envFrom: [{secretRef: {name: ml-secrets}}]
readinessProbe: {httpGet: {path: /healthz, port: 8080}, periodSeconds: 5}
livenessProbe: {httpGet: {path: /livez,  port: 8080}, periodSeconds: 10}

yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: {name: ml-score-hpa}
spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: ml-score}
minReplicas: 3 maxReplicas: 50 metrics:
- type: Pods pods:
metric: {name: requests_per_second}
target: {type: AverageValue, averageValue: "15"}
- type: Resource resource: {name: cpu, target: {type: Utilization, averageUtilization: 70}}

6) Yzyna almak strategiýalary

Shadow (garaňky başlangyç): täze model haýyşlaryň nusgalaryny gaýtadan işleýär, jogaplar äsgerilmeýär; metrikany/gizlinligi/bahasyny deňeşdirmek.
Canary: 1-10% traffik → 25% → 50% → 100% ýaşyl SLO; pese gaçanda awtomatiki çykmak.
Blue-Green: paralel akymlar; derrew marşrutlaşdyryş switch.
Gradual Feature Flag: bazarlar/tenantlar/enjamlar boýunça marşrut.
A/B/n: sequential testing bilen onlaýn synaglar.

yaml routes:
- match: {tenant: "EEA"} # feature-flags/rules to: {model: "1. 8. 3", weight: 20}
- match: {tenant: "EEA"}
to: {model: "1. 7. 9", weight: 80}

7) Onlaýn/awtonom çyzgylar: ekwiwalentlik

Bitewi özgerişler kitaphanasy we Feature Store (online + offline).
Ekwiwalentlik synagy: MAE/MAPE onlaýn şekiller bilen standart nusgadaky oflayn standartyň arasynda.
Penjire alamatlary üçin Redis/Scylla, TTL; wagt we fallbacks.

8) Syn etmek, SLO we alerting

8. 1 SLI/SLO görkezmeleri

Gizlinlik: p95 ≤ 150 ms (şahsylaşdyrma), p99 ≤ 300 ms; RG/AML alert ≤ 5 s end-to-end.
Elýeterlilik: ≥ 99. 9%.
Täsir hatasy: ≤ 0. 5% 5xx; coverage ≥ 99%.
Drift: PSI fich/skora <bosagasy, ECE (kalibrlemek) durnukly.
Бизнес: uplift Net Revenue, fraud saved, time-to-intervene.

8. 2 Metrikler (Prometheus)

yaml
- http_request_duration_seconds{quantile="0. 95"}
- http_requests_total{code=~"5.."}
- model_inference_latency_ms_bucket
- feature_fetch_latency_ms_bucket
- model_score_distribution_bucket
- psi_feature_{name}
- expected_cost_live

yaml
- alert: HighP95Latency expr: histogram_quantile(0. 95, sum(rate(model_inference_latency_ms_bucket[5m])) by (le)) > 0. 15 for: 10m
- alert: DriftDetected expr: psi_feature_amount_base > 0. 25 for: 15m

Трейсинг: OpenTelemetry — span’ы `feature_fetch`, `score`, `postprocess`, `guardrail`.

9) RG/AML goragçylary we howpsuzlyk syýasaty

Pre-/Post-filter: gadagan edilen hereketleriň maskalary (görkeziş ýygylygy, cooldown, agressiw offerleriň gadagan edilmegi).
Policy Shielding: RG bosagasyndan ýokary tizlik → ýumşak interwensiýa/arakesme.
Audit: loging 'policy _ id', 'propensity', 'mask', 'decision', 'reason'.
PII we rezidentlik: ID ýerine bellikler, aýry-aýry şifrlemek açarlary we EEA/UK/BR klaster; sebitleýin join 'olaryň esassyz gadagan edilmegi.
Syrlar: KMS/CMK, Secret Manager; bloglarda/söwdalarda PII ýok.

10) Çözgütleriň kalibrlenmegi, çäkleri we syýasaty

Kalibrlemek (Platt/Isotonic) artefakt hökmünde.
expected cost bosagasy; reýestrde/fiç-baýdakda konfigurasiýa edilýän.
Howpsuzlyk kapslary: hereketleriň ýokarky/aşaky çäkleri, gabat gelmek üçin el bilen override.

11) Yza gaýdyp gelmek, pese gaçmak we DR

One-click rollback: marşruty öňki 'model _ version' -e geçirmek.
Runbook: ssenariýalar "gizlinlik ↑", "ýalňyşlyklar 5xx ↑", "süýşmek/kalibrlemek döwüldi", "daşarky fiç-üpjün ediji elýeterli däl".
Şowsuzlyklaryň izolýasiýasy: circuit breaker, retry/backoff, iň soňky tassyklanan çözgüt keşi.
DR: artefaktlaryň/sanawyň yzlary, "ýyly" sebite köpeltmek, maşklar.

python try:
features = fetch_features(timeout=30)
except TimeoutError:
features = last_known_good(user_id) # fallback

12) Cost-in engineeringenerçilik we öndürijilik

Wayoluň profili: çitler (30-60%), model (20-40%), tor/IO (10-30%).
Bahanyň peselmegi: gyzgyn fiçleri kesmek, repleýlere kwotalar, lightweight modelleri, INT8/FP16 (ýerlikli bolsa), lazy-postprocess.
RPS/CPU/latency üçin HPA, akym şekilinde state-size çäklendirme.
Chargeback: cost/request, cost/feature; bazarlar/toparlar üçin býudjetler.

13) Howpsuz çykarmalar we laýyklyk

Önüm taýýarlygynyň ölçegleri: SLO shadow-da ýaşyl, süýşme/syzma ýok, model kartoçkasy dolduryldy, fairness-slaýslar kadaly.
Düzgünleşdiriji: WORM-reliz arhiwi (agramlar, kalibrlemek, bosagalar, metrikler, synaglaryň ýazgylary), üýtgemeýän eksport hasabatlary.
DSAR/RTBF: keş/log/fich.

14) Çykarmazdan öň QA

Integrasiýa synaglary: API şertnamasy, surat shemalary, "boş/aşa" wakalar.
Ýükleýiş: p99, paýlanyş guýruklary, burst-traffik.
Ekwiwalentlik synagy online/offline.
Chaos-synaglar: fiç-kesşiň/bazanyň öçürilmegi, daşarky hyzmatlaryň wagtlary.

15) Konfigurasiýa mysallary

yaml
- match: [headers: {x-exp: "canary"}]
route:
- destination: {host: ml-score-v1-8-3, weight: 20}
- destination: {host: ml-score-v1-7-9, weight: 80}

python
@app. get("/healthz")
def health():
return {"ok": True, "model_version": "1. 8. 3", "registry_sig_ok": verify_signature()}

16) Amallar we RACI

R (Responsible): MLOps (serving/orkestr/gözegçilik), Data Eng (çeňňek/kesiş/şertnama), Data Science (modelleriň kartoçkalary/kalibrlemek/bosagalar).
A (Accountable): Head of Data / CDO.
C (Consulted): Compliance/DPO (PII/RG/AML/DSAR), Security (KMS/syrlar/audit), SRE (SLO/hadysalar), Finance (ROI/býudjetler).
I (Informed): Önüm/Marketing/Amallar/Goldaw.

17) Durmuşa geçirmegiň ýol kartasy

1. Model sanawy we imutable artefaktlar; FastAPI/gRPC-hyzmaty + K8s/Helm.

2. Şadow-r95/5xx/drift gözegçiligi bilen başlangyç, deňlik synagy.

3. Canary 10% → 50% → 100% awtoskript we alertler bilen.

4. Model kartoçkasy, kalibrlemek, ekspected-cost bosagasy we Guardrails v1.

• Ficha-kesh, circuit breakers, runbooks/DR-maşklar.
• RPS/latency, cost-dashboard we chargeback.
• Slaýs-monitoring fairness, WORM-arhiw relizleri.

• Serwing grafy (kandidatlar → re-rank), propensiti slate-loging.
• Köp sebit, aýratyn açarlary bolan rezidentlik (EEA/UK/BR).
• Awto-perekat/dreýf, hil/kalibrleme hasabatlarynyň awtogen.

18) Azyk taýýarlygynyň çek-sanawy

• Model kartoçkasy dolduryldy; maglumatlar/çyzgylar/kalibrlemek/bosagalar wersiýalanýar.
• Fiç şertnamalary we online/offline ekwiwalentlik synagy - ýaşyl.
• SLO: p95, 5xx, coverage - şadowda ýaşyl we 10% canary ≥ 24 sagat.
• Alertler we daşbordlar (gizlinlik/ýalňyşlyklar/drift/expected-cost) goşuldy.
• Guardrails RG/AML we çözgütleriň barlagy işjeň; PII/rezidentlik berjaý edildi.
• Bir-click rollback we runbook hadysalary synagdan geçirildi.
• Bahasy býudjetine laýyk gelýär; HPA we kesh sazlandy.

19) Anti-patternler we töwekgelçilikler

Sanawsyz we imutable artefaktlarsyz el bilen çykarmalar.
Ylalaşylmadyk online/offline fiçleri → önümdäki gapma-garşylyklar.
Shadow/canary → gizlin regressiýalaryň ýoklugy.
Bosagasy expected cost däl, kalibrlemek ýok.
Wagt/kesişsiz sinhronly daşarky lookups.
DR/rollback ýok, WORM arşiwi ýok.

20) Jemleýji

ML önümçiligi "modeliň kynçylygy" däl-de, in engineeringenerçilik platformasy: reýestr we wersiýalar, howpsuz çykmalar (shadow/canary/blue-green), düzgün-nyzam we kalibrlemek, syn etmek we SLO, RG/AML üçin garawullar we anyk yza gaýdyp gelmek meýilnamasy. Bu pleýbuka eýerip, gözegçilik edilýän gymmaty bilen işewürlik gymmatyny yzygiderli getirýän çalt, ygtybarly we oňat inferens alarsyňyz.