ML modellarini joylashtirish

1) Roli va maqsadlari

Tarqatish = RG/AML/Legal va byudjetlarga rioya qilgan holda infensi mahsulot va operatsiyalarga ishonchli yetkazib berish.
Maqsadlar: past latentlik, yuqori ochiqlik, takrorlanuvchanlik, xavfsizlik va tezkor qaytaruvchanlik (rollback).

2) Serving arxitekturasi

2. 1 Pattern

Online (real-time): REST/gRPC, p95 personallashtirish uchun 50-150 ms; RG/AML-alertlar uchun 2-5 s ≤.
Near-real-time: mikrobatchi 1-5 min (OLAP-vitrinalar).
Batch/offline: Gold tungi vitrinalar, WORM-eksport regulyator uchun.
In-process scoring: light modelini xizmatga kiritish (past kechikish).
Serverless: noyob vazifalar uchun sovuq boshlash funksiyalari.

2. 2 Topologiyalar

Single Model Service → oson va tez.
Ensemble/Graph (router → preprocess → model → postprocess) → murakkab payplaynlar.
Sidecar Feature Fetcher → onlayn-chichlarni keshdan (Redis/Scylla) tortib oladi.

3) Model reyestri va versiyasi

Registry:’model _ id’,’version’,’stage = {Staging, Production, Archived}’, artefaktlar (og’irlik, preprotsessor, kalibrlash), talablar (CPU/GPU/xotira), model kartochkasi (ma’lumotlar, metriklar, xavflar, fairness).
Immutable artefaktlar: content-hash; Relizlarning WORM nusxalari.
Chiqish siyosati: faqat reyestr va deklarativ manifestlar orqali.

4) Konteynerlashtirish va qadoqlash

dockerfile
FROM python:3. 11-slim
ENV PYTHONUNBUFFERED=1
WORKDIR /app
COPY requirements. txt.
RUN pip install -r requirements. txt --no-cache-dir
COPY artifacts/./artifacts/
COPY src/./src/
CMD ["python", "src/serve. py"]

python serve. py from fastapi import FastAPI import joblib, time app = FastAPI()
model = joblib. load("artifacts/model. joblib")
scaler = joblib. load("artifacts/scaler. joblib")

@app. post("/score")
def score(payload: dict):
t0 = time. time()
x = preprocess(payload, scaler)
y = float(model. predict_proba([x])[0,1])
return {"score": y, "latency_ms": int((time. time()-t0)1000), "model_version": "1. 8. 3"}

5) Kubernetes/Helm va avtoskeyling

yaml apiVersion: apps/v1 kind: Deployment metadata: {name: ml-score, labels: {app: ml-score}}
spec:
replicas: 3 selector: {matchLabels: {app: ml-score}}
template:
metadata: {labels: {app: ml-score}}
spec:
containers:
- name: api image: registry/ml-score:1. 8. 3@sha256:...
ports: [{containerPort: 8080}]
resources:
requests: {cpu: "500m", memory: "512Mi"}
limits:  {cpu: "2",  memory: "2Gi"}
envFrom: [{secretRef: {name: ml-secrets}}]
readinessProbe: {httpGet: {path: /healthz, port: 8080}, periodSeconds: 5}
livenessProbe: {httpGet: {path: /livez,  port: 8080}, periodSeconds: 10}

yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: {name: ml-score-hpa}
spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: ml-score}
minReplicas: 3 maxReplicas: 50 metrics:
- type: Pods pods:
metric: {name: requests_per_second}
target: {type: AverageValue, averageValue: "15"}
- type: Resource resource: {name: cpu, target: {type: Utilization, averageUtilization: 70}}

6) Qaytarib olish strategiyasi

Shadow (qorong’u ishga tushirish): yangi model so’rovlarning nusxalarini ko’rib chiqadi, javoblar e’tiborsiz qoldiriladi; metrik/latentlik/qiymatni solishtirish.
Canary: 1-10% trafik → 25% → 50% → 100% yashil SLOda; degradatsiyada avtomatik ravishda to’xtatib qo’yish.
Blue-Green: parallel stakanlar; tezkor marshrutlash svitch.
Gradual Feature Flag: bozorlar/tenantlar/qurilmalar bo’yicha marshrutlash.
A/B/n: sequential testing bilan onlayn tajribalar.

yaml routes:
- match: {tenant: "EEA"} # feature-flags/rules to: {model: "1. 8. 3", weight: 20}
- match: {tenant: "EEA"}
to: {model: "1. 7. 9", weight: 80}

7) Fichi online/offline: ekvivalentlik

Yagona transformatsiyalar kutubxonasi va Feature Store (online + offline).
Ekvivalentlik testi: MAE/MAPE onlayn fichlar va etalon namunasidagi oflayn etalon o’rtasida.
Kesh fich: Redis/Scylla, oyna belgilari uchun TTL; taymautlar va fallbacks.

8) Kuzatish, SLO va alerting

8. 1 SLI/SLO mo’ljallari

Latentlik: p95 ≤ 150 ms (personallashtirish), p99 ≤ 300 ms; RG/AML alert ≤ 5 s end-to-end.
Foydalanish imkoniyati: 99 ≥. 9%.
Inferens xatosi: ≤ 0. 5% 5xx; coverage ≥ 99%.
Dreyf: PSI fich/skora <chegara, ECE (kalibrlash) barqaror.
Бизнес: uplift Net Revenue, fraud saved, time-to-intervene.

8. 2 Metrika (Prometheus)

yaml
- http_request_duration_seconds{quantile="0. 95"}
- http_requests_total{code=~"5.."}
- model_inference_latency_ms_bucket
- feature_fetch_latency_ms_bucket
- model_score_distribution_bucket
- psi_feature_{name}
- expected_cost_live

yaml
- alert: HighP95Latency expr: histogram_quantile(0. 95, sum(rate(model_inference_latency_ms_bucket[5m])) by (le)) > 0. 15 for: 10m
- alert: DriftDetected expr: psi_feature_amount_base > 0. 25 for: 15m

Трейсинг: OpenTelemetry — span’ы `feature_fetch`, `score`, `postprocess`, `guardrail`.

9) RG/AML guardrails va xavfsizlik siyosati

Pre-/Post-filter: taqiqlangan harakatlar niqoblari (ko’rsatuv chastotalari, cooldown, tajovuzkor offerlarni taqiqlash).
Policy Shielding: RG chegarasidan yuqori tezlik → yumshoq intervensiya/pauza.
Audit: loging’policy _ id’,’propensity’,’mask’,’decision’,’reason’.
PII va rezidentlik: ID o’rniga tokenlar, EEA/UK/BRdagi alohida shifrlash va klaster kalitlari; asossiz kross-mintaqaviy join’onlarni taqiqlash.
Sirlar: KMS/CMK, Secret Manager; log/treyslarda PII yo’q.

10) Yechimlarni kalibrlash, chegaralari va siyosati

Kalibrlash (Platt/Isotonic) artefakt sifatida.
expected cost bo’yicha chegara; reyestrda/fich-bayroqda konfiguratsiya qilinadigan.
Safety caps: harakatlarning yuqori/pastki chegaralari, komplayens uchun qoʻlda override.

11) Orqaga qaytish, degradatsiya va DR

One-click rollback: yo’nalishni oldingi’model _ version’ga o’tkazish.
Runbook: skriptlar «latentlik ↑», «xatolar 5xx ↑», «dreyf/kalibrlash buzilgan», «tashqi fich-provayder mavjud emas».
Muvaffaqiyatsizliklarni izolyatsiya qilish: circuit breaker, retry/backoff, oxirgi valid yechim kesh.
DR: artefaktlar/reyestr arxaplari, «iliq» mintaqaga replikatsiya qilish, mashqlar.

python try:
features = fetch_features(timeout=30)
except TimeoutError:
features = last_known_good(user_id) # fallback

12) Cost-injiniring va unumdorlik

Yo’lni profillash: chi (30-60%), model (20-40%), tarmoq/IO (10-30%).
Qiymatni pasaytirish: issiq fichlarni keshlash, repleylarga kvotalar, lightweight-modellar, INT8/FP16 (agar o’rinli bo’lsa), lazy-postprocess.
RPS/CPU/latency uchun HPA, strim-fich uchun state-size chegarasi.
Chargeback: cost/request, cost/feature; bozorlar/jamoalar uchun budjetlar.

13) Xavfsiz relizlar va muvofiqlik

Prod-tayyorlik mezonlari: SLO shadow da yashil, dreyf/oqish yo’q, model kartasi to’ldirilgan, fairness-slayslar normal.
Regulyator: WORM-reliz arxivi (og’irlik, kalibrlash, bostirma, metrika, test loglari), o’zgarmas eksport hisobotlari.
DSAR/RTBF: kesh/log/fich.

14) Ko’tarishdan oldin QA

Integratsiya testlari: API kontrakti, fich sxemalari, «bo’sh/haddan tashqari» keyslar.
Yuklama: p99, taqsimot dumlari, burst-trafik.
Online/offline ekvivalentlik testi.
Chaos-testlar: fich-kesh/bazani o’chirish, tashqi xizmatlarning taymautlari.

15) Konfiguratsiya namunalari

yaml
- match: [headers: {x-exp: "canary"}]
route:
- destination: {host: ml-score-v1-8-3, weight: 20}
- destination: {host: ml-score-v1-7-9, weight: 80}

python
@app. get("/healthz")
def health():
return {"ok": True, "model_version": "1. 8. 3", "registry_sig_ok": verify_signature()}

16) Jarayonlar va RACI

R (Responsible): MLOps (serving/orkestr/kuzatuv), Data Eng (fichi/kesh/kontraktlar), Data Science (model kartochkalari/kalibrlash/chegara).
A (Accountable): Head of Data / CDO.
C (Consulted): Compliance/DPO (PII/RG/AML/DSAR), Security (KMS/sirlar/audit), SRE (SLO/hodisalar), Finance (ROI/byudjetlar).
I (Informed): Mahsulot/Marketing/Operatsiyalar/Qo’llab-quvvatlash.

17) Joriy etish yo’l xaritasi

1. Model reyestri va immunutable artefaktlar; FastAPI/gRPC-servis + K8s/Helm.

2. Shadow-r95/5xx/dreyf monitoringi bilan ishga tushirish, fich ekvivalentlik testi.

3. Canary 10% → 50% → 100% otskript va alertlar bilan.

4. Model kartochkasi, kalibrlash, expected-cost chegara va Guardrails v1.

• Ficha-kesh, circuit breakers, runbooks/DR-mashqlar.
• RPS/latency, cost-dashboard va chargebackda avtoskeyling.
• Slays-monitoring fairness, WORM-relizlar arxivi.

• Serving grafigi (nomzodlar → re-rank), propensitni slate-loglash.
• Ko’p mintaqa, alohida kalitli rezidentlik (EEA/UK/BR).
• Avtotransport/dreyf bo’yicha qayta ishlangan, avtogen sifat/kalibrlash hisobotlari.

18) Oziq-ovqat tayyorgarligi chek-varaqasi

• Model kartochkasi to’ldirilgan; ma’lumotlar/fichlar/kalibrlash/chegaralar versiyalangan.
• Kontraktlar fich va ekvivalentlik testi online/offline - yashil.
• SLO: p95, 5xx, coverage - shadow da yashil va 10% canary ≥ 24 soat.
• Alertlar va dashbordlar (yashirin/xato/dreyf/expected-cost) kiritilgan.
• Guardrails RG/AML va yechimlar auditlari faol; PII/rezidentlikka rioya qilingan.
• One-click rollback va runbook hodisalari sinovdan o’tkazildi.
• Qiymati budjetga to’g "ri keladi; HPA va kesh oʻrnatilgan.

19) Anti-patternlar va xavflar

Reyestrsiz qo’lda chiqishlar va immutable artefaktlar.
Kelishilmagan online/offline fichlari → mahsulotdagi tafovutlar.
Shadow/canary → yashirin regressiyalar mavjud emas.
Chegara expected cost emas, kalibrlash yoʻq.
Taymaut/keshsiz sinxron tashqi lookups.
Hech qanday DR/rollback, hech qanday WORM arxivi mavjud emas.

20) Jami

ML ishlab chiqarish - bu «model muammosi» emas, balki muhandislik platformasi: reyestr va versiyalar, xavfsiz chiqishlar (shadow/canary/blue-green), fich va kalibrlash intizomi, kuzatuv va SLO, RG/AML uchun guardrails va aniq orqaga qaytish rejasi. Ushbu pleybukga amal qilib, siz tezkor, ishonchli va qulay inferens olasiz, bu esa nazorat qilinadigan qiymatda barqaror biznes qiymatini keltiradi.