AI-paýlanyş we okatmagyň awtomatlaşdyrylmagy

1) Bellenilmegi we ýörelgeleri

Maksat: iň az wagt-to-value we töwekgelçilik/baha gözegçiligi bilen maglumatlary ygtybarly we köpeldip boljak → çip → modeller → çözgütler → seslenme.

• Pipeline-as-Code: Hemme zat (DAG, konfigi, synaglar, syýasatçylar) - Git-e, PR we rewyu arkaly.
• Determinism: Maglumatlaryň/koduň/konteýnerleriň/garaşlylyklaryň kesgitlenen görnüşleri.
• Separation of Concerns: DataOps, FeatureOps, TrainOps, DeployOps, MonitorOps.
• Guarded Automation: Biz awtomatlaşdyrýarys, ýöne hil, howpsuzlyk we laýyklyk bellikleri bilen.
• Privacy by Design: PII-ni azaltmak, rezidentlik, audit.

2) Konweýeriň gatlaklary we arhitekturasy

1. Ingest & Bronze: wakalaryň/batçalaryň ygtybarly kabul edilmegi (CDC, tekerler, retralar, DLQ).
2. Kümüş (kadalaşmak/baýlaşdyrmak): SCD, walýuta/wagt, arassalamak, dedup.
3. Gold (vitrinler): okuw/hasabat üçin predmet tablisalary we datasetler.
4. Feature Store: bitewi formulalar online/offline, wersiýalar we SLO.
5. Train & Validate: saýlawlary taýýarlamak, öwrenmek, kalibrlemek, baha bermek/barlag-geýtleri.
6. Registry & Promotion: modelleriň sanawy, hil kartoçkalary, mahabat syýasaty.
7. Serving: REST/gRPC/Batch, fiç-keşler, fiç-baýdaklar, canary/shadow.
8. Monitor & Feedback: SLI/SLO, süýşmek/kalibrlemek, onlaýn bellikler, awto-retrain.

3) Orkestrasiýa: DAG patternleri

Daily CT (D + 1): gijeki maglumat aýlawy → surat → okuw → tassyklama → reýestre dalaşgär.
"Event-Driven Retrain": PSI/ECE/expected-cost ýa-da shemalaryň çykarylyşy boýunça trigger.
Rolling Windows: "süýşýän penjire" maglumatlary bilen weekly/monthly gaýtadan taýýarlamak.
Blue/Green Artifacts: ähli artefaktlar immutabel (hash), paralel wersiýalar.
Dual-write v1/v2: iki gezek ýazmak we ekwiwalentligi deňeşdirmek arkaly shemalaryň/şekiliň göçmegi.

python with DAG("ct_daily", schedule="@daily", start_date=..., catchup=False) as dag:
bronze = BashOperator(task_id="ingest_cdc", bash_command="ingest.sh")
silver = BashOperator(task_id="silver_norm", bash_command="dbt run --models silver")
gold  = BashOperator(task_id="gold_marts", bash_command="dbt run --models gold")
feats = BashOperator(task_id="feature_store_publish", bash_command="features publish")
ds   = BashOperator(task_id="build_dataset", bash_command="dataset build --asof {{ ds }}")
train = BashOperator(task_id="train", bash_command="trainer run --config conf.yaml")
eval  = BashOperator(task_id="evaluate", bash_command="eval run --gate conf/gates.yaml")
reg  = BashOperator(task_id="register", bash_command="registry add --stage Staging")
bronze >> silver >> gold >> feats >> ds >> train >> eval >> reg

4) Maglumatlar toplumy we nusgalary

"Point-in-time join" we "gelejeksiz".
Bazarlar/tenantlar/wagt, holdout we syzmak üçin "gap" boýunça stratifikasiýa.
Wersiýalaşdyrmak: 'data _ version', 'logic _ version', 'asof _ date'; WORM snapshotlar.

5) Feature Store we ekwiwalentlik online/offline

Bitewi aýratynlyk (ady, formulasy, eýesi, SLO, synaglary).
Onlaýn = oflayn: üýtgeşmeleriň umumy kody; ekwiwalentlik synagy (MAE/MAPE).
TTL we nagt: penjireler 10m/1h/1d; taýmautlar/retralar; "last_known_good".

yaml name: bets_sum_7d owner: ml-risk offline: {source: silver.fact_bets, window: "[-7d,0)"}
online: {compute: "streaming_window: 7d", ttl: "10m"}
tests:
- compare_online_offline_max_abs_diff: 0.5 slo: {latency_ms_p95: 20, availability: 0.999}

6) Okuwy awtomatlaşdyrmak (CT) we hil geýtleri

CT-sikl: taýýarlyk → okuw → kalibrlemek → baha bermek → dalaşgäri hasaba almak.

• Off-line: PR-AUC ≥ bençmark − δ; ECE ≤ 0. 05; çäk ≤ expected-cost.
• Slice/Fairness: islendik slaýsda metrleriň düşmegi ≤ Y%; disparate impact kadaly.
• Fiçiň ekwiwalentligi: ok.
• Bahasy: ≤ býujetiň wagty/serişdeleri.

yaml gates:
pr_auc_min: 0.42 ece_max: 0.05 expected_cost_delta_max: 0.0 slice_drop_max_pct: 10 features_equivalence_p95_abs_diff_max: 0.5

7) Modelleriň we mahabatlaryň sanawy

Model kartoçkasy: maglumatlar, penjireler, çyzgylar, metrikler off/online, kalibrlemek, töwekgelçilikler, eýesi.
Stages: `Staging → Production → Archived`; diňe barlanan geýtlar arkaly mahabatlandyrylýar.
Yzyna gaýtarmak syýasaty: Soňky önümçilik wersiýalarynyň N ≥ saklamak; one-click rollback.

8) CI/CD/CT: nädip birikdirmeli

CI (kod/synaglar): birlik/integrasiýa/şertnama synaglary, linterler, howpsuzlyk-skanerler.
CD (serwing): Docker/K8s/Helm/ficha-baýdaklar, canary/shadow/blue-green.
CT (maglumatlar/okuw): meýilnama/wakalar boýunça orkestrator; artefaktlar → reýestr.
Promotion Gates: ýaşyl SLO-da awto-goýberiş (canary ≥ X sagatda).

9) Köp kärende we rezidentlik

Tenantlar/sebitler: izolirlenen paýlaýnlar we şifrlemek açarlary (EEA/UK/BR); sebitara join 'olaryň esassyz gadagan edilmegi.
Syrlar: KMS/CMK, Secret Manager; bloglarda bellikli ID.

DSAR/RTBF syýasaty: şekillerde we ýazgylarda hasaplap bolýan proýeksiýalar we saýlap redaktirlemeler; "Legal Hold"

10) Gözegçilik → seslenme → retrain

SLI/SLO: latency p95/p99, 5xx, coverage, cost/request; PSI/KL, ECE, ekspected-cost.
Onlaýn bellikler: proksi (sagat/gün) we tussag edilenler (D + 7/D + 30/D + 90).
Awto-hereketler: recalibration/threshold update → shadow retrain → canary → promotion.
Runbooks: zaýalanma ssenarileri (süýşmek, kalibrlemek, fiç-kesh, üpjün edijiler).

11) Howpsuzlyk, RG/AML we çözgüt syýasaty

Guardrails: pre/post-filter, caps ýygylyk, cooldown, gadaganlyk sanawlary.
Policy Shielding: model → çözgüt → süzgüç syýasaty → hereket.
Audit: 'model _ id/version', 'feature _ version', 'threshold', 'policy _ id', sebäpler.
WORM-arhiw: relizler, hil hasabatlary, synag/mahabat žurnallary.

12) Bahasy we öndürijiligi

Wayoluň profili: çitler (30-60%), inferens (20-40%), IO/tor.
Cost-daşbordlar: cost/request, cost/feature, GPU/CPU-sagatlar, small-files.
Optimizasiýa: oflayn agyr şekilleriň materiallaşdyrylmagy, gyzgyn penjireleriň kesesi, INT8/FP16, repleýler/backfill üçin kwotalar.
Çargeback: Býudjeti toparlar/bazarlar boýunça paýlaýarys, "gymmat" hilelere gözegçilik edýäris.

13) Mysallar (bölekler)

yaml apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: {name: ct-daily}
spec:
entrypoint: pipeline templates:
- name: pipeline dag:
tasks:
- name: gold template: task arguments: {parameters: [{name: cmd, value: "dbt run --models gold"}]}
- name: features dependencies: [gold]
template: task arguments: {parameters: [{name: cmd, value: "features publish"}]}
- name: train dependencies: [features]
template: task arguments: {parameters: [{name: cmd, value: "trainer run --config conf.yaml"}]}
- name: eval dependencies: [train]
template: task arguments: {parameters: [{name: cmd, value: "eval run --gate conf/gates.yaml"}]}
- name: task inputs: {parameters: [{name: cmd}]}
container: {image: "ml/ct:latest", command: ["/bin/bash","-lc"], args: ["{{inputs.parameters.cmd}}"]}

python ok = (pr_auc >= gate.pr_auc_min and ece <= gate.ece_max and expected_cost_delta <= gate.expected_cost_delta_max and slice_drop_pct <= gate.slice_drop_max_pct and features_equivalence_p95_abs_diff <= gate.features_equivalence_p95_abs_diff_max)
exit(0 if ok else 1)

yaml promotion:
require:
- offline_gates_passed
- canary_online_hours >= 24
- slo_green: [latency_p95, error_rate, coverage]
- drift_warn_rate <= 5%

14) Amallar we RACI

• Data Eng — Ingest/Silver/Gold, Feature Store, CDC/Backfill;
• Data Science - nusgalar/okuw/kalibrlemek/geýt;
• MLOps - orkestr/reýestr/serwing/syn ediliş.
• A (Accountable): Head of Data / CDO.
• C (Consulted): Compliance/DPO (PII/RG/AML/DSAR), Security (KMS/audit), SRE (SLO/bahasy), Finance (býudjetler/ROI), Product.
• I (Informed): Marketing/Amallar/Goldaw.

15) Durmuşa geçirmegiň ýol kartasy

1. DAG «daily CT»: Bronze→Silver→Gold→Feature Store→Train→Eval→Registry(Staging).

2. Feature Store v1 we online/offline ekwiwalentlik synagy.

3. Hil geýtleri (PR-AUC/ECE/expected-cost/slice).

4. Modelleriň sanawy, kartoçka we WORM-relizleriň arhiwi.

• Awto-recalibration/threshold update, canary-promotion by online SLO.
• Dreýf boýunça event-driven retrain; göçmek üçin dual-write v1/v2.
• Cost-daşbordlar we backfill/repleýler üçin kwotalar; multi-tenant izolýasiýa.

• Slaýtlar we awto-hasabat boýunça fairness syýasaty.
• Aýry-aýry açarlary bolan köp sebitli rezidentlik (EEA/UK/BR).
• Awto-retreýn meýilnama we wakalar boýunça, paýlaýynlaryň awtogen dokumentleri.

16) Azyk taýýarlygynyň çek-sanawy

• Pipeline-as-Code в Git; CI synaglary (birlik/integrasiýa/şertnamalar/howpsuzlyk).
• Bronze/Silver/Gold we Feature Store durnukly; ekwiwalentligi ýaşyl.
• Oflayn oýunlar geçdi; modeliň kartoçkasy dolduryldy; WORM arhiwi döredildi.
• Canary ≥ 24 s ýaşyl SLO; rollback we kill-switch düwmesi işleýär.
• Drift/ECE/expected-cost we onlaýn belliklere gözegçilik ediň.
• PII/rezidentlik/DSAR/RTBF/Legal Hold berjaý edildi; audit düzüldi.
• Býudjetdäki bahasy; nagt pul/kwotalar/çitler we repleýler üçin çäkler işjeň.

17) Anti-patternler we töwekgelçilikler

Orkestratordan daşarda "bir gezek ulanylýan" el ädimleri; Git taryhy ýok.
Geýtsiz we kartoçkasyz okamak; "el bilen" mahabatlandyrylýar.
Utgaşdyrylmadyk online/offline fiçleri → önümdäki gapma-garşylyklar.
Dreýf/kalibrlemek ignory/expected-cost; diňe ROC-AUC "görnüş üçin".
Rezidentligiň ýoklugy/PII-syýasat; "çig" şahsyýetnamany ýazmak.
Çäksiz backfill/repliterler → bahanyň partlamagy we SLA-a täsiri.

18) Jemleýji

AI-paýlaýynlar noutbuklaryň toplumy däl-de, gymmatlygyň konweýeridir. Maglumat gatlaklaryny resmileşdiriň, "Feature Store" we "CT/CI/CD", hil we howpsuzlyk goşuň, "retrain" -ni awtomatlaşdyryň, onlaýn/awtonom we aç-açan ykdysadyýeti saklaň. Şeýlelik bilen, bazarlar we wagt boýunça masştably çalt, öňünden aýdyp boljak we "maglumatlar → model → effekt" siklini alarsyňyz.