Real vaqt tahlillari

1) Vazifasi va biznes qiymati

• AML/Antifrod: depozitlarni tuzish, velocity-hujumlar, tavakkalchilik-tranzaksiyalar.
• Responsible Gaming (RG): limitlardan oshib ketish, xavf patternlari, o’zini istisno qilish.
• SRE/Operatsiyalar: SLA tanazzullarini erta aniqlash, xatolar portlashi, klaster qizib ketishi.
• Mahsulot va marketing: personallashtirish triggeri, missiyalar/kvestlar, real-time segmentatsiyasi.
• Operatsion hisobot: near-real-time GGR/NGR, zallar/provayderlar dashbordlari.

Maqsadli yo’nalishlar: p95 end-to-end 0. 5–5 с, completeness ≥ 99. 5%, foydalanish imkoniyati ≥ 99. 9%.

2) Etalon arxitekturasi

1. Ingest/Edge — `/events/batch` (HTTP/2/3), gRPC, OTel Collector; sxemalarni validatsiya qilish, anti-dubli, geo-yo’naltirish.
2. Voqealar shinasi - Kafka/Redpanda (’user _ id/tenant/market’, DLQ, retenshn 3-7 kun).
3. Stream-ishlov berish - Flink/Spark Structured Streaming/Beam: stateful-operatorlar, CEP, watermarks, allowed lateness, dedup.
4. Onlayn boyitish - Redis/Scylla/ClickHouse lookups (RG-limitlar, KYC, BIN → MCC, IP → Geo/ASN), taymaut va fallback bilan asinxron qo’ng’iroqlar.
5. Serving - ClickHouse/Pinot/Druid (operativ vitrinalar 1-5 daqiqa), Feature Store (onlayn belgilar), webhooks/tiketing/SOAR.
6. Lakehouse - uzoq muddatli konsolidatsiya, replay va taqqoslash uchun Bronze/Silver/Gold.
7. Kuzatilganlik - payplaynlar metrikasi, treysing (OTel), loglar, lineage va cost-dashbordlar.

3) Signallar va taksonomiya

To’lovlar:’payment. deposit/withdraw/chargeback`.
Oʻyin:’game. bet/payout’, sessiyalar.
Autentifikatsiya va xulq-atvor:’auth. login/failure`, device-switch, velocity.
Operatsion: latency, error-rate, podlarni qayta ishga tushirish, saturation.
Komplayens: sanksiya skrining, RG bayroqlari, DSAR tadbirlari.

Har bir turning egasi (domain owner), sxemasi, SLO yangiligi va late data siyosati mavjud.

4) Derazalar, watermarks va late data

Oynalar: tumbling , hopping (qoplash), session (harakatsizligi bo’yicha).
Watermark: «vaqt bo’yicha bilimlar» chegarasi (odatda 2-5 daqiqa).
Kechiktirilgan hodisalar: tuzatishlar chiqarilgunga qadar, bayroq’late = true’, DLQ kuchli kechikganda.

sql
SELECT user_id,
TUMBLE_START(event_time, INTERVAL '10' MINUTE) AS win_start,
COUNT() AS deposits_10m,
SUM(amount_base) AS sum_10m
FROM stream.payments
GROUP BY user_id, TUMBLE(event_time, INTERVAL '10' MINUTE);

5) CEP va stateful-agregatsiyalar

Ulanish:’user _ id’,’device _ id’,’payment. account_id`.
Holati: siljuvchi hisoblagichlar/summalar, dedup uchun bloom-filtrlar, TTL.
CEP-patternlar: structuring (<chegara, ≥ N marta, T oynasi uchun), device-switch, RG-fatigue.

python if cnt_deposits(last=10MIN) >= 3 and sum_deposits(last=10MIN) > THRESH and all(d.amount < REPORTING_THRESHOLD):
emit_alert("AML_STRUCTURING", user_id, snapshot())

6) Exactly-Once, tartib va idempotentlik

At-least-once shina bilan yetkazib berish + qayta ishlashda’event _ id’bo’yicha dedup (TTL 24-72 soat).
Tartib: kalitlar bo’yicha partiyalashtirish (mahalliy tartib kafolatlangan).
Sink: tranzaksion kommitalar (2-phase) yoki idempotent upsert/merge.
Outbox/Inbox: OLTP domen voqealarini tranzaksion nashr etish.

7) Online-boyitish va Feature Store

Lookup: RG-limitlar, KYC-maqomlar, BIN → MCC, IP → Geo/ASN, bozorlar/soliqlar, FX.
Asinxron qo’ng’iroqlar: sanksiya/PER API taymautlar bilan; xato boʻlganda -’unknown’+ retray/kesh.
Feature Store: kelishish online/offline; bitta transformatsiya kod bazasi.

8) Real-time vitrinalar va serving

ClickHouse/Pinot/Druid: sekundli/daqiqali agregatlar, materialized views, SLA kechikish uchun 1-5 daqiqa.
API/GraphQL: dashbordlar/vidjetlar uchun past latentlik.
Alertlar: vebxuki/Jira/SOAR (trace_id, last events).

sql
CREATE MATERIALIZED VIEW mv_ggr_1m
ENGINE = AggregatingMergeTree()
PARTITION BY toDate(event_time)
ORDER BY (toStartOfMinute(event_time), market, provider_id) AS
SELECT toStartOfMinute(event_time) AS ts_min,
market,
provider_id,
sumState(stake_base) AS s_stake,
sumState(payout_base) AS s_payout
FROM stream.game_events
GROUP BY ts_min, market, provider_id;

9) Metriklar, SLI/SLO va dashbordlar

• p95 ingest → alert ≤ 2 s (tanqidiy qoidalar), ≤ 5 s (boshqalar).
• Completeness T ≥ 99. 5%; Schema validity ≥ 99. 9%; Trace coverage ≥ 98%.
• Strim-servisdan foydalanish imkoniyati ≥ 99. 9%; late-ratio ≤ 1%.

• Partiyalar/topiklar bo’yicha lag; busy time operatorlar; davlat miqdori.
• «Hodisa → qoida → keys» hunisi, domenlar boʻyicha precision/recall.
• Issiqlik kartasi late/completeness; «issiq» kalitlar xaritasi.

10) Oqimli DQ (sifat)

Ingest-validatsiyalar: schema/enums/size-limits, anti-dubli.
Oqimda: completeness/dup-rate/late-ratio, oynalarning toʻgʻriligi (ikki marta hisobga olinmagan holda).
Reaksiya siyosati: critical → DLQ + pager; major/minor → teglash + hisobot.

yaml stream: payments rules:
- name: schema_valid type: schema severity: critical
- name: currency_whitelist type: in_set column: currency set: [EUR,USD,GBP,TRY,BRL]
- name: dedup_window type: unique keys: [event_id]
window_minutes: 1440

11) Maxfiylik, xavfsizlik va rezidentlik

PII-minimallashtirish: ID taxallusi, sezgir maydonlarni kamuflyaj qilish, PAN/IBAN tokenizatsiyasi.
Data residency: mintaqaviy konveyerlar (EEA/UK/BR), alohida KMS-kalitlar.
DSAR/RTBF: downstream vitrinalarida selektiv tahrirlash; Keys/hisobotlar uchun Legal Hold.
Audit: kirish/qoidalarni o’zgartirishning o’zgarmas loglari, relizlarni jurnallashtirish.

12) Iqtisodiyot va unumdorlik

Sharding/kalitlar: «issiq» kalitlardan qoching (salting/composite), partiyalar balansi.
Holat: TTL, compact snapshots, tyuning RocksDB/state backend.
Oldindan agregatsiya: shovqinli mavzular uchun dastlabki bosqichlarda reduce.
Sampling: faqat kritik boʻlmagan metriklar uchun (tranzaksiya/komplayens emas).
Chargeback: mavzular/joblar uchun byudjetlar, replay kvotalari va og’ir so’rovlar.

13) Jarayonlar va RACI

R: Streaming Platform (infra/relizlar), Domain Analytics (qoidalar/fichlar), MLOps (skoring/Feature Store).
A: Head of Data/Risk/Compliance domenlar bo’yicha.
C: DPO/Legal (PII/retention), SRE (SLO/hodisalar), Arxitektura.
I: Mahsulot, Qo’llab-quvvatlash, Marketing, Moliya.

14) Joriy etish yo’l xaritasi

1. Kafka/Redpanda + 2 tanqidiy topika (masalan,’payments’,’auth’).

2. Flink-job watermark, dedup va 1 CEP-qoidasi (AML yoki RG) bilan.

3. Tezkor vitrin ClickHouse/Pinot (1-5 daqiqa), dashbordlar lag/completeness.

4. Hodisa kanali (vebxuki/Jira), asosiy SLO va alertlar.

• Onlayn boyitish (Redis/Scylla), Feature Store, asinxron lookups.
• Qoidalarni kod, canary/A-B, oqimli DQ sifatida boshqarish.
• Konveyerlarni hududlashtirish, DSAR/RTBF protseduralari, keyslar uchun Legal Hold.

• Multi-region active-active, «replay & what-if» simulyatori, ostonalarni avto-kalibrlash.
• Gold-stream vitrinasi (GGR/RG/AML), near-real-time hisoboti.
• Cost-dashbordlar, chargeback, DR-mashqlar.

15) Misollar (parchalar)

sql
MATCH_RECOGNIZE (
PARTITION BY user_id
ORDER BY event_time
MEASURES
FIRST(A.device_id) AS d1,
LAST(B.device_id) AS d2,
COUNT()      AS cnt
PATTERN (A B+)
DEFINE
B AS B.device_id <> PREV(device_id) AND B.ip_asn <> PREV(ip_asn)
) MR

java if (seenStore.putIfAbsent(eventId, now()) == null) {
context.forward(event);
}

16) Sotishdan oldingi chek-varaq

• Registry, back-compat testlaridagi sxemalar/kontraktlar yashil rangda.
• watermark/allowed lateness, dedup va DLQ kiritilgan.
• SLO va alertlar (lag/late/dup/state size) moslashtirilgan.
• Keshlar va taymautlar bilan boyitish; fallback «unknown».
• RBAC/dual-control qoidalari/modellariga; oʻzgarishlar jurnali kiritilgan.
• Qoidalar/vitrinalar hujjatlari; runbook’va replay/qaytarish.

17) Tez-tez xatolar va ulardan qanday qochish mumkin

Ignor event-time: watermarkssiz metriklar «suzadi».
Dedup yo’q: yolg’on alerta, ikki tomonlama hisob.
Issiq kalitlar: notekis partiyalar → salting/resharding.
Issiq yoʻldagi sinxron tashqi API: faqat async + kesh.
Boshqarilmaydigan qiymat: oldindan agregatsiya qilish, TTL holati, kvotalar, cost-monitoring.
Simulyator yo’qligi: «replay» → regressiyasiz chiqishlar.

18) Jami

Real vaqt tahlillari - bu «tezkor BI» emas, balki kontraktlar, stateful-mantiq, CEP, watermarks, onlayn boyitish va qattiq SLO bilan boshqariladigan kontur. Ushbu amaliyotlarga amal qilib, platforma komplayens, mahsulot stsenariylari va nazorat qilinadigan qiymat bo’yicha operatsion barqarorlikni qo’llab-quvvatlagan holda aniq signallar va yechimlarni sekundlar ichida oladi.