Maʼlumotlarni turli zanjirlardan birlashtirish

(Bo’lim: Ekotizim va Tarmoq)

1) Nega qo’shilish kerak

• Haqiqatning yagona manbai (canonical facts).
• Reorglar va kechikishlarga chidamlilik: to’g "ri yakunlash va qayta hisoblash.
• Tarmoqlar va aktivlar o’rtasidagi metrikalarning taqqoslanishi.
• Audit va regulyatorlar uchun shaffof lineage va sifat nazorati.

2) Ma’lumotlar manbalari va klasslari

1. Oncheyn: bloklar, tranzaksiyalar, shartnomalar loglari, sarlavhalar, holatlar.
2. Ko’priklar/releyerlar: buyurtmalar, kvitansiyalar, dalillar, tugatish maqomi.
3. L2/DA qatlamlar: batchi, nashrlar, pruflar, bahsli derazalar.
4. PSP/KYC/KYB/AML: to’lov maqomlari, tekshirishlar, sanksiya hitlari.
5. Mahsulot hodisalari: onbording, depozitlar/to’lovlar, o’yin va xulq-atvor hodisalari.
6. Maʼlumotlar: tarmoqlar, aktivlar, decimals, chainId, manzillar, SDK versiyalari.

Har bir manba uchun: egasi, sxema, yangilanish orqasi, yakunlash oynasi, dalillar formati va SLO qayd etiladi.

3) Qo’shilish payplaynining arxitekturasi

Ingest (agentlar/indekserlar/webhook) → Raw/Bronze (o’zgarmas xomashyo) → Clean/Silver (normallashtirish va dedup) → Merge/Core/Gold (kanonik faktlar va aloqalar) → Marts (moliya/mahsulot/xavf/operatsiya) → Serve (OLAP/API/qidirish).
Asosiy xususiyatlar: idempotentlik, sxemalarni versiyalash, replay/backfill, late data handling.

4) Kanonik sxemalar (soddalashtirilgan)

4. 1 Voqealar (YAML)

yaml event:
id: uuid observed_at: timestamp # when saw event_at: timestamp # when happened (by source)
chain_id: string       # 'eth-mainnet'    'polygon'...
block_height: long tx_hash: string log_index: int type: string         # transfer    bridge. lock    bridge. mint...
status: string        # observed    confirmed    finalized    invalid src: string # address/peer-id/org _ id dst: string asset: string # canonical character (USDC)
amount: decimal usd_value: decimal # normalization at the rate on the meta observed_at: object # gas, fee, contract, sdk_version...
idempotency_key: string    # chainId    block    tx    logIndex    type proof_ref: string # proof/anchor reference

4. 2 O’tkazmalar va ko’priklar (SQL)

sql
CREATE TABLE bridge_transfers (
id TEXT PRIMARY KEY,
src_chain TEXT, dst_chain TEXT,
asset TEXT, amount NUMERIC,
created_at TIMESTAMPTZ,
finalized_at TIMESTAMPTZ,
status TEXT,          -- requested    inflight    finalized    failed    reversed src_tx TEXT, dst_tx TEXT,
proof_ref TEXT, meta JSONB
);

4. 3 Aktivlar/tarmoqlar ma’lumotnomasi (YAML)

yaml catalog:
assets:
- symbol: USDC decimals: { eth-mainnet: 6, polygon: 6 }
contracts: { eth-mainnet: "0xA0b8...", polygon: "0x2791..." }
networks:
- id: eth-mainnet k_confirmations: 12
- id: polygon k_confirmations: 256

5) Yakunlash, reorglar va maqomlar

Состояния: `observed → confirmed(K) → finalized → invalidated(reorg)` (+ `challenged` для optimistic).

• tarmoq/aktiv/tavakkalchilik bo’yicha K-tasdiqnomalar.
• Yirik summalar uchun Delayed Finalization.
• Reorg handling: avtomatik nogironlik va replay.
• Proof coverage: maqsadli SLO ≥ pruflar/ankerlar bilan yozuvlar ulushi.

6) Vaqt va valyutalarni normallashtirish

Vaqt: barcha timestampts UTC’da,’observed _ at’va’event _ at’ni saqlash.
FX/aktivlar narxlari:’observed _ at’(yoki’event _ at’- hisobot uchun, siyosatda belgilangan) kursi bo’yicha’usd _ value’ni qayta hisoblash.
Decimals/scale: solishtirish uchun miqdorlarni qatʼiy kanonlashtirish.
Hisobotlardagi vaqt mintaqalari: tanlashda (vitrin), core-da emas.

7) Idempotentlik va deduplikatsiya

• `idempotency_key = chainId|block_height|tx_hash|log_index|type`

• Bir nechta indekslardan takrorlash - idempotency_key boʻyicha upsert.
• Payload mojarosida - policy of truth (manba/versiya/vaqt ustuvorligi) ishga tushadi.
• Boboning derazasi 48-72 soatdan ≥ «adashgan» takrorlash uchun saqlanadi.

8) Entity Resolution (mohiyatlarni taqqoslash)

Manzillar → aktyorlar: hamyon/kontrakt → foydalanuvchi/tashkilot/rol.
Kross-zanjir aloqalari: hard-link (imzo/kyc), soft-link (xulq-atvor/grafa).
Taxalluslashtirish: barqaror PID/ORG_ID; PII ma’lumotlar nazoratchisida saqlanadi.

9) Qo’shilish qoidalari va ustuvorliklari (Policy)

1. Tarjimaning haqiqat manbai - «finalized» + pruf.
2. Agregatlar bo’yicha haqiqat manbai - «xomashyo» emas, «transfers’bridge _ transfers» jadvalining core.
3. Vaqt to’qnashuvi (event_at vs observed_at) - hisobot siyosati bo’yicha (moliya - event_at; operatsiya - observed_at).
4. Summalar/aktivlar to’qnashuvi - merjni to’xtatish va aktivlar katalogi solishtirilgunga qadar karantin.
5. Ko’prik bog’lamalari - har ikki tomonning kvitansiyalari (src/dst) + receipt pairing talab qilinadi.

10) Soxta so’rovlar va algoritmlar

10. 1 Voqealarni kanonik «operatsiya» ga kiritish

sql
WITH base AS (
SELECT e.,
CONCAT(e. chain_id,'    ',e. block_height,'    ',e. tx_hash,'    ',e. log_index,'    ',e. type) AS idem
FROM raw_events e
)
INSERT INTO core_events AS c (id, observed_at, event_at, chain_id, block_height,
tx_hash, log_index, type, status, src, dst, asset, amount, usd_value, meta, idempotency_key, proof_ref)
SELECT gen_random_uuid(), observed_at, event_at, chain_id, block_height,
tx_hash, log_index, type, status, src, dst, asset, amount, usd_value, meta, idem, proof_ref
FROM base
ON CONFLICT (idempotency_key) DO UPDATE
SET status = EXCLUDED. status,
usd_value = COALESCE(EXCLUDED. usd_value, core_events. usd_value),
proof_ref = COALESCE(EXCLUDED. proof_ref, core_events. proof_ref),
meta   = core_events. meta          EXCLUDED. meta;

10. 2 Ko’prik juftliklari o’yini (maqsad manbai)

sql
INSERT INTO bridge_transfers (id, src_chain, dst_chain, asset, amount, created_at, status, src_tx, proof_ref)
SELECT
CONCAT('br:', e. tx_hash) AS id,
e. chain_id, b. dst_chain, e. asset, e. amount, e. event_at, 'inflight', e. tx_hash, e. proof_ref
FROM core_events e
JOIN bridge_book b ON e. type='bridge. lock' AND e. asset=b. asset AND e. chain_id=b. src_chain
ON CONFLICT (id) DO NOTHING;

UPDATE bridge_transfers bt
SET finalized_at = e. event_at,
dst_tx    = e. tx_hash,
status    = 'finalized'
FROM core_events e
WHERE e. type='bridge. mint'
AND bt. status='inflight'
AND bt. asset=e. asset
AND bt. src_chain=bridge_book. src_chain
AND bt. dst_chain=bridge_book. dst_chain
AND abs(e. amount - bt. amount) < 1e-9;

10. 3 Reorglarga ishlov berish

sql
UPDATE core_events
SET status='invalidated'
WHERE chain_id=$1 AND block_height BETWEEN $2 AND $3
AND status IN ('observed','confirmed','finalized');

-- Reassembly of aggregates (example)
CALL recompute_materialized_views($1, $2, $3);

11) Sxemalar va evolyutsiyani boshqarish

Versionlash:’schema _ version’maʼlumotlar toʻplami shapkasida, migratsiyalar jurnalga yoziladi.
Moslik siyosati: Hodisalar uchun’BACKWARD’(faqat maydonlarni qoʻshish).
Manbalar bilan Data Contracts: CI shartnomalari testlari, sxemalar linterlari.

12) Ma’lumotlar sifati: SLI/SLO

• Freshness p95: lag ingest → Gold (min).
• Completeness%: oynada’finalized’ga yetgan yozuvlar ulushi.
• Correctness%: valid sxemalar/imzolar/pruflar.
• Proof Coverage%: pruflar/ankerlar bilan kanonik yozuvlar ulushi.
• Dedup Efficiency: idempotent bilan yutilgan dubllar ulushi.
• Reorg Handling Success%: to’g’ri nogironlik va replays.

SLO (taxminlar): Freshness ≤ 3 min (oqim )/15 min (batch); Completeness ≥ 99. 7%; Correctness ≥ 99. 9%; Proof Coverage ≥ 99. 0%; Reorg Success ≥ 99. 9%; Merge MTTR (hodisa) ≤ 30 daqiqa.

13) Dashbordlar (maketlar)

Merge Ops (реал-тайм/час): Freshness, Queue lag, Dedup rate, Finalized %, Reorg spikes, Error-budget burn.
Proof & Finality: proof coverage, p95 finality per chain, challenge/reorg события.
Catalog Health: aktivlar, decimals, SDK versiyalari meppinglarining tafovutlari.
Quality & Drift: completeness/correctness, schema drift, late data.
Finance Lens: GTV, Net Flow, zanjirlar/ko’priklar bo’yicha TVL (faqat’finalized’).

14) Konfiguratsiyalar (YAML)

Yakunlash oynalari

yaml finality:
eth-mainnet: { k: 12, delayed_for_usd_gt: 100000 }
polygon:   { k: 256 }
optimistic-L2:
k: 0 challenge_minutes: 20 delayed_for_usd_gt: 50000

Merj va ustuvorliklar siyosati

yaml merge_policy:
source_priority: [onchain, bridge, psp, product]
conflict:
time: { prefer: "event_at" }
amount: { action: "quarantine" }
proof_required_for: ["bridge_transfers", "payouts"]
quarantine_topics: ["asset_mismatch", "decimals_mismatch", "time_skew_gt_5m"]

Idempotentlik/dedup

yaml dedup:
key_template: "${chain_id}    ${block_height}    ${tx_hash}    ${log_index}    ${type}"
ttl_hours: 72

15) Maxfiylik va komplayens

PII-minimallashtirish: PID/ORG_ID, metrik/leybllarda PII taqiqlash.
Data residency: mintaqalarni segregatsiya qilish (EU/ROW), «tinch/yo’lda» shifrlash.
Olib tashlash huquqi: tasdiqlangan tombstone/redaction.
Audit: o’zgarmas jurnallar, ankering xeshlar, rollar bo’yicha kirishni tekshirish.

16) Operatsion reglamentlar

Har kuni: proof coverage taqqoslash, zanjirlar bo’yicha yakunlash, ko’priklar reyestri va -dreyf.
Har hafta: aktivlar/decimals katalogini taftish qilish, FX-normallashuvning to’g "riligi.
Har oyda: reorg/replay testlari, SLO tekshiruvi va ish unumdorligi stress testi.
Change Management: merge siyosatini oʻzgartirish uchun timelock, yechimlar jurnali.

17) Hodisalar Playbook

A. Rassinxron aktivlar/decimals

Tegishli aktivlarni to’xtatish, katalogni qaytarib olish, vitrinalarni qayta hisoblash, hisobot ≤ 24 soat

B. Proof Coverage qulashi

Merkling/ankeringni qayta ishga tushirish, loging darajasini oshirish, 100 keysni qoʻlda tanlash, hisobot.

C. Piki Reorg/Challenge

«k »/nizo oynasini ko’paytirish, yirik summalar uchun delayed finalizatsiyani kiritish, manfaatdor tomonlarni xabardor qilish.

D. Dubl/takrorlarni portlatish

TTL/kalit dedupini kuchaytirish, shovqinli manbalarni cheklash, karantin konturini yoqish.

E. vaqt dreyfi (time skew)

NTP/PTP sinxronizatsiyasi, oynalarni qayta hisoblash,’prefer: observed_at' siyosatining vaqtinchalik siljishi.

18) Joriy etish chek-varaqasi

1. Manbalar, yakuniy oynalar va dalillarni yozib oling.
2. Hodisalarning kanonik sxemasini va idempotentlik kalitini kiriting.
3. Quarantine kontur bilan dedup va merge siyosatini moslash.
4. Aktivlar/tarmoqlar reyestrini va FX-normallashtirishni ko’taring.
5. Replay/backfill va late data jarayonini amalga oshiring.
6. SLI/SLO va sifat dashbordlarini aniqlang.
7. Muntazam ankering va audit jurnallarini ishga tushiring.
8. Uchuvchini reorglar/ko’prik kechikishlari simulyatsiyalari bilan o’tkazing va MTTRni o’rnating.

19) Glossariy

Finality - holat/hodisaning qaytarilmasligi.
Reorg - bloklarning bir qismini bekor qilish bilan zanjirni qayta yig’ish.
Idempotency - qayta yetkazib berishga chidamlilik.
Proof Coverage - haqiqiy dalillar bilan yozuvlar ulushi.
Entity Resolution - yagona mohiyatdagi manzillar/akkauntlarni taqqoslash.
Delayed Finalization - yuqori xavfli summalar uchun agregatlarga kechiktirilgan qabul qilish.
Quarantine - ziddiyatli/shubhali yozuvlar uchun izolyatsiya qilingan oqim.

Xulosa: zanjirlararo ma’lumotlarni to’g "ri birlashtirish - bu boshqariladigan intizom: kanonik sxema, yakuniy va pruflar, qat’iy idempotentlik, shaffof qo’shilish siyosati va sifat kuzatilishi. Ushbu freymvorga amal qilib, ekotizim yagona, tekshiriladigan va barqaror ma’lumotlar qatlamiga ega bo’ladi - bu audit, tahlil va mahsulotlarni xavfsiz ko’paytirish uchun asos bo’ladi.