Real-time monitoring

(Section: Operations and Management)

1) Why real-time monitoring

• instant visibility of availability and delays (p50/p95/p99) of critical routes;
• Event integrity control (webhooks, payments, RTP/limits)
• financial security (egress/cost of 1k events, clearing/escrow);
• compliance (receipts, PII hygiene).

2) Architectural outline

1. Producers: services, SDKs, edge nodes, payment/content providers.

2. Ingest gateways: 'metrics/traces/logs/events' receivers with backpressure and quotas.

3. Bus/streaming: broker with participation (tenant/region/route), retention for replay.

4. Stream-processing: window aggregations (T + 5s/T + 1m), dedup, time normalization, SLI calculation.

5. Storages: time-series (RAM), OLAP (history), WORM logs (audit).

6. Analytics and alerting: SLO rules, statistical detectors, anomalous.

7. Dashboards and runes: UI for actions (pause/re-route/rollback/raise-limit).

• Data contracts for metrics/events (schemes, versions, validation).
• Outbox/CDC for guaranteed publishing of domain events.
• Idempotency and dedup by 'trace _ id/event _ id'.
• Clock sync: NTP/PTP, 'skew'correction, time waterfalls (event vs processing time).

3) Telemetry types and semantics

Metrics (SLI): p-percentile counters/gages/histograms.
Traces: end-to-end 'trace _ id/span _ id', bundle RPC↔sobytiya↔vebkhuki.
Logs: structured, with 'tenant _ id/region/version'.
Business events: `PaymentAuthorized`, `WebhookDelivered`, `RTPWindowClosed`.
Receipts: receipts/signatures (for finance/critical operations).

4) Time and windows

Types of time: event-time, ingest-time, processing-time.
Windows: sliding (5-30 s), toggle (1-5 min), with water retention (watermark) for late events.
Compactness: aggregate in a stream (histogram sketches) → store only the necessary percentile bins.

5) Normalization and data quality

Input validation: scheme/ranges/required fields; rejected - quarantined with reason label.
Deduplication: by '(event_id, producer, seq)'; store "seen-cache" in + KV memory.
Correction of metrics: against "double count" and "flatline" (sensors are silent).
Sampling: for high-QPS - adaptive, with an error; critical SLI - full.

6) SLI/SLO (reference)

North Star: E2E Success Rate at target p95 by region.

• Availability per-channel/region.
• p50/p95/p99 latency along key routes.
• Error-rate/Retry-rate.
• Webhook delivery success rate (% confirmed by receipts).
• Price/tax consistency ('quote = = checkout', ± 1 minor unit).
• Cost-SLI: cost of 1k events, egress/ingress per unit.

• Availability ≥ 99. 95% in the 28-day window.
• p95: showcase ≤ 120ms, quote/checkout ≤ 250ms.
• Webhooks are successful ≥ 99. 5 %/5-min window.
• Δ quote↔checkout = 0 (±1 minor unit).
• Reaction to P1 ≤ 10 min, MTTR ≤ 60 min.

7) Alerting and runes (auto-actions)

Levels: P1 (SLO failure/hopelessness), P2 (degradation), P3 (trend/risks).
Noise cancellation: dedup by 'trace _ id', correlation of causal chains.

• "PriceMismatch" → directory refresh, reconciliation 'fx _ version/tax _ rule _ version', compensation policy;
• WebhookLag → rearranging workers, increasing batch, prioritizing queues;
• "RTP Drift →" pause promo, check paytable/version, roll back profile;
• "Egress Surge" → enable compression/cache pinning/alternate route.
• Escalation: matrix 24 × 7, on-call rotation, channels (chat/call/SMS).

8) Dashboards (operational widgets)

Platform health: availability, p95/p99, error-rate, burn-down error-budget.
Integrations/webhooks: success, lag, doubles/idempotence, receipts.
Checkout/prices: vitrina↔checkout discrepancies, FX/Tax versions, refusal cases.
RTP/limits: theor. vs observed RTP, actuation of limits, exposure.
FinOps: cost per 1k, egress/ingress, budgets/cap-alerts.
Security/Compliance: SoD, JIT, MFA, PII requests, Crete signatures. operations.
Release/Flags: feature statuses, canary regions, link with incidents.

9) Multi-region and multi-tenant

Partitioning by 'tenant/region'.
Independent SLOs/quotas by region; restrictions of cross-regional alerts (so that a local failure does not "paint" the whole world).
Data confidence zones: PII/finance - only where allowed; in general dashboard - aggregates/hashes.

10) Security, privacy, provability

Ingest authentication: keys/mutual-TLS, rate-limits, packet signatures.
PII minimization: tokens instead of primitives, masks/hash identifiers.
Receipts: DSSE/signatures for financial/critical events.
WORM logs: immutable logs for audit, Merkle slices.
Access Control: RBAC/ABAC/ReBAC, JIT for sensitive panels.

11) Anomalous and correlations

Guardrails: static thresholds by SLI.
Stats: Shewhart/CUSUM/EWMA for trends.
ML/signals: seasonality/channels/ASN/providers; impact of releases/ficheflags.
Correlations: Associate incidents with releases, config changes, traffic spikes, promotions.

12) Performance and cost

Telemetry budget: cap per QPS/volume; rejection of "chatty" metrics.
Compression/aggregation: downsampling history (1s→10s→1min), store percentile sketches.
Egress control: local caches/aggregates, edge preprocessing.
Cost-aware alerts: a signal if the cost of/1k events or egress goes beyond the plan.

13) API Integrations and Contracts

'POST/ingest/metrics' (JSON/OTLP): authentication, quotas, schema/version.
'POST/ingest/events' (signed): dedup/TTL/nonce.
`GET /kpis? filters = region, tenant, route '- aggregates for UI.
'GET/traces/{ trace _ id} '- unwind the chain.
Вебхуки: `IncidentRaised`, `QuotaCapReached`, `PriceMismatch`, `WebhookLag`, `RTPDrift`.

14) Incident playbooks (short-form)

P1 Dostupnost↓: switch routing, enable circuit-breakers, reduce customer timeouts, emergency status post.
P1 Quote≠Checkout: freeze promo/price dynamics, cache force disability, FX/Tax version comparison, compensation.
P1 WebhookLag: increase workers/competitiveness, batch size, disable insignificant webhooks.
P2 RTP Drift: bonus pause, paytable/version verification, monitoring window extension, report.
P2 Egress Surge: compression, edge cache, moving part of the traffic, temporary quotas.

15) Quality metrics of monitoring itself

UI/API availability ≥ 99. 9%.
Freshness: update log ≤ 30 s for operational panels.
Completeness: ≥ 99. 5% of sources sent data to the window.
Correctness: discrepancy with reference standard ≤ 0. 1%.
MTTA/MTTR alert pipeline: P1 ≤ 1/10 min.

16) Implementation checklist

• Define North Star and SLI/SLO set by region/channel.
• Enter data contracts and schemas for all telemetry streams.
• Configure ingest with quotas, backpressure, and deduplication.
• Deploy bus/streaming and window aggregations with watermarks.
• Build time-series/OLAP/WORM and bill bundle.
• Start alerts + auto-runes, escalation matrix 24 × 7.
• Create dashboards by role: SRE/Product/FinOps/Compliance/Partners.
• Include PII minimization, signatures, and RBAC/ABAC/ReBAC.
• Enter FinOps metrics (cost/1k, egress, storage) and mouthguards.
• Hold GameDay: webhook lag, price out of sync, retray-burst, region failure.

17) Link to iGaming/fintech

RTP & Limits: control of observed RTP and limits in minutes/hours, alerts on "over/under pay."

Payments/disbursements: end-to-end tracing of authorizations, clearing and receipts; SLA PSP.
Affiliates: shipping conversions (webhooks) and disputes → escrow/reconciliation.
Promo: traffic spikes → queue protection and egress price; guardrails on budgets.

18) FAQ

Is real-time mandatory everywhere?
No, it isn't. "Hot" contours - seconds/minutes (incidents, payments, webhooks). Economics/analytics - minutes/hours.

How to deal with false alarms?
SLO-oriented conditions, aggregation and dedup by 'trace _ id', correlation with releases, threshold hysteresis.

Do I need to keep all the logs forever?
No, it isn't. WORM - for audit/critical threads only; the rest is downsampling/TTL.

Why is "quote≠checkout" found?
FX/Tax versions, cache disability, rounding. Treated with versions, SWR strategy and consistency tests.

Summary: Real-time monitoring is a discipline: strict data contracts, window calculations, normalized time, a bundle with receipts and SLO alerts, plus an action button in each widget. By doing it right, you are reducing MTTR, keeping the budget under control and confidently scaling the ecosystem by region and by tenant.