Observability and telemetry

(Section: Technology and Infrastructure)

Brief Summary

Observability is the ability to respond to "why does it work like this?" without the release of new builds. In iGaming, this is critical: peak tournaments, payment peaks, multi-regionality and responsible gambling/PII requirements. Basis - metrics, logs, traces, united by common identifiers and standards (OpenTelemetry), with SLO contracts, noise-resistant alert and cost control.

1) Observability framework: what it consists of

Metrics (numbers by time): RED/USE, business KPI, SLI. Stored in TSDB.
Logs (events in text/JSON): audit, errors, business facts, security.
Traces (spans): request path through services, latencies, causes of delays.
Profiling: CPU/memory/eBPF streams, heap/lock content.
RUM and synthetics: real users (web/app) + robot checks.
Telemetry catalog: schemas, PII policies, retention periods, cost tags.

2) Signal taxonomy and principles

RED для API: Rate, Errors, Duration.
USE for infrastructure: Utilization, Saturation, Errors (CPU, disks, network, queues).
SLI/SLO: measurable indicators (e.g. successful requests/all, p95 latency), accessibility goals (e.g. "99. 9% in 30 days"), error budget → process triggers.
High-cardinality wisely: labels should be useful in cuts (region/tenant/provider), but not blow up TSDB.

3) Standards and end-to-end correlation

OpenTelemetry (OTel): a single SDK/protocol for metrics, logs and traces.
Identifiers: 'trace _ id', 'span _ id', 'correlation _ id', 'player _ id' (pseudonymized), 'payment _ route'.
Flow ID: ingress gateway → all microservices → payments/PSP → queues/jobs → logs/metrics/spans.

Example: Correlation Headers

traceparent: 00-<trace_id>-<span_id>-01 x-request-id: <correlation_id>

4) Metrics: what and how we measure

Naming/Labels

`service="payments-api"`, `env="prod"`, `region="eu-west"`, `tenant`, `provider="pspX"`.

Prometheus Examples

prometheus
RED http_requests_total{service="api",route="/deposit",method="POST",status="200"}
http_request_duration_seconds_bucket{service="api",le="0. 25",route="/deposit"} 1234 http_request_errors_total{service="api",route="/deposit"}

USE cpu_utilization_ratio{node="n1"} 0. 71 queue_depth{queue="withdrawals"} 128

Бизнес payments_success_total{psp="X",currency="EUR"} 4521 payment_conversion_ratio{route="pspX"} 0. 948

Histograms and exemplars

Store latency histograms (native-histograms/ β uckets) and bind exemplar with 'trace _ id' to jump from the "slow bucket" to a specific track.

5) Logs: structured and safe

Only JSON (no "free-form" in the prod).
Поля: `timestamp`, `severity`, `service`, `trace_id`, `correlation_id`, `player_id_hash`, `event`, `amount`, `currency`, `ip_hash`.
PII masking/hashing, separate indexes/retention for sensitive.
Log pipelines: parsing → normalization → enrichment (geo/ASN) → PII editing → indexing.

Example of a JSON event

json
{
"ts":"2025-11-05T10:42:31Z",
"sev":"ERROR",
"service":"payments-api",
"event":"psp_timeout",
"trace_id":"9c5e...e2",
"route":"pspX",
"duration_ms": 3100,
"attempt":2,
"player_id_hash":"p:1b7f...",
"pii_redacted":true
}

6) Traces: where time is lost

Spans: input request, provider calls (PSP/game providers), database/cache, interservice RPCs.
Attributes: 'db. system`, `net. peer. name`, `messaging. system`, `psp. route`, `game. provider`.

• head-based for volume,
• tail-based (by conditions: errors, p95 +, VIP-segment),
• guaranteed-keep for payment/PII-critical.

7) Observability of the front and mobile

RUM: TTFB, FCP/LCP/CLS/INP, JS errors, networks and SPA routing.
Crash reports: symbolism, deobfuscation, build version, device/OS.
Synthetics: entry/deposit/rate scenarios; geodistributed checks.

8) SLO, SLI and error budget

SLO Example (Pseudo-YAML)

yaml service: payments-api sli:
- name: availability expr: sum(rate(http_requests_total{status=~"2..    3.."}[5m]))
/ sum(rate(http_requests_total[5m]))
- name: latency_p95 expr: histogram_quantile(0. 95, rate(http_request_duration_seconds_bucket[5m]))
targets:
availability: "99. 9%/30d"
latency_p95: "<=250ms/30d"
error_budget_policy:
fast_burn: 5% for 1h -> page, freeze deploy slow_burn: 20% for 24h -> incident, improvement plan

Alerting by error budget, not by "every metric."

Freeze procedures for budget combustion: limit releases/canaries.

9) Alerting without noise

Multi-window, multi-burn rules: short/long window.
Deduplication/rooting: by service/region/criticality in on-call.
Runbook URL and context autocollection (latest dispatches, config changes, dependency graph).
Quiet hours and suppression during scheduled work.

Example rule (PromQL idea)

promql alert: PaymentsSLOFastBurn expr: slo_error_rate_5m > 2 slo_budget_rate for: 15m labels: { severity="page", service="payments-api" }
annotations:
summary: "SLO fast burn"
runbook: "https://runbooks/payments/slo"

10) Profiling and eBPF

eBPF/profilers: CPU/alloc flame graphs, I/O latency, network drops, Syscall anomalies.
Useful for p99 bottlenecks, jitter and rare freezes.

11) Business observability (product & risk)

Finance/monetization: deposit conversion, TTW (time-to-wallet), author ./settle, cancellations/chargebacks.
Game activity: retention/streak, share of live bets, "stickiness" of providers.
Antifraud/abuse: speed of action, device/IP matches, correlations.
RG indicators: long sessions, "dogon," steak growth.
Business metrics are correlated with technical metrics and annotation events.

12) Safety, PII and compliance

Data-zones: dataset/log tags ('pii = true', 'region = EU').
Masking before indexing, aliasing of identifiers.
WORM stores for audit; role-based log access.
Retention periods: different for techlogs/audits/businesses.
Prohibition of raw secrets in the logs; scan checks in CI.

13) Value Management (FinOps)

Cardinality limit: careful with 'user _ id', 'session _ id'.
Participation/retention: hot (7-14 days), warm (30-90), cold (archive).
Tail-based sampling and downsampling metrics.

Billing by tags' team ',' service ',' tenant ': reports "who burns observability."

14) Tools (reference stack)

Metrics: Prometheus/lake for metrics, dashboards Grafana.
Logs: Loki/ELK; ingestion rules, reduction/parsing.
Trails: Tempo/Jaeger/OTel collectors; exemplars-links from metrics.
Synthetics: Blackbox exporter, browser robots.
Alerting: Alertmanager/chat integration, on-call rotation.
Profiling: eBPF/continuous profiling.

15) Examples: Implement foundation quickly

python from prometheus_client import Counter, Histogram, start_http_server reqs = Counter('http_requests_total','', ['route','method','status'])
lat = Histogram('http_request_duration_seconds','', ['route'])
def handle(req):
with lat. labels(route=req. route). time():
status = app(req)
reqs. labels(route=req. route,method=req. method,status=str(status)). inc()

go tid:= ctx. Value("trace_id")
logger = logger. With("trace_id", tid)
logger. Info("deposit-accepted", "amount", amt, "route", route)

prometheus http_request_duration_seconds_bucket{..., le="0. 25"} 1023 # exemplar: trace_id=9c5e...

16) Processes and operating system

Unified dictionary of metrics/labels (naming-guide) and dashboard template.
Release-annotations automatically in columns.
Incidents: card, timeline, RCA no charges, action items.
Training alarms ("game-day"): simulated drops, PSP delays, cache overheating.
Runbooks: step-by-step instructions and auto-links from alerts.

17) Maturity checklist

1. OTel SDK/collector → single export of metrics/logs/trails.
2. RED/USE covers all services + SLI/SLO by key API.
3. Correlation 'trace _ id' ⇄ logs ⇄ metrics (exemplars, jump-links).
4. Alerts on budget errors with multi-burn and runabook links.

5. RUM + synthetics for "deposit/rate/withdrawal."

6. Profiling (eBPF) on the white list sale.
7. PII policies: masking, zones, access, retention periods.
8. Financial report on telemetry cost ('team/service' tags).
9. "Readiness for peak load": test plan, warm up caches, alert templates.
10. Regular RCAs and SLO/threshold revisions.

18) Antipatterns

Logs "sheets" without structure and 'trace _ id'.
Alerts for each metric → alert FAT.
Histograms without correct buckets → "flat" p95.
The unlimited cardinality of labels → an explosion of value.
The lack of RUM/synthetics is "all ok," but the user is not.
Mixing PII with technologists, indefinite retention.

Telemetry isolation from business KPIs - "latency is falling, revenue too."

Summary

Strong observability is a common language between product, SRE, security and payments. By connecting metrics, logs, tracks for OTel, introducing SLO with a budget of errors, making alerting smart and cost manageable, you get a system that notices problems earlier, recovers faster and predictably passes traffic peaks and tournament loads.