Voqealarni loglash va izlash

Voqealarni loglash va izlash

1) Maqsad va ramka

Loglar va treyslar - kuzatish uchun poydevor.
Loglar «nima boʻldi» va «qanday kontekstda» deb javob berishadi.
Treyslar «qayerda va nima uchun sekin/xato» deb javob berishadi.

• Structured by default (JSON); Trace-first: har bir log’trace _ id ’/’ span _ id’ga bogʻlanadi.
• Minimal shovqin, maksimal signal: darajalar, sampling, anti-kardinallik.
• Xavfsizlik va maxfiylik: niqoblash, tahrir qilish, kirishni chegaralash.
• Loglar va hodisalarning versionlangan sxemalari.

2) Voqealar taksonomiyasi

1. Texnik loglar (runtime, xatolar, tarmoq taymautlari, retralar).

2. Biznes hodisalar (ro’yxatdan o’tish, depozit, stavka, xulosa, KYC bosqichi) - «pul» yo’llari bo’yicha oziq-ovqat tahlillari va hodisalar uchun yaroqlidir.

3. Audit (kim/qachon o’zgartirdi: konfigurlar, kirish joylari, bayroqlar, limitlar) - o’zgarmas jurnal.

4. Xavfsizlik (autentifikatsiya, imtiyozlar eskalatsiyasi, sanksiya/RER bayroqlari).

5. Infratuzilma (K8s events, autoscaling, HPA/VPA, tugunlar/disk/tarmoq).

Har bir oqim uchun - alohida retenshn, indeksatsiya va kirish qoidalari.

3) Tarkibiy log (JSON etaloni)

json
{
"ts": "2025-11-03T14:28:15.123Z",
"level": "ERROR",
"service": "payments-api",
"env": "prod",
"region": "eu-central-1",
"trace_id": "8a4f0c2e9b1f42d7",
"span_id": "c7d1f3a4b8b6e912",
"parent_span_id": "a1b2c3d4e5f60789",
"logger": "withdraw.handler",
"event": "psp_decline",
"msg": "PSP declined transaction",
"http": { "method": "POST", "route": "/withdraw", "status": 502, "latency_ms": 842 },
"user": { "tenant_id": "t_9f2", "user_key": "hash_0a7c", "vip_tier": 3 },
"payment": { "psp": "acme", "amount": 120.50, "currency": "EUR", "idempotency_key": "u123:wd:7845" },
"safe": true,         // пройдена проверка на секреты
"version": "1.14.2",     // версия сервиса (SemVer)
"build": "sha-1f2a3b4",
"kubernetes": { "pod": "payments-7cbdf", "node": "ip-10-0-2-41" }
}

Talablar: tekis sxema + domen bo’yicha qo’shimchalar, majburiy maydonlar (’ts, level, service, env, trace_id, msg’), son qiymatlari - satrlar bilan emas, sonlar bilan.

4) Darajalar, kardinallik va hajm

Darajalar:’DEBUG’(prodda emas),’INFO’(biznes-faktlar),’WARN’(anomaliyalar),’ERROR’(xatolar),’FATAL’(krashlar).
Kardinallik: ixtiyoriy kalitlardan/dinamik labellardan qoching. Hech qanday «id-kalitda».
Sampling log: rate-limit takrorlanuvchi xabarlar; ’DEBUG’ ni faqat scoped va vaqt boʻyicha yoqing (feature flag).
Idempotentlik:’idempotency _ key’manzilini yozib oling.

5) Maxfiylik va xavfsizlik

(’email’,’card’,’token’,’authorization’).
’user _ key’ xesh qiling, faqat kerakli kontekstni saqlang (mamlakat, KYC darajasi, VIP-tier).
Saqlash joylarini ajrating: issiq (tezkor qidiruv) va sovuq (PII/s qisqartirilgan kontekstsiz arxiv).
Audit - append-only, WORM-saqlash, kirish faqat least privilege printsipi bo’yicha.

6) Trassalash: standartlar va kontekst

W3C Trace Context:’traceparent ’/’ tracestate’sarlavhalari, plyus xavfsiz kalitlar uchun baggage (masalan,’tenant _ id’,’region’).
Metrik va treys aloqasi: Exemplars -’trace _ id’ni gistogrammalarning semple nuqtalariga uzating (RCA tezlashtiradi).
Sampling: asosiy sampling 1-5% + dinamik «xato/sekin p95» muammoli soʻrovlar uchun 100% gacha.
Links: Asinxron navbatlar/saglar uchun uyquni faqat’parent’orqali emas, balki’links’orqali bogʻlang.

7) Yig’ish va yo’naltirish

Agentlar: Fluent Bit/Vector log uchun; OpenTelemetry Collector’ga OTLP eksport qilish.
Collector: markaziy shlyuz (batch/transform/filter/routing).

App → (OTLP logs/traces/metrics) → OTel Collector
→ logs: redact → route(security    audit    tech    biz) → hot index / cold archive
→ traces: tail_sampling(errors    p95>threshold) → APM backend
→ metrics: Prometheus exporter (for SLO/alerts)

yaml processors:
batch: {}
attributes:
actions:
- key: env value: prod action: insert filter/logs:
logs:
include:
match_type: strict resource_attributes:
- key: service.name value: payments-api exporters:
otlp/traces: { endpoint: "apm:4317", tls: { insecure: true } }
loki: { endpoint: "http://loki:3100/loki/api/v1/push" }
prometheus: {}
service:
pipelines:
logs: { receivers: [otlp], processors: [attributes,batch], exporters: [loki] }
traces: { receivers: [otlp], processors: [batch], exporters: [otlp/traces] }
metrics: { receivers: [otlp], processors: [batch], exporters: [prometheus] }

8) Instrumentatsiya: SDK namunalari

8. 1 Node. js (Pino + OTel)

js import pino from "pino";
import { context, trace } from "@opentelemetry/api";

const logger = pino({ level: process.env.LOG_LEVEL          "info" });

function log(info) {
const span = trace.getSpan(context.active());
const base = span? { trace_id: span.spanContext().traceId, span_id: span.spanContext().spanId }: {};
logger.info({...base,...info });
}

// пример log({ event: "deposit.created", amount: 50, currency: "EUR", user: { user_key: "hash_0a7c" } });

8. 2 Java (SLF4J + OTel)

java
MDC.put("trace_id", Span.current().getSpanContext().getTraceId());
MDC.put("span_id", Span.current().getSpanContext().getSpanId());
log.info("psp_response status={} latency_ms={}", status, latency);

8. 3 Python (structlog + OTel)

python import structlog from opentelemetry import trace log = structlog.get_logger()

def log_json(event, kwargs):
span = trace.get_current_span()
ctx = {}
if span and span.get_span_context().is_valid:
ctx = {"trace_id": span.get_span_context().trace_id, "span_id": span.get_span_context().span_id}
log.msg(event=event, ctx, kwargs)

8. 4 NGINX → sarlavhalarni izlash

nginx proxy_set_header traceparent $http_traceparent;
proxy_set_header tracestate $http_tracestate;

9) Alertlar va avto-harakatlar uchun signal sifatida loglar

Xato patternlar (’psp _ decline’,’fraud _ flag’) SLO bilan bogʻlansin.
Pattern-rate alertlari: "5xx po/withdraw> 0. 10 m uchun 5%", "fraud_flag spike> + 200% bazaviy".
Avtomatik harakatlar:’withdrawals _ manual _ mode = true’logida bayroqlar platformasi orqali kill-switchni yoqing.

rate(count_over_time({service="payments-api", level="ERROR", event="psp_decline"}[5m])) > 5

10) Retenshn, indeksatsiya, saqlash

Issiq: 7-14 kun (tezkor tergov).
Issiq: 30-90 kun (trendlar, RCA).
Sovuq: 180-365 + (arxiv, audit) - siqish, arzon sinflar, ehtimol to’liq matnli qidiruvsiz.
Indeksatsiya: oʻrnatilgan kalitlar (’service, env, level, event, trace_id, user. tenant_id'), «hammasi ketma-ket» indeksiga taqiq.
Hodisa o’lchami bo’yicha limitlar (masalan, 32KB ≤), trim/pastki qismida: «storage ortiqcha - MTTR dushmani».

11) Audit va o’zgarmaslik

Audit voqealarini imzolar/xeshlar, server vaqti,’who/what/when/why’bilan alohida oqimda yozing.
«DEda 100% bonus bayrogʻini kim yoqdi?» - javob 1-2 so’rovda bo’lishi kerak.

json
{
"ts": "2025-11-03T14:00:00.000Z",
"actor": "alice@company",
"action": "feature_flag.update",
"target": "bonus.enable_vip",
"old": {"rollout": 10},
"new": {"rollout": 100},
"reason": "campaign_2311",
"ticket": "OPS-3481",
"trace_id": "cf12ab.."
}

12) Biznes-voqealar va ma’lumotlar modeli

• `event_type`, `event_id`, `occurred_at`, `actor`, `subject`, `amount`, `currency`, `status`, `idempotency_key`.
• Outbox va «at-least-once» dan foydalaning.

13) Kubernetes va pipeline loglari

Sidecar/DaemonSet agentlari (tarmoq tanaffuslarida).
Marshrutlash uchun pod izohlari (’log. type`, `retention. tier`).
K8s-nazoratchilar loglarini alohida yig’ing (klaster indeksi).

ini
[FILTER]
Name     modify
Match
Remove    authorization, password, card_number

14) Anti-patternlar

«Kerakli» satr loglari,’trace _ id’mavjud emas.
PII/sahifalardagi sirlar, to’liq payload dampalari.
Millionlab noyob kalitlar → «portlatilgan» indeksatsiya.
DEBUG proda 24/7.
Audit, xavfsizlik va texnloglarni bitta indeksga aralashtirish.
Arxivdan qayta tiklash testi va retenshn siyosati yoʻq.

15) Joriy etish chek-varaqasi (0-45 kun)

0-10 kun

Gateway/mijozlarda W3C Trace Context dasturini yoqish.
Ilova yozuvlarini JSON ga tarjima qilish,’trace _ id ’/’ span _ id’qoʻshish.
PII/sirlarni (agentda yashirish) taqiqlash, maydonlar ro’yxatini tasdiqlash.

11-25 kun

Quyidagi oqimlarni ajratish: tech/biz/audit/security/infra, retenshn va ACL.
OTel Collector’ni yoqish, xatolarni/sekin soʻrovlarni tail-sampling qilish.
«Log Rate/Error by route» + Jump-to-trace (Exemplars) dashbordlari.

26-45 kun

Voqealar shablonlariga ko’ra alertlar va SLO bilan bog’lanish.
Sovuq loglar uchun arxiv/tiklash (DR-test).
SIda loglar sxemalari linteri, biznes voqealari uchun kontrakt.

16) Etuklik metrikasi

’trace _ id’ soʻrovlari 95% ga ≥.
JSON log ulushi ≥ 99%.
via «jump-to-trace» topilgan hodisalar <15 min (p50) hal qilindi.
0 ta holat PII (sizib chiqish skaneri).
Retenshn barcha oqimlar bo’yicha bajarilgan (auditni avtomatik ravishda isbotlaymiz).

17) Ilovalar: mini-snippetlar

W3C traceparent generatsiyasi (psevdo)

txt traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

PromQL - loglar va SLO bogʻlamasi (misol)

high_error_logs = rate(log_events_total{service="payments-api",level="ERROR"}[5m])
5xx_rate = sum(rate(http_requests_total{service="payments-api",status=~"5.."}[5m])) / sum(rate(http_requests_total{service="payments-api"}[5m]))
alert if high_error_logs > 10 and 5xx_rate > 0.005

OpenAPI - korelatsiya sarlavhalari

yaml components:
parameters:
Traceparent:
name: traceparent in: header required: false schema: { type: string }

18) Xulosa

Logotip va traskirlashning kuchli konturi - bu bitimlar + intizom: tuzilmaviy JSON-loglar, yagona’trace _ id’, xavfsiz PII ishlov berish, oqimlar bo’yicha marshrutlash va retenshn, shuningdek SLO, alerting va qaytish bilan yaqin bog’lanish. «Matnlar to’planishidan» voqealar va trassalar shartnomalariga o’ting va prod-hodisalarni tashxislash tezkor, oldindan aytib bo’ladigan va tekshiriladigan bo’ladi.