Prometheus: Metrleri ýygnamak

(Bölüm: Tehnologiýalar we infrastruktura)

Gysgaça gysgaça

Prometheus - wagt ölçegleri üçin senagat standartydyr: HTTP boýunça nyşanlary gysýar, TSDB-de seriýalary saklaýar, PromQL-de agregatlara we Alertmanager arkaly alertleriň triggeritine ynanýar. iGaming üçin bu SLO çemeleşmesiniň (RED/USE, tölegleriň iş ölçegleri), p95/p99 çalt diagnostika we awtomatiki çözgütleriň (freeze/rollback) esasyny düzýär.

1) Maglumatlaryň nusgasy we kardinallygy

Metrika:' name {label1 =" v1", label2 =" v2"} value @ timestamp'.
Kardinallyk = bellikleriň ähli täsin toplumlarynyň kuwwatlyklarynyň köpelmegi; bahanyň esasy faktory.

• базовые: `service`, `env`, `region`, `instance`, `pod`, `container`, `version`;
• domenler: 'route', 'psp', 'tenant' (seresap boluň!), 'game _ provider'.
• 'user _ id', 'session _ id', tötänleýin/ýokary kardinal bahalary goýup bolmaz.

2) Metrikleriň görnüşleri

Counter - diňe ösýär (mysal üçin 'http _ requests _ total').
Gauge - momential bahalar (mysal üçin 'queue _ depth').
Histogram/Summary - gizlinlik paýlanyşy. Prodda - Histogram ('histogram _ quantile ()' we exemplars goldawy bilen).
Native Histograms - üýtgeýän baketler, takyklygy ýokarlandyrýar we ululygy tygşytlaýar (elýeterli ýerlerde goşuň).

go var httpLatency = prometheus. NewHistogramVec(
prometheus. HistogramOpts{
Name:  "http_request_duration_seconds",
Help:  "HTTP latency",
Buckets: prometheus. DefBuckets ,//or custom
},
[]string{"route","method"},
)

3) Eksportçylar we näme ölçemek

Hyzmat: Koduňyz (Go/Java/Node/Python üçin SDK), RED-metrika API, biznes-metrika (töleg öwrülişigi).
Ulgam: node_exporter, cAdvisor/kubelet.
Daşarky: DB/keşler (mysqld_exporter, postgres_exporter, redis_exporter), NGINX/HAProxy, Kafka/RabbitMQ.
OTel-metrikler: OpenTelemetry Collector → Prometheus Remote Write ýa-da Prometheus-receiver → arkaly umumy stek.

4) Skrape we relabel: nyşanlary nädip birikdirmeli

Esasy 'prometheus. yml`

yaml global:
scrape_interval: 15s evaluation_interval: 15s external_labels:
env: "prod"
region: "eu-west"

scrape_configs:
- job_name: 'node'
static_configs:
- targets: ['10. 0. 1. 10:9100','10. 0. 1. 11:9100']

- job_name: 'payments-api'
metrics_path: /metrics scheme: https tls_config:
ca_file: /etc/ssl/ca. crt cert_file: /etc/ssl/tls. crt key_file: /etc/ssl/tls. key relabel_configs:
- source_labels: [__address__]
regex: '(.):\d+'
target_label: instance replacement: '$1'

Kubernetes через Prometheus Operator

El bilen 'scrape _ configs' ýerine ServiceMonitor/PodMonitor ulanyň.

yaml apiVersion: monitoring. coreos. com/v1 kind: ServiceMonitor metadata: { name: payments-api }
spec:
selector: { matchLabels: { app: payments-api } }
namespaceSelector: { matchNames: [ "prod" ] }
endpoints:
- port: metrics interval: 15s scheme: http relabelings:
- action: replace targetLabel: service replacement: "payments-api"

K8s düşündirişler (Operatorsyz, ýönekeýleşdirilen)

yaml metadata:
annotations:
prometheus. io/scrape: "true"
prometheus. io/port: "9102"
prometheus. io/path: "/metrics"

5) Saklamak: TSDB, WAL we retenşn

WAL (Write-Ahead Log) → täzeden başlandan soň çalt dikeldiş.
Compaction: bloklary gysmak, disk/CPU tygşytlamak.
Gaýtadan işlemek: gyzgyn maglumatlary 7-30 gün saklaň; uzak möhletli - çykaryň ("Masştab" -a serediň).

• `--storage. tsdb. retention. time=15d`
• `--storage. tsdb. max-block-chunk-segment-size`
• Disk: çalt SSD/NVMe; Gerek bolmasa tor jiltlerinden gaça duruň.

6) PromQL: esaslar we ýygy-ýygydan nagyşlar

Rate/irate

promql rate(http_requests_total{route="/deposit"}[5m])

Ýalňyşlyklar we üstünlikli

promql sum(rate(http_requests_total{status=~"2..    3.."}[5m]))
/ sum(rate(http_requests_total[5m]))

p95 gizlinlik

promql histogram_quantile(0. 95,
sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)

Nobatlar/doýmak

promql max(queue_depth{queue="withdrawals"}) by (region)

7) Recording rules we öndürijilik

Agyr sözleri öňünden hasaplaň we seriýa hökmünde saklaň.

yaml groups:
- name: api. rules interval: 30s rules:
- record: job:http:request_duration_seconds:p95 expr:
histogram_quantile(0. 95,
sum by (le, job) (rate(http_request_duration_seconds_bucket[5m])))
- record: job:http:success_ratio expr:
sum(rate(http_requests_total{status=~"2..    3.."}[5m]))
/ sum(rate(http_requests_total[5m]))

Goşmaça: çalt dashbordlar, Prometheus CPU-da az ýük.

8) Alerting и SLO (burn rate)

Burn-rate alert (multi-window, multi-burn)

yaml groups:
- name: slo. payments rules:
- alert: PaymentsSLOFastBurn expr: (1 - job:http:success_ratio{job="payments-api"}) > (1 - 0. 999) 14 for: 5m labels: { severity: "page" }
annotations:
summary: "SLO fast burn"
runbook: "https://runbooks/payments/slo"
- alert: PaymentsSLOSlowBurn expr: (1 - job:http:success_ratio{job="payments-api"}) > (1 - 0. 999) 6 for: 1h labels: { severity: "ticket" }

Alertmanager: hyzmatlar/sebitler boýunça marşrut, dublikatlary basyp ýatyrmak, ChatOps.

9) Ýollar we çukurlar bilen baglanyşyk

exemplars: 'trace _ id' -ni histogram baketlerine goşuň.
"release compare" üçin 'service', 'version', 'region' belliklerini goýuň.
Dashbordlarda - relizleriň düşündirişleri (Git SHA/wersiýa).

10) Masştab etmek we uzak möhletleýin saklamak

Federasiýa: ýokarky Prometheus aşakdakylardan birleşýär (job/label-süzgüçler boýunça).
Remote Write: Uzak möhletli saklaýyş/klaster (Thanos/Cortex/Mimir).

Artykmaçlyklary: tükeniksiz retenşn, gorizontal masştab, global view.
Minuslar: has çylşyrymly ekspluatasiýa, çykdajy.
Aýratynlyklar boýunça şarding: ulgamlaýyn metrikler, işewürlik, howpsuzlyk üçin aýry-aýry ýagdaýlar.

11) Howpsuzlyk

TLS/mTLS, Prometheus target/Alertmanager/remote _ write.
/ Targets we API üçin esasy/token-autentification (proks şlýuzynyň öňünde).
RBAC: rollar boýunça UI/seriýalara girmegi çäklendiriň; şahsy bellikleri gizläň.
PII-arassaçylyk: PII-ni metriklere ýazmaň; heş/lakamlary ulanyň.

12) Kubernetes-praktikasy

Prometheus Operator: CRD (ServiceMonitor, PodMonitor, Alertmanager, Prometheus).
kube-state-metrics + cAdvisor → klasteriň doly suraty.
Taýningler we çeşmeler: gözegçilik üçin bölünip berlen nodlar; CPU/RAM çäkleri.
Sesiň peselmegi: "önümçilik" nyşanlary üçin bellik saýlaýjylar, padding mümkin bolan ýerlerde scrape_interval.

13) Iş ölçegleri we önüm

Платежи: `payments_success_total{psp, currency}`, `payment_conversion_ratio`, `ttw_seconds_histogram`.
Oýun işjeňligi: nyrhlar/min, sessiýalary gauge, ýalňyşlyklar ýüze çykan halatynda drop-off hökmünde saklamak.
Töwekgelçilik/frod: tizlik/geo anomaliýalary boýunça triggerler; aýratyn logirlemek, metrikler - agregatlar.

14) Bahasy we öndürijiligi (FinOps)

Kardinallyga gözegçilik ediň (täze bellik goşmazdan ozal tag-revyu).
Gistogramma sampling/seýrek eksportçylar → 'scrape _ interval' ↑ kritiki däl nyşanlar üçin.
Uzak möhletli saklaýyş arkalarynda downsampling.
Dashbordlary kesmek we recording rules-a giň goldaw.

15) "Çalt başlamak" mysallary

Programmada RED eksport ediji (Python)

python from prometheus_client import Counter, Histogram, start_http_server reqs = Counter('http_requests_total','', ['route','method','status'])
lat = Histogram('http_request_duration_seconds','', ['route','method'])
start_http_server(8000)

def handle(req):
with lat. labels(req. route, req. method). time():
status = app(req)
reqs. labels(req. route, req. method, str(status)). inc()
return status

Çäkli alertler p95

promql alert: HighLatencyP95 expr: histogram_quantile(0. 95,
sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))) > 0. 25 for: 10m labels: { severity: "page", service: "api" }

16) Girizmegiň çek-sanawy

1. Esasy metrleriň (RED/USE) we domen görkezijileriniň toplumyny kesgitläň.
2. Kardinallyk boýunça bellikleri we gollanmalary ylalaşyň.
3. Scrape/ServiceMonitor, TLS/mTLS, relabel.
4. Esasy ýollar we exemplars üçin gistogrammalary goşuň.
5. p95, success ratio, iş agregatlary üçin recording rules dörediň.
6. SLO-alertleri (burn rate) we Alertmanager rutingini giriziň.
7. Dashbordlary galdyryň: service map, release compare, tölegler.
8. Federasiýa/remote _ write we retenşn barada karar beriň.
9. Girişi çäklendiriň (RBAC), PII-iň ýoklugyny barlaň.
10. Runbooks we game-day barlamalaryny açyň.

17) Anti-patternler

Ýokary kardinally nyşanlar (user/session/request_id).
Esasy SLO üçin Histogramyň ýerine Summary → ýok 'histogram _ quantile'.
Süzgüçsiz/aýlanmazdan Scrape "yzly-yzyna" → çykdajylaryň we sesiň ýokarlanmagy.
SLO-syz çig metrler boýunça alertler → alert-fetig.
recording rules → "agyr" dashbordlaryň ýoklugy.
TLS/mTLS → çalyşmak/syzmak töwekgelçiligi bolmazdan metriklere ynam.

Netijeler

Prometheus, iGaming platformasyna maksatlara bagly gözegçilik edýär: takyk gistogrammalar, durnukly agregatlar, takyk SLO-alertler we köp sebit kartasyna çenli ulalmak. Bellikleriň tertip-düzgüni, dogry recording rules, yzarlamalar/loglar bilen aragatnaşyklar we oýlanyşykly saklaýyş arhitekturasy, iň ýokary pursatlarda-da çalt goýberilmegini we öňünden aýdyp boljak p99-ny üpjün edýär.