Alertlar va xabarnomalar: PagerDuty, Opsgenie

Alertlar va xabarnomalar: PagerDuty, Opsgenie

1) Nima uchun alertlarning alohida platformasi

• Servislar/teglar/muhit bo’yicha yo’naltirish.
• Eskalatsiya va jadvallar (navbatchilik, follow-the-sun).
• Voqealarni duplikatsiya/korrelyatsiya qilish.
• Jim oynalar (maintenance/freeze) va myut qoidalari.
• Monitoring, CI/CD va ChatOps bilan integratsiya qilish.

Tayanch: SLO ostonasi → alert → odam/avtomat → runbook → qaytish/fiks → postmortem.

2) Signallar modeli va jiddiyligi (severity)

• critical (page) - SLO buzilishi/pul yo’lining xatosi (depozit/chiqarish), foydalanish imkoniyatining pasayishi, burn-rate.
• high (page/ticet) - aniq SLO-teshiksiz sezilarli tanazzul.
• medium (tiket) - sig’imi, bekning degradatsiyasi, retrasi.
• low (inform) - trendlar, ogohlantirishlar.

Qoida: page faqat SLO yoki aniq biznes-trigger bo’yicha.

3) Yo’naltirish arxitekturasi

1. Manba (Prometheus/Alertmanager, Grafana, bulutli monitoring, o’z vebxuklari).
2. Шлюз (PagerDuty/Opsgenie service/integration).
3. Siyosatlar: teglar (’service’,’env’,’region’), severity, payload.
4. Eskalatsiyalar: navbatchilik darajalari ketma-ketligi (L1 → L2 → menejer).
5. Kommunikatsiyalar: ChatOps kanallari, maqom sahifalari, tarqatmalar.

Asosiy teglar namunasi (standartlashtiring)

’service’,’env’,’region’,’version’,’runbook’,’release _ id’,’route’,’tenant’(agar V2B/multi-tenant boʻlsa).

4) On-call va eskalatsiya jadvallari

Schedules: primary/secondary, роли (SRE, DBRE, Sec).
Rotations: kunduzgi/tungi, follow-the-sun, dam olish kunlari.
Overrides: ta’til/kasallik.
Eskalatsiyalar: ack-taymaut 5-10 daqiqa → keyingi qatlam. Ish soatlari bo’yicha - ixtisoslashtirilgan bo’limga; tashqarida - on-call maydonchasi.

Maslahat: kechasi (kamroq charchoq) va kunduzi (kontekst mavjud) qisqa eskalatsiya qadamlarini ushlab turing.

5) Alertmanager bilan integratsiya (bazaviy pattern)

yaml receivers:
- name: pagerduty pagerduty_configs:
- routing_key: ${PAGERDUTY_ROUTING_KEY}
severity: '{{ if eq. Labels. severity "critical" }}critical{{ else }}error{{ end }}'
class: '{{.Labels. service }}'
component: '{{.Labels. env }}'
group: '{{.Labels. region }}'
description: '{{.Annotations. summary }}'
details:
service: '{{.Labels. service }}'
env: '{{.Labels. env }}'
runbook: '{{.Annotations. runbook }}'
release: '{{.Annotations. release }}'
route:
receiver: pagerduty group_by: ["service","env","region"]
group_wait: 30s group_interval: 5m repeat_interval: 2h

Opsgenie (webhook)

yaml receivers:
- name: opsgenie opsgenie_configs:
- api_key: ${OPSGENIE_API_KEY}
responders:
- name: "SRE Primary"
type: team priority: '{{ if eq. Labels. severity "critical" }}P1{{ else }}P3{{ end }}'
details:
trace: '{{.Labels. trace_id }}'
runbook: '{{.Annotations. runbook }}'

6) Shovqin, dedup va korrelyatsiya

Dedup kaliti: barqaror fingerprint (masalan, service + route + code) dan foydalaning.
Guruh:’group _ by’server/muhit boʻyicha, 5xx kaskadi oʻnlab sahifalarni yaratmasligi uchun.
Myutlar/jim oynalar: migratsiya/relizlar/yuklash testlari uchun.
Agar’api-gateway @prod’uchun P1 hodisasi allaqachon sodir bo’layotgan bo’lsa, P2/P3 bostiring.

Anti-pattern: SLOga tasdiqlangan ta’sirsiz CPU/Memory bo’yicha peyj.

7) Relizlar va avto-harakatlar bilan aloqa

PagerDuty/Opsgenie kanareykali deployda SLO-gate → webhook to CI/CD → pause/rollback (Argo Rollouts/Helm) dan alert olinadi.
Alert’release _ id’,’image’ni oʻz ichiga oladi. tag’, payplaynga va qaytarish qadamining runbook’iga havola.

Izohlardagi runbook bogʻlamalari namunasi

runbook: https://runbooks. company/rollback/api-gateway#canary

8) ChatOps va kommunikatsiyalar

Slack/Teams’da avto-hodisa kanalini yaratish.
Slash-команды: `ack`, `assign @user`, `status set`, `postmortem start`.
Status-sahifa: P1/P2 avtomatik yangilash.

9) Hodisaning hayot sikli (minimal)

1. Trigger (SLO/datchiklardan alert).
2. Page (primary on-call).
3. Ack (tasdiqlash, TTA).
4. Communicate (kanal/maqom).
5. Mitigate (rollback/feature-flag/izolyatsiya).
6. Resolve (TTR).
7. Postmortem (taymline, sabablar, harakatlar, darslar, vazifalar egasi).

Role-kit: IC (incident commander), Ops lead, Comms, Scribe.

10) Foydali maydonlar (normalize)

json
{
"service": "payments-api",
"env": "prod",
"region": "eu-central-1",
"severity": "critical",
"event_class": "slo_burn",
"summary": "Withdraw 5xx > 0. 5% for 10m",
"runbook": "https://runbooks/payments/withdraw-5xx",
"release_id": "rel-2025-11-03-14-20",
"image": "ghcr. io/org/payments:1. 14. 2",
"trace_id": "8a4f0c2e9b1f42d7",
"annotations": { "canary": "25%" }
}

11) Signal manbalarining integratsiyasi

Prometheus/Alertmanager - SLO/RED ning asosiy manbai.
Grafana Alerting - dashbordlar/biznes metriklar uchun osonroq.
OpenTelemetry/SpanMetrics - latency/error yo’nalishlari bo’yicha.
K8s-hodisalar - klaster avariyalari (control-plane, PDB buzilishlar).
DB/Navbatlar - lag/locks/replication.
Dastur vebxuklari - domen signallari (PSP xatosi, frodning koʻtarilishi).

12) Siyosat va komplayens

RBAC siyosatlar, jadvallar, myutlarni yaratish/o’zgartirish uchun.
Audit: kim maqomini tan oldi/tayinladi/o’zgartirdi, taymstemplar.
PII - pailoadlarda minimallashtirish (foydalanuvchining email/telefoni o’rniga ID bileti).
DR-reja: PagerDuty/Opsgenie (fallback kanali) mavjud boʻlmaganda nima qilish kerak.

13) Amaliyot namunalari (PagerDuty vs Opsgenie)

14) Sokin derazalar va sovuqlar

Freeze: Faqat P1 qoldirib, rejalashtirilgan relizlar oynalariga peyjing qilishni taqiqlash.
’env = stage’,’region = dr’,’service = batch’teglari bo’yicha mulk.
Vaqtinchalik mute: ko’chib o’tishda - aniq egasi bilan.

15) Samaradorlik metrikasi (alertlar uchun SRE/DORA)

MTTA/MTTR (buyruqlar/servislar/smenalar kesimida).
Runbook’dan% alert (maqsad ≥ 95%).
SLO bo’yicha page-alertlar ulushi (maqsad ≥ 90%).
Ratio foydali/shovqinli (maqsad ≥ 3:1).
% avto yordam (veb-xuk orqali pause/rollback) - o’stirish.
Burn-down postmortem action items 14/30 kun oldin.

16) Anti-patternlar

Foydalanuvchiga ta’sir qilmasdan «temir» (CPU, disk) bo’yicha peyj.
’group _ by’ → «bo’ron» ning yo’qligi.
Hech qanday oyna yo’q - relizlar hamma narsani qizil rangga bo’yalgan.
’service/env/runbook’ bo’lmaganda fayllarni yo’naltirish/harakat qilish mumkin emas.
Severity va qoidalarning yagona shkalasi mavjud emas (har bir manba o’z-o’zidan).
Hech kim tuzatmaydigan «abadiy» ogohlantirishlar.

17) Joriy etish chek-varaqasi (0-45 kun)

0-10 kun

Severity shkalasini moslash va tag/izohlarni standartlashtirish.
PagerDuty/Opsgenie’da xizmatlar yaratish, schedules va asosiy eskalatsiyalarni sozlash.
Alertmanager/Grafanani bogʻlash,’group _ by’va dedupni yoqish.

11-25 kun

SLO alertlarini (multi-window burn) kiriting va runbook havolalarini qoʻshing.
ChatOps moslamalarini moslash.
Reliz/migratsiyada jim oynalarni yoqish.

26-45 kun

Kanareykalar (vebxuklar) uchun avto-pause/rollbackni birlashtirish.
MTTA/MTTR va allert-gigiyena (shovqinlarni tozalash) hisobotlarini kiritish.
action items ustidan postmortem va nazoratni standartlashtirish.

18) Tayyor snippetlar

Grafana Alerting → PagerDuty (JSON body mapping)

json
{
"routing_key": "${PAGERDUTY_ROUTING_KEY}",
"event_action": "trigger",
"payload": {
"summary": "{{.RuleName }}: {{ index. Labels \"service\" }}",
"severity": "{{ if eq (index. Labels \"severity\") \"critical\" }}critical{{ else }}error{{ end }}",
"source": "grafana",
"component": "{{ index. Labels \"env\" }}",
"group": "{{ index. Labels \"region\" }}"
},
"links": [
{ "href": "{{.DashboardURL }}", "text": "Dashboard" },
{ "href": "{{ index. Labels \"runbook\" }}", "text": "Runbook" }
]
}

Alertdan Webhook → Argo Rollouts pause

bash curl -X POST "$ARGO_API/rollouts/pause" \
-H "Authorization: Bearer $TOKEN" \
-d '{"name":"api-gateway","namespace":"prod"}'

Opsgenie - Routing Rule (psevdo)

yaml if:
tags: ["service:payments","env:prod"]
severity: ["P1","P2"]
then:
route_to: "SRE-Payments"
notify: ["Primary OnCall","Secondary"]

19) Xulosa

Alertlarning kuchli konturi - bu jarayon + intizom: SLOga asoslangan stratifikatsiya, malakali marshrutlash va eskalatsiyalar, yagona teglar va pailoadlar, sokin oynalar, ChatOps va avtomatik harakatlar (pause/rollback). PagerDuty yoki Opsgenie va UX ni tanlang, lekin bir xil shovqin, navbatchilik va javobgarlik qoidalariga rioya qiling - keyin peyj kamdan-kam, aniq va foydali bo’ladi va hodisalar qisqa va boshqariladi.