Operations playbooks

1) What is a playbook and how it differs from a runbook

Runbook is a linear step-by-step instruction for a typical operation/alert ("do one, two, three").
Playbook is a decision tree for scenarios with forks: different symptoms → different hypotheses → different branches of actions. Includes selection criteria, gate conditions, and fallback branches.
The purpose of the playbook is to reduce the MTTA/MTTR and the level of improvisation under uncertainty.

2) Where playbooks are needed first

Incidents: SLO drop (availability/latency/success), business SLI failure (conversion/success of payments).
Changes: releases, migrations, feature flags, configs (canary/rollback).
Maintenance windows: database/broker upgrades, certificate rotations.
Providers: PSP/KYC/CDN/IDP - degradation and swing over.
Security: compromised key, suspicious activity.
DataOps: delayed freshness, drift of the circuit, degradation of the pipeline.

3) Playbook standards (minimum composition)

1. Card: ID, Version/Date, Owner (Team/Role), Services/Regions/Tenants, Related Policies/Standards.
2. Purpose and launch conditions: which SLO/SLI we protect, which alerts/triggers are applicable.
3. Symptoms ↔ Hypotheses: correspondence table, how to quickly cut off incorrect hypotheses.
4. Decision tree: forks, security gates, stop/continue criteria.
5. Actions: step blocks with commands/links to the runbook 'and.
6. Communications: update template (Impakt→Diagnostika→Deystviya→Sled. update), channels and frequencies.
7. Rollback/folback: clear backout plan, limits and UX degradation flag.
8. Completion criteria: metrics, observation time windows.
9. Evidence: what to save (logs, graphs, screenshots, ticket ID).
10. History of changes: changelog, known limitations.

4) Playbook taxonomy (example catalog)

INC- - incidents (SLO/SLI, providers, infrastructure).
REL- - releases, rollbacks, configs/flags.
MW- - maintenance windows (DB/queue/cert/OS).
SEC- - security (accesses, keys, suspicious actions).
DATA- - freshness/quality/schemes.
PROV- - external providers (PSP/KYC/CDN/Email/SMS).

5) Life cycle and ownership

1. Initiation: based on incident/simulation/change.
2. Draft: author = service owner; review: SRE/security/data (by domain).
3. Pilot: tabletop/game-day; recording of passage time and defects.
4. Publication: in repo (Docs-as-Code), version, tags, links to dashboards.
5. Update: according to RCA/CAPA, at least once a quarter; SLA freshness.
6. Archive/depletion: in case of replacement/loss of relevance.

6) Integration with tools

Alert → Playbook: Each Page rule references exactly one basic playbook.
ChatOps: '/play start <id> 'opens the card, fixes evidence, sets update timers.
CMDB/catalog: the service has a list of relevant playbooks, owners, SLO, dashboards.
GitOps: playbooks and runbooks live in Git, have PR reviews and linters.

7) Playbook quality metrics

Actionability: ≥ 90% of runs result in specific actions without unknowingly escalating.
Time-to-first-action: a minute or two from Page to the first meaningful step.
Coverage:% Page alerts that have a bound playbook (100% target).
Freshness: the proportion of playbooks is fresher than 90 days.
Defect rate: comments on reviews/simulations for 100 playbooks.
Reuse: how many times the playbook has actually been applied (and what outcomes it led to).

8) Anti-patterns

"Playbook Encyclopedia" with 20 pages without a decision tree.
Commands without expectations of the result ("execute X" - what should change?).
There is no backout plan and limits - the risk of escalating the problem.
Communication channels/intervals are not indicated - the growth of PR risks.
Playbook without owner/update date - no one believes in its relevance.
Dozens of similar playbooks instead of one parameterizable.

9) Playbook mini-template (YAML idea)

yaml id: INC-PAY-001 name: "Payment Success Down"
version: 2. 4 (2025-10-15)
owner: team-payments@sre scope: [prod, region: eu, tenants: all]
goal: "Restore success_ratio ≥ 98% without violating SLA"
triggers:
- alert: slo. burn. payment_success_ratio
- external_status: psp-a partial outage symptoms:
- "5xx growth in payments-api"
- "p95 latency> 400ms on PSP-A"
decision_tree:
- if: "quorum(eu,us) confirms drop AND PSP-A status=partial"
then:
- action: "Reduce PSP-A weight to 30%"
runbook: rb://payments/traffic-shift guardrails: ["success_ratio improving 10m", "p95<300ms"]
- action: "Enable degrade_payments_ux"
runbook: rb://payments/feature-flags
- action: "Status update (30m) by template"
comms: statuspage://payments else:
- action: "Check database/cache/queue"
runbook: rb://payments/diag-stack fallback:
- action: "Failover на PSP-B 70%"
guardrails: ["fraud_rate stable", "chargeback risk noted"]
rollback:
- condition: "PSP-A green 60m"
- steps:
- "Weight of PSP-A 30→70→80 (every 30 m at green SLI)"
evidence:
- "SLI screenshots, p95/5xx graphs, links to logs/trails"
completion:
- "success_ratio ≥98% during 30 m, no burn in 6 h"

10) Ready-made examples (fragments)

A) Payments: "Provider degrades in one region"

Symptoms: decreased success_ratio of the TR cohort, increased PSP-A timeouts.
Solutions: reduce the weight of PSP-A for TR, enable degrade-UX, strengthen retrays with a budget ≤ SLA, prepare a client update.
Backout: Regain weights at a green SLI of 60 minutes.

B) DB: "Growth p99 and connection errors"

Symptoms: p99↑, connection reset errors, growth wait events.
Solutions: enable read-only scripts, limit write load, scale pool/replicas, if necessary - hot failover.
Backout: parameter rollback, replica-prime.

C) Cache: "Miss rate ↑ → database load"

Symptoms: miss rate> 40%, growth of CPU DB.
Solutions: balance eviction policy, increase memory/sharding, temporarily enable read-through, limit RPS on hot keys.
Backout: return the policy, recreate the problematic shard.

D) CDN: "Regional content degradation"

Symptoms: increase in latency/timeout in one country, RUM complaints.
Solutions: change routing map/GSLB, bypass problematic POP, reduce TTL, enable origin-shield.
Comms: status updates with geography of influence.

E) KYC: "Failed identifications"

Symptoms: drop approve rate, vendor_error growth.
Solutions: switch part of the traffic to an alternative provider, reduce the severity of the rules (within the framework of the policy), initiate a manual review for VIP.
Compliance: log of all changes, Risk/Legal notifications if necessary.

11) Communications (update template)

Impact: EU payment success drop (-3. 1% to SLO, 25 min).
Diagnosis: confirmed by quorum; PSP-A partial outage; p95 = 420ms.
Action: PSP-A weight reduced to 30%, degrade-UX included; next update 18:30 UTC.

12) Playbook Author Checklist

• Target, owners, SLO/SLI and triggers specified.
• There is a table "Symptoms ↔ Hypotheses" and a decision tree.
• Executable steps with expected results and security gates.
• Backout/fallback and return conditions are spelled out.
• Communication template and update frequency.
• Links to dashboards/alerts/log searches/trails.
• Required evidence section and completion criteria.
• Version, date, SLA freshness, change history.

13) Review checklist

• Playbook is playable on tabletop/game-day.
• Steps are safe (limits/canary/auto-rollback), secrets are not disclosed.
• Roles and escalations are clear; IC/Comms are indicated.
• No duplication with adjacent playbooks; parameters are removed.
• It is clear when to stop and go to fallback/rollback.
• The document is available from alert in 1 click.

14) Parameterization and reuse

Carry out variables (region, provider, thresholds) in'values. '.
General steps (for example, "reduce the weight of the provider," "enable degrade-UX") should be issued in separate runbooks.
Support generators from templates: 'plb new --type = INC --service = payments'.

15) Implementation Roadmap (4-6 weeks)

1. An inventory of Page alerts → map to each basic playbook.
2. Templates: approve YAML/Markdown structure, checklists and linters.
3. Top 5 scenarios (payments/DB/CDN/KYC/cache) → write/roll back to tabletop.
4. Integration: links from alerts, ChatOps commands, evidence-bot.
5. Drill: weekly mini-drill one playbook at a time; AAR→uluchsheniya.
6. Freshness SLAs and Quarterly Reviews; quality metrics report.

16) The bottom line

Playbooks are operational scenarios with forks and railings that translate the chaos of "what to do?!" into a predictable sequence of decisions. When playbooks are standardized, integrated with alerts and regularly trained, the team responds faster, risks are controlled, and the business sees the stability and maturity of exploitation.