Incident-bot and chat operations
1) Purpose and value
Incident bot is an incident management interface directly from corporate chat (Slack/Teams/Telegram): one text input → actions in dozens of systems. Is he:- reduces MTTA/MTTR by automating routine;
- creates a single loop of facts (SoT) and communications;
- provides provability (audit, timelines, SLA updates);
- reduces the load on on-call and "burial" in consoles.
2) Roles and RACI in chat operations
Incident Commander (IC) - incident owner: opening/closing, priority, solutions.
Comms Lead (CL) - texts and schedule of updates (external/internal).
Domain Leads (Payments/Games/Core/Infra) - technical facts and fixes.
Scribe - timeline, action log.
Bot Admin - bot rights/policies/integrations.
Rule: one incident has one IC and one CL; role reversal - by explicit bot command.
3) End-to-end scenarios
1. Start of the incident: alert → '/incident new p1 "Deposits EU down" '→ the bot creates a card, var-room, assigns IC/CL, sets a timer for the first update.
2. Ведение: `/incident add-facts`, `/incident status set degraded`, `/incident assign @payments-lead`, `/incident timer 20m`.
3. Communications: '/incident publish status' (draft CL), '/incident partners notify ', '/incident regulator draft'.
4. Действия: `/runbook psp-failover PSP1→PSP2`, `/feature toggle replay-center off 60m`, `/traffic shift 30% eu→uk`.
5. Closing and post-mortem: '/incident resolve ', auto-collect timeline, '/postmortem generate'.
4) Bot commands (core)
Create/Classify
`/incident new p{1|2|3|4} "
`/incident severity set p2`, `/incident tag add payments,psp`
Ownership and roles
`/incident ic @user`, `/incident comms @user`, `/incident assign @user [domain]`
Timers and SLO updates
'/incident next-update 15m ', '/incident remind' (bot pings CL), '/incident eta set 18: 30'
Facts and status
`/incident fact "auth-success PSP1 -25% TR/EU"`, `/incident status {investigating|degraded|monitoring|resolved}`
Comm packs
`/incident draft public|partners|regulator`, `/incident publish public`
Integration
'/runbook
Closure/post-mortem
`/incident resolve [reason=…]`, `/postmortem generate`, `/postmortem assign @owner`
5) Integrations (minimum required)
Monitoring: alerts, SLI/SLO (burn-rate), links to dashboards.
Incident Manager (ITSM): two-way status/field synchronization.
Status page: drafts and publishing via CL (policy-gate).
Providers (PSP/KYC/Game Studios): contact directories, quick letters/channels.
Release/Feature Flags: canary stops/pullbacks, release links.
Runbooks/Auto-remediation: Safe Action Catalog with guardrails.
CMDB/owners: auto-assignment of domain leads, escalation.
Timeline storage: WORM/immutable for audit/post-mortems.
6) Bot architecture
Gateway (Chat Adapter): Slack/Teams/Telegram interfaces.
Command Parser + Policy Engine: authorization, validation, SoD and tolerances.
Orchestrator: incident scenarios, timers, reminders.
Integrations Layer: clients to ITSM, monitoring, status page, releases, runbooks.
Evidence Store: events, facts, message diffuses, attachments (WORM).
Metrics & Audit: quality metrics, action logs, command tracing.
7) Policies, rights and security
RBAC/ABAC: who can create/close, change severity, publish outside.
SoD: Comms publish requires CL role; high-risk actions (PSP routing, PII export) - dual control.
JIT rights: temporary issuance to domain leaders at the time of the incident.
Signature and encryption: webhooks/requests to systems - HMAC/mTLS.
Fat-finger protection: confirmation of dangerous commands, dry-run and TTL for action.
PII hygiene: masking in drafts/logs; PII inhibition in open channels.
8) Automation flows (example)
Alert P1 → bot creates a var-room ('# inc-2025-11-01-001'), pings on duty (IC, CL, Payments/Infra).
Binds dashboards/SLI, opens a ticket in ITSM, prepares a template for the first public update.
Sets timers: "next update in 15 minutes," CL reminders.
Предлагает runbooks: “PSP reroute 30% → PSP2”, “degrade replay-center”, “autoscale settle-workers”.
When publishing - fixes the version of the text and publishes it to the status page/social network (via CL).
When closing - collects timeline, metrics, draft post-mortem, VIP/partner mailing.
9) Timelines and provability
Each event is logged: 'T + mm: description, author/bot, command, result, links'.
Edits of messages (diff), binding to releases/feature flags/planned work are supported.
Export: PDF/CSV for audit and regulators.
10) Metrics (KPI/KRI ChatOps)
MTTA (chat): alert to '/incident new '.
MTTS (setup): before the var room is ready and roles are assigned.
Cadence adherence: adherence to public update intervals.
Runbook usage rate: proportion of incidents with automated activities.
Consistency score: discrepancies between channels = 0 - target.
Pager fatigue↓: reduced manual pagers with the same/better SLO.
Postmortem SLA: proportion of post-mortems collected ≤ D + 5.
11) Template catalog (fragments)
Create P1:
/incident new p1 "Deposits EU down" components=payments,deposits regions=EU
First public update (via CL):
/incident draft public
/incident publish public
PSP routing and feature degradation:
/runbook psp-failover PSP1→PSP2 30%
/feature toggle replay-center off 45m
Post-mortem:
/postmortem generate
/postmortem assign @owner
12) Embedding in processes
Communications: Link to Incident Communication and System Status Pages.
Observability: quick links to SLO/SLI and synthetics; auto-attachment of graphs.
Alerting: auto-creation of an incident during P1/P2; a single stream of signals.
Auto-fixes: one-button runbooks with guardrails and rollbacks.
Workflow Engine: human-tasks (4-eyes), escalation timers, checklists.
13) Implementation Roadmap (4-8 weeks)
Ned. 1-2: MVP teams: '/incident new ', roles (IC/CL), var room, update timer, communication with ITSM and monitoring.
Ned. 3-4: message templates (public/partners/regulators), status page (chernovik→publikatsiya), catalog 5-7 runbooks.
Ned. 5-6: policy-as-code (RBAC/SoD/JIT), dual control at high-risk, WORM magazine, ChatOps KPI dashboard.
Ned. 7-8: tabletop P1/P2 exercises, integration with releases/feature flags, auto-collection of post-mortem, localization.
14) Antipatterns
"All over the bot" without guardrails → random dangerous actions.
Posts to status page without CL/Legal-review role.
Commands without logs/versions → unprovability.
Complex forms (20 + fields) in chat - the speed drops; better short commands + links.
There are no update timers → "silence" at P1.
Lack of CMDB/owner integration → assignment chaos.
15) The bottom line
Incident-bot and ChatOps are not a "bot with commands," but an operating platform: a quick start to an incident, update discipline, automated actions with safe restrictions, end-to-end observability and provability. Such a circuit predictably reduces MTTR, improves the quality of communications and protects the revenue of the iGaming business at peak times.