Automatic error correction

1) Purpose and principles

Objective: To reduce MTTR and prevent escalation of incidents by preserving SLO, revenue and compliance.

• SLO-first: Auto-actions are allowed only if there is a confirmed threat to the error budget.
• Security first: minimal blast-radius, explicit limits and timeboxes.
• Explainable by design: Each action is explainable and auditable.
• Rollback-ready: any step is accompanied by return criteria.
• Human-in-the-loop where the risk is high: P1-critical changes - through dual control or IC/on-call confirmation (unless otherwise established by policy).

2) Terms

Auto-remediation: programmatic reaction to an event (alert/anomaly) without human intervention.
Guardrails: restriction policy (threshold, duration, number of attempts, impact area).
Runbook-Action: atomic operation with pre/post checks and rollback.
Decision Engine - A service that maps an event to policies and triggers actions.

3) Solution architecture

1. Signals: SLO/burn-rate, KRI, synthetics, RUM, deep-health.
2. Context correlation: releases, feature flags, planned work, dependent providers.
3. Decision Engine: rules/policies (policy-as-code), impact and risk assessment, scenario selection.
4. Execution: orchestrator of runbook actions (idempotency, retrai with jitter).
5. Control: pre-validators, post-verifiers, timebox, rollback.
6. Audit and observability: activity trace, success metrics, log (WORM/immutable).
7. Communication: status page (via Comms Lead), var-room, macros for support.

4) Policy-as-code

• `allow if burn_rate(payments. auth) > fast && impact>threshold && psp_alt. healthy && within_limits("psp_reroute")`

• `allow if p99(bet_settlement)>3x && queue_lag>limit && feature("replay_center"). enabled`

• `allow if consumer_lag>target && cost_budget. ok && region_capacity. available`

• `allow if export_spike && no_ticket && data_class=PII -> action=block + notify(Compliance)`

Each policy contains: condition, action, limit (scope/time/frequency), success criteria, rollback.

5) Safe actions directory (atomic runbook-actions)

Payments: switch traffic to an alternative PSP/bank; change the priorities of routing health × fee × conversion; Enable simplified 3DS raise retray limits with jitter.
Betting/Gaming: Scale Settle Workers; Enable cache-warmup temporarily disable non-critical features (animations, secondary feeds); enable waiting-room/queue-page.
Infrastructure: remove degraded instances (outlier-detector), evacuate traffic to the neighboring AZ/region; Increase pool/quota restart the workers with lint checks.
Data/queues: redistribute parties; raise consumers to cap; switch read traffic to a healthy replica; Enable adaptive route sampling.
Security/compliance: temporarily block PII exports without a ticket; Strengthen velocity output limits enable dual control on sensitive operations.
Comma layer: auto-draft status + update slots for Comms Lead; notifying partners when the PSP degrades.

6) Pre- and post-validation

• Check that the problem is real and fresh (N-of-M windows; no silence/planned work).
• Verify that the action is allowed by policy and that there is a resource budget.
• Estimate cost (FinOps) and compliance constraints.

• Confirm burn-rate/metrics reduction; record the result; Schedule auto-rollback according to conditions.

7) Rollback и “escape hatch”

Auto-return when stabilizing metrics and through max-TTL actions.
Roll back button for IC/on-call in var room.
Break-glass for emergency access only; post-audit is required.

8) Integration with alert and incidents

Any auto-action is attached to the incident card: who/what/when/why, result, links to graphs.
The pager is muted for duplicates, but not for failed auto-fixes (escalation).
The status page is updated via Comms Lead from the template.

9) Safety and compliance design

Least privileges for the orchestrator; individual roles per action/domain.
SoD and dual control for high-risk: PSP routing, bonus limits, PII export.
Audit the WORM/immutable of all automatic solutions, including inputs and policy versions.
PII hygiene: without personal identifiers in labels and action logs.

10) Observability of auto-loops

Metrics: success-rate of actions, reaction time,% rollbacks, MTTR savings, effects on SLO.
Traces: end-to-end traces for signal → decision → action → effect.
Logs: structured, with policy_id, versions and pre/post checks.
Dashboards: Exec (revenue impact/SLO), Ops (action matrix × domains), FinOps (cost of auto-measures).

11) Example scenarios (iGaming)

11. 1 PSP degradation (TR/EU)

Signal: auth-success in PSP-1 ↓ by 25% in 10 minutes, coverage> 30% of transactions.
Actions: redistribute 40% of traffic to the PSP-2/3; Enable simplified 3DS raise retrays of Bank X requests with jitter.
Boundaries: no more than 60% of total traffic per alternate PSP; TTL 45 min.
Rollback: at normalization of success-rate ≥ target for 15 min.

11. 2 Rising p99 at settle stakes

Signal: p99 "bet→settle"> 3 × norm + consumer-lag> threshold.

Actions: scale-out of workers before cap; coefficient cache warm-up; temporarily turn off "redo history."

Rollback: after headroom> X and p99 normally 20 min.

11. 3 Database replica lags behind

Signal: replication-lag> N seconds, lock-wait growth.
Actions: divert read traffic to a healthy replica; enable low priority throttling write operations.
Rollback: after lag normalization and lock errors.

11. 4 PII export spike

Signal: export rate> baseline × K, no tickets.
Actions: export block, Compliance notification, dual control enabled.
Rollback: after confirming requests and closing the anomaly.

12) KPI и KRI

MTTR↓ for incidents where the auto-fix worked.
TTD→Action: the time from the detection to the action.
Success-rate of actions and Rollback-rate (low - good, if not due to false positives).
False-action rate (actions with no effect or with a negative effect).
SLO impact saved.
Pager fatigue↓ (fewer manual pagers with the same/better SLOs).

13) Implementation Roadmap (8-12 weeks)

Ned. 1-2: select 3-5 high ROI scenarios (PSP-feilover, autoscale by lag, feature-degrade); describe policies/limits/rollbacks.
Ned. 3-4: implement action orchestrator, secrets and roles, integration with incident platform; add observability and auditing.
Ned. 5-6: pilot in "shadow" mode (simulate-only) → A/B effect estimate; then include in the product with low coverage.
Ned. 7-8: expand the directory of scripts (database/cache/queues/front), associate with the status page and Comms.
Ned. 9-10: add FinOps limit rules (cost/SLI), implement dual control for high-risk.
Ned. 11-12: tabletop/chaos teachings, KPI/KRI revision, publication of guidelines and on-call training.

14) Artifacts and patterns

Auto-Remediation Policy: condition, action, limits, TTL, rollback, owner, risk class.
Runbook-Action Spec: preconditions, steps, checks, errors, monitoring, rollback logic.
Change-Control: Who can rule policies, PR reviews, tests, diff and version.
Evidence Pack: SLO impact logs/trails/metrics, report for post-mortem/audit.

15) Antipatterns

"Treating the symptom" without checking the cause and SLO → flapping.
Actions without rollback and TTL → frozen degradation.
Universal scripts without guardrails → cascading crashes.
Lack of audit and policy versioning.
Ignoring cost (autoscale without a limit) and compliance (PII exports).
Full autonomy without Human-in-the-loop in P1 risks.

Total

Automatic error correction is a managed loop: SLO signals → policies with guardrails → secure runbook actions with rollback → observability and audit → incident training. This approach measurably reduces MTTR, keeps revenue in spades, and removes the routine from on-call while remaining compliant with safety and regulatory requirements.