Automatic rollback of releases

1) Why do you need an auto-rollback

• reduces CFR and MTTR;
• protects SLO (auth-success, p99 "stavka→settl," error-rate);
• prevents compliance incidents (PII/RG/AML).

2) Principles

1. Revert is a feature: Rollback planned for release design.
2. Policy-as-Code: thresholds, windows, exceptions - validation in the pipeline.
3. Canary-first: wash along the steps, rollback - mirror steps.
4. Data-safe: migrations are reversible/summative; configs - versionable.
5. SLO-gates: red SLI/guardrails → immediate auto-rollback.
6. Explainability: timeline, diffuses, reasons - to the WORM log.
7. No single button of doom: restrictions, confirmations for risk actions, SoD.

3) Auto-rollback triggers (signals)

3. 1 Technical SLI/KRI

auth_success_rate drop by GEO/PSP/BIN (e.g. − 10% in TR ≥10 min).
latency p99/error-rate key paths (deposit/output/settle).
queue lag / DLQ rate / retry storm.
db replication lag / cache miss surge.

3. 2 Business signals

deposit_conversion − X pp on canary vs. control.
settle throughput drop from baseline.
chargeback/decline spikes (soft/hard).

3. 3 Critical events

SRM failure in active A/B (traffic distortion).
Triggering security/PII guardrail.
Incompatibility of circuits/configs (validator/linter).

4) Architectural reversibility patterns

Canary → Ramp → Full: 5%→25%→100% promotion; rollback - in reverse order (100→25→5→0).
Blue-Green: atomic traffic switch between Blue and Green, rollback - instant return.
Feature Flags: kill-switch for behavioral change (TTL, guardrails, SoD).
Config as Data: GitOps promotion/re-promotion of the previous version; runtime snapshots.

• two-phase (expand→contract),
• reversible (down scripts),
• write-shadow (new fields are written duplicated),
• read-compat (old code understands the new scheme).

5) Policy-engine

• `auto_rollback if auth_success_rate. drop(geo="TR") > 10% for 10m AND coverage>=5%`
• `auto_rollback if bet_settle_p99 > SLO1. 25 for 15m`
• `auto_pause_flag if api_error_rate > 1. 5% for 5m`
• `deny_promote if slo_red in {"auth_success","withdraw_tat_p95"}`
• `require_dual_control if change. affects in {"PSP_ROUTING","PII_EXPORT"}`

All rules are versioned, tested and reviewed.

6) End-to-end flow

1. The regression detector is triggered (metric/alert/validator).
2. Checking exceptions (holiday peaks, test windows).

7) Integrations

Incident bot: '/release rollback <id> ', auto-timelines, links to dashboards and diffuses.
Metrics API: ready SLO view and guardrail statuses; exemplars for RCA.
Feature Flags: '/flag off <id> ', autopause by guardrail.
GitOps/Config: `/config rollback <snapshot>`; drift detector confirms the result.
Status page: optional public updates (via CL/policy).

8) Observability and rollback telemetry

Release Dashboard: auth-success, error-rate, p95/p99, settle throughput, PSP по GEO/BIN.
Guardrail Board: active/triggered rules, windows, hysteresis.
Coverage history:% of canaries/flags/regions over time.
Audit: who/what/when/why; artifact diffusions; policy version; result.

9) Security, SoD and Compliance

4-eyes/JIT for activities affecting payments/PII/RG.
Geo-fences: Rollbacks affecting regulatory requirements are applied locally.
WORM logs: immutable trace for checks.
Public Comm Packs: Consistent with CL/Legal; the details of the experiments were not disclosed to the outside.

10) Examples of artifacts

10. 1 Auto-Rollback Policy (YAML)

yaml apiVersion: policy.platform/v1 kind: AutoRollbackRule metadata:
id: "payments-auth-success-tr"
spec:
scope: { tenants: ["brandA","brandB"], regions: ["EU"], geo: ["TR"] }
signal:
metric: "auth_success_rate"
condition: "drop > 10% for 10m"
compareTo: "canary_control"
action:
strategy: "step_down"  # 100%->25%->5%->0%
cooldown: "15m"
exceptions:
calendar: ["2025-11-29:black_friday"]
manualOverride: false audit:
owner: "Payments SO"
riskClass: "high"

10. 2 Configuration rollback manifest

yaml apiVersion: cfg.platform/v1 kind: ConfigRollback metadata:
id: "psp-routing-revert-2025-11-01"
spec:
from: "payments-routing-2025-11-01"
to:  "payments-routing-2025-10-29"
criteria:
- metric: "auth_success_rate"
where: "geo=TR"
condition: "drop>10% for 10m"
notify:
incidentBot: true stakeholders: ["Payments","SRE","Support"]

10. 3 Kill-switch flag

yaml apiVersion: flag.platform/v1 kind: KillSwitch metadata:
id: "deposit.flow.v3"
spec:
guardrails: ["api_error_rate<1.5%","latency_p99<2s","slo_green:auth_success"]
autoPauseOnBreach: true ttl: "30d"

11) Working with data migrations

• Expand: add new columns/indexes without breaking reading.
• Migrate: double entry/replay, consistency check.
• Contract: delete old only after successful release + observation window.
• Down scripts: required; evaluation of time and locks.
• Shadow reads: comparison of the results of the old/new path (without side effects).
• Cancellation criteria contract: any guardrail "red."

12) Processes and RACI

Release Manager: pipeline owner and policies.
Service Owner: approves domain rules, accepts risk.
SRE: implements detectors, pullback mechanics, dashboards.
Security/Compliance: SoD, PII/RG control, audit.
On-call IC/CL: communications, status page.
CAB: post-factum overview of auto-rollbacks, rule adjustments.

13) KPI/KRI functions

Auto-Rollback Rate: the proportion of releases that rolled back automatically (norm: low, but not zero).
Time-to-Rollback: detekt→otkat (median/p95).
SLO-Breach Avoided: Instances where auto-backtracking prevented targets from being breached.
False Positives: the proportion of "false" rollbacks (target - ↓).
CFR before/after implementation of auto-rollback.
Cost of Rollbacks: extra time, canaries, computing resources.
Audit Completeness:% events with full timeline and diffuses.

14) Implementation Roadmap (6-10 weeks)

Ned. 1-2: catalog of critical metrics and basic thresholds; selection of strategies (canary/blue-green/flags); migration reversibility inventory.
Ned. 3-4: implementation of detectors and policy-engine; integration with incident-bot; GitOps-rollback for configs; dashboards guardrails.
Ned. 5-6: pilot on the Payments domain (auth-success, PSP-routing), tabletop training; WORM log and reports.
Ned. 7-8: expansion on Games/KYC; automatic flag pause; DR exercises with blue-green.
Ned. 9-10: threshold calibration, false positive reduction, FinOps cost estimation, RACI and learning formalization.

15) Antipatterns

"Roll back somehow": the lack of a plan and reversibility of migrations.
Global instantaneous activation/deactivation without steps.
Crude metrics rollback without context (no GEO/PSP/BIN stratification).
Ignoring SRM and peeking in experiments.
Release alerts without hysteresis → rollback flapping.
Manual editing of configs in the product without Git/Audit.
Deletes the old schema before passing the observation window.

Result

Automatic release rollback is the platform's protective grid: policies as code, correctly selected signals and thresholds, reversible architectural solutions (canary/blue-green/flags/reversible migrations), built-in communications and full auditing. This loop dramatically reduces the risk of releases, protects SLO and revenue, and increases the confidence of regulators and partners.