Operations and → Management Operations Quality Control

Quality control of operations

1) Why do you need it

Transaction quality is the predictability and reproducibility of activities on which revenue, SLAs, and user trust depend. A strong quality control system reduces variability, speeds up handovers between shifts, reduces the number of errors during releases and increases the speed of response to incidents.

• Make processes measurable and manageable.
• Reduce performance variability (stability).
• Reduce waste (waiting, alterations, "hand crutches").
• Build continuous improvement (Kaizen) into daily work.

2) Quality model: QA vs QC

QA (Quality Assurance) - built-in quality: standards, SOPs, trainings, gates, automated checks before and during the process.
QC (Quality Control) - result check/sampling/audit after execution (ticket review, log check, SPC card control).

Principle: maximum quality - during the design and execution (QA) phase, QC remains the "insurance" and data source for improvements.

3) Key elements of the system

1. Standards and SOPs: step-by-step instructions, role model, checklists.
2. Process map: inputs/outputs, owners, process SLO, artifacts.
3. Quality gates: pre-checks, stop-tap for risk.
4. SPC (statistical process control): control cards, triggers.
5. Audits and sampling: regular verification of compliance with standards.

6. Feedback and RCA: postmortems, 5 Why/" fish bone. "

7. Training and Certification: Skills Matrix, Shadow Shifts.
8. Automation: auto-checks, bots, policies, integration tests.

4) Quality control processes (examples)

Shift routines (monitoring, key rotation, backups, duty checks).
Handovers and escalations (escalation matrix, communication channels, timings).
Incident management (detection → communication → recovery).
Releases/feature connections/traffic transfers.
Operations with providers (PSP/KYC), reconciliations, reports.
Content management/limits, jackpots/bonus.
Work with data (ETL, archiving, confidentiality).

5) Process SLO and Quality KPIs

• FPY (First Pass Yield) - the proportion of processes that have passed without rework.
• RFT (Right First Time) - percentage of tasks without errors/returns.
• DPMO: defects per million opportunities (for bulk operations).
• Process SLO: p95/p99 duration,% of successful completions.
• Compliance Rate: compliance with mandatory SOPs/checklists.
• Change Failure Rate: Share of rollback/incident releases.
• Process MTTD/MTTR Fault Detection/Recovery.
• Handoff Quality Score: Handoff quality (completeness, timeliness).

6) Standards and checklists (QA)

• Health check of key dashboards (API p99, lag, DB connections).
• Provider statuses (PSP/KYC/studio), quotas and limits.
• Incident queues and open post-mortems.
• Release/phicheflag plan for shift interval.
• Redundant communication channels and escalation availability.
• Backups/keys/secrets - scheduled control.
• Handover from previous shift (artifacts, risks, observations).

• All tests/linters/safety green.
• CDC/external instruments contracts posted.
• Rollback plan and phicheflags; canary ready.
• Current runbook, attendant confirmed, provider windows considered.
• Release annotations in dashboards included.

7) SPC and control cards

• What we monitor: duration of operations,% of defects, reaction time to alerts, handover time.
• Rules: 1 point outside the limits, 7 consecutive points with growth/fall, 8 points on one side of the average - a signal of a change in the process.
• Actions: for SPC signals → short RCA and corrective measures (SOP correction, training, automation).

8) Sampling and Audits (QC)

Sampling plan: critical processes - daily spot checks; average - weekly; low - by triggers.
Audit criteria: completeness of checklists, accuracy of execution, correctness of communications, compliance with SLO, safety compliance.
Scoring of the audit: 0-100 with weights by criticality; results - to the overall quality dashboard.

9) Quality of handovers and shifts

Handoff package: short status, risks, "observed trends," unfinished activities, SLO per interval.
Communications: a single format for updates (template), SLA for responding to an incident channel, time boxes for making decisions.
Shadow shifts: new operators are on duty "in the shadows," then move on to independent shifts according to the certification checklist.

10) Quality of incident management

Definition of Done: The incident is closed only after restoring the SLO, publishing the update for the business/support and creating tasks for fixes.

Postmortem without accusations: facts, chronology, "what will go differently next time."

Action Items SLA: Deadlines and Owners; Weekly status reconciliation

Metrics:% of incidents without regression, average time to first update, timeline completeness.

11) Quality control automation

Auto-checkers: bots check the filling of checklists, the presence of release annotations, the correctness of Alertmanager routes.
Policies/rules: mandatory gates in CI/CD, configuration validation (JSON/YAML), secret scanners.
Process mining: analysis of logs to find bottlenecks and deviations from the "reference" route.
Auto-reminders: expired post-mortems, unclosed action items, missed SOP items.

12) Metrics and dashboards (minimum set)

Operations Quality Overview: FPY, RFT, DPMO, SLO process, Change Failure Rate, open action items.
Shifts Board: checklists, Handoff Quality Score, alert response time, monitoring coverage.
Incidents Quality: MTTD/MTTR, first client update, RCA completeness, regressions.
Release Quality: percentage of canaries with degradation, rollbacks, average duration of stakeholder updates.
Compliance & Security: implementation of mandatory procedures (backups, key rotation, access), violations and deadlines for elimination.

13) Quality alerts (ideas)

ALERT ShiftChecklistMissed
IF operations_shift_checklist_completed == 0 FOR 15m
LABELS {severity="warning", team="ops"}

ALERT HandoffQualityLow
IF handoff_quality_score < 80 FOR 1h
LABELS {severity="warning", team="ops"}

ALERT IncidentUpdatesSLA
IF incident_first_update_minutes > 10
LABELS {severity="critical", team="incident"}

ALERT ChangeFailureRateSpike
IF rate(release_rollbacks_total[7d]) > 1. 5 baseline_28d
LABELS {severity="warning", team="platform"}

14) Improvement procedure (PDCA loop)

1. Plan: select metrics/targets, identify bottlenecks based on SPC/audit data.
2. Do: change pilot (SOP, training, automation) in a limited area.
3. Check: compare metrics (FPY/RFT/SLO/incidents) before/after.
4. Act: scale successful, roll back unsuccessful; update standards.

15) Roles and responsibilities

Process owner: SLO, standards, dashboards, improvements.
Operators: execution, checklists, incident communications.
SRE/Platform: automation, monitoring, Alertmanager routes.
QA operations: audits, sampling, checklists, training.
Quality Manager: PDCA coordination, prioritization of improvements.

16) Anti-patterns

"Let's check later" - absence of QA, reliance only on post-factum QC.
Checklists for the sake of a tick (without consequences for omissions).
There is no single standard for handovers → loss of context and repetition of errors.
Measure "all in a row" without a goal → metrics without actions.
Postmortems without action items and deadlines → constant regressions.
Manual checks of what can be automated.

17) Implementation checklist

• Process map, owners, inputs/outputs, SLO.
• SOPs and checklists (shifts, releases, incidents, providers).
• Quality gates in CI/CD and operational tools.
• SPC dashboards and control cards.
• Sampling plan and regular audits.
• Handover template and Shadow shift training.
• Post-mortem regulations and tracking action items.
• Automate checks and reminders.
• Quarterly improvement targets (FPY/RFT/SLO/MTTR).

18) Templates (fragments)

Handoff: <date/time>
SLO summary: <p95 API, errors, incidents>
Releases/features: <what's at work, risks, windows>
Providers: <statuses, quotas, restrictions>
Risks/observations: <trends, potential bottlenecks>
Action items before <time>: <list, owners>
Contacts: <on-call, escalations>

Impact: <who was affected, metrics>
Timeline: <UTC + timezone, key events>
Root cause: <5 Why / fishbone>
Corrective actions: <what we change now>
Preventive actions: <what we will change in the process/tools>
Owners & Due dates: <who and when>
Signals to watch: <metrics and alerts>

19) Fast start (30 days)

Week 1: describe 3-5 critical processes, SLOs, owners; Start basic shift/release checklists.
Week 2: include quality dashboards and 3 alerts (ShiftChecklist, Handoff, IncidentSLA).
Week 3: Run samples/audits and SPCs for 1-2 metrics.
Week 4: Conduct 2 method postmortems and approve the PDCA plan for the quarter.

20) FAQ

Q: How to quickly see the effect?
A: Start with handovers and IncidentSLA: this gives an instant reduction in MTTR and increased predictability.

Q: Are SPCs needed if there are already alerts?
A: Yes. Alerts catch "fires," SPC - process shifts before a fire.

Q: What to automate first?
A: Release gates, checking shift checklists, release annotations and reminders on action items.