DevOps practices and CI/CD

1) Goals and principles

Fast and safe: short cycles, small batches of changes, automatic checks.
Repeatability: infrastructure as code (IaC), environment = code + policy.
Observability: metrics/trails/logs out of the box, SLO as a contract.
Compliance: audit, change control, regional data isolation.

The golden rule: "First quality, then speed - otherwise speed will never appear."

2) Branches and environments

Trunk-based + feature flags - basic selection.

Short feature lines (≤ 2-5 days), daily merge in trunk.
Server-side flags for incremental delivery and secure rollbacks.
Git environments: 'dev' → 'stage' → 'prod' (+ regional 'prod-eu', 'prod-latam').
Promotion of artifacts: one collected image is promoted through the media (immutable tag by digest).

When GitFlow: rare releases of regulatory assemblies/SDKs - then release branches + "hardening."

3) Pyramid of quality and "red line"

1. Static analysis (SAST, linters, licenses).
2. Unit/Property-based tests (seconds).
3. Contract-tests (CDC) for APIs and events (OpenAPI/AsyncAPI, Schema Registry).
4. Integration (Testcontainers, local brokers).
5. E2E critical paths: registration → KYC → deposit → game launch → output.
6. Load/chaos tests for payments/wallet/game providers.

The quality does not pass → the deposit is blocked. There are no "manual exceptions" without a change-record.

4) Supply chain

SBOM for each image/package (CycloneDX/SPDX).
Artifact signatures (cosign), "signed only" policy in admission.
SCA/Dependabot: vulnerabilities and licenses.
Provenance/SLSA: reproducible assemblies, closed build agent, attestations.
Secrets: in the manager (KMS/External Secrets), not a single secret in the repo/logs.

5) GitOps и IaC

Infra as Code: Terraform/Pulumi for Cloud; Helm/Kustomize for k8s.
GitOps controller (ArgoCD/Flux): declarative manifestos, PR review, audit trail.
Windows/freezes: tournament weeks/peak hours - auto-pause of production releases.
OPA/Kyverno policies: no ': latest', non-root, read-only FS, hostPath disallow.

6) Progressive delivery

Canary: 1→5→10→25→50→100% on guardrail metrics (p95 latency, 5xx, error budget burn).
Blue-Green: fast switch + rollback plan.
Shadow/Mirroring: copy requests without affecting the response (for new PSP adapters).
Feature flags: inclusion by segment (region/role/partner/channel) + kill-switch.

7) Database migrations (expand-and-contract)

Step 1: expand the scheme (new columns/indexes) - compatible with the old code.
Step 2: dump code that writes to both versions/fields.
Step 3: Background joba data migration, progress metrics.
Step 4: Switch reading to new fields.
Step 5: Removing the old one is a separate release.
Blocking DDL ban in prime time; for high tables - online migrations.

8) Observability and SLO

Metrics: RPS, p50/95/99, 4xx/5xx, saturation (CPU/mem/queue), DLQ/broker lag.
Business metrics: TTP (time-to-play), TtW (time-to-wallet), FTD-success, KYC-TtV.
Traces: trace-id from gateway to database.
SLO: for example, 'Deposit p95 ≤ 300-500 ms', 'success ≥ 98. 5%`, `availability ≥ 99. 9%`.
Burn rate alerts + auto-pause releases during degradation.

9) Incidents, post-mortems, shifts

Runbooks on critical streams (deposit/output/ACC, live games).
Priority scale: P1...P4, owner, ETA, communication (banner, status page, partners).
Blameless postmortem with action items and dates.
Alternations on-call, chat alerts, status updates every N minutes.
Dock trail: who/when/what posted (commit, artifact, environment, flag).

10) Security and Compliance (DevSecOps)

SAST/DAST/IAST, secret scan in CI.
mTLS servis↔servis, JWT with small TTL, key rotation.
Masking PII/PAN in logs/tracks; WORM admin activity logs.
Geo-segregation: clusters/databases by region, gateway routing.
Change management: ticket/approval for sensitive areas (wallet/limits).

11) DORA metrics and FinOps

Deployment Frequency (daily small releases).
Lead Time for Changes (ideal: watch).
MTTR (recovery: minutes/hours).
Change Failure Rate (target ≤ 15%).

FinOps: cost of environments, RPS caching, warm pools, auto-pause of workers, "cost per transaction."

12) iGaming specificity

Peaks (tournaments/live): freezing major changes, warming up cache/images, quota boosts.
Payments/purse: individual pools/nodes, elevated SLOs, canary rollout by region, dual telemetry by PSP providers.
CC/compliance: separate cadence of releases, mandatory post-updates of compliance.
Partners/Affiliates: secure SDK, API version with support window and monitoring of old clients.

13) Example CI/CD (YAML, GitHub Actions → ArgoCD)

yaml name: ci-cd on:
push:
branches: [ main ]
paths: [ "services/wallet/" ]
jobs:
build_test_scan:
runs-on: ubuntu-latest steps:
- uses: actions/checkout@v4
- name: Setup Node uses: actions/setup-node@v4 with: { node-version: 22 }
- run: npm ci --omit=dev working-directory: services/wallet
- run: npm test -- --ci working-directory: services/wallet
- name: Lint & SAST run: npm run lint && npm run sast working-directory: services/wallet
- name: Build image run:
docker build -t registry. local/wallet:${{ github. sha }} -f Dockerfile.
cosign sign --key $COSIGN_KEY registry. local/wallet:${{ github. sha }}
- name: SBOM & Scan run:
syft packages registry. local/wallet:${{ github. sha }} -o cyclonedx-json > sbom. json trivy image --exit-code 1 --severity HIGH,CRITICAL registry. local/wallet:${{ github. sha }}
- name: Push image run: docker push registry. local/wallet:${{ github. sha }}

deploy_stage:
needs: build_test_scan runs-on: ubuntu-latest steps:
- uses: actions/checkout@v4
- name: Bump Helm values (image tag)
run: yq -i '.image. tag = "${{ github. sha }}"' helm/wallet/values-stage. yaml
- name: Create PR to gitops repo run: gh pr create -R org/gitops -B stage -H stage-bump/wallet-${{ github. sha }} -t "wallet:${{ github. sha }}" -b "Promote to stage"

promote_prod:
if: github. ref == 'refs/heads/main'
needs: deploy_stage runs-on: ubuntu-latest steps:
- name: Gate: SLO/quality checks run:./scripts/gates/check_stage_health. sh # p95, 5xx, e2e ok
- name: Canary 10%
run:./scripts/gitops/canary. sh wallet ${{ github. sha }} 10
- name: Auto-pause on degradation run:./scripts/gates/guardrails. sh./scripts/gitops/rollback. sh wallet
- name: Roll to 100%
run:./scripts/gitops/rollout. sh wallet ${{ github. sha }} 100

14) Checklists

Before merge to main

• Unit/CDC/integration green.
• Linters/SAST/licenses are clean.
• Updated OpenAPI/AsyncAPI schemas and database migrations.
• Fiche flags added, follbacks defined.

Before release in prod

• Image signed, SBOM attached, HIGH/CRIT vulnerabilities closed.
• Dashboards/alerts created; SLO gates are connected.
• Rollback plan, kill-switch, Shadow (if necessary).
• Regional restrictions and data policy confirmed.

Incidents

• Runbook found and up to date.
• Communication to users/partners (template, ETA).
• Postmortem at 48 hours, action items with dates.

15) Anti-patterns

"Reassembling for every environment" (no artifact promotion).
Manual deploy steps without audit/repeatability.
Database migrations "head-on," incompatible API responses.
Secrets in CI variables or in the repository.
Catastrophic features without flag/rollback.
Lack of SLO/guardrails on canary release.
Logs with PII/PAN, no masking.

16) Useful microcopy templates

• "We are rolling out the update of the payment module in stages (10%→100%). Short-term enrollment delays of up to 2 minutes are possible. ETA of completion - 9 p.m. EET"

• "Payment provider X is unstable. Enrollment can take up to 15 minutes. We are working on a fix. The next status update is in 30 minutes"

• "The update is on hold due to increasing delays. We return the previous version. Data and operations have been saved"

17) Implementation process (4 sprints)

1. Quality standards and pipeline: SAST/Unit/CDC, single image, signatures, SBOM.
2. GitOps + environments: Helm/Kustomize, ArgoCD, artifact promotion, secret policy.
3. Progressive releases and SLO gates: canary/shadow, guardrails, auto-hub.
4. Reliability and cost: chaos tests, autoscale/warm pools, FinOps dashboards.

Final cheat sheet

Trunk + flags + small batches = stress-free speed.
Single signed artifact + SBOM = controlled supply chain.
GitOps + policies = reproducibility and auditing.
Canary/Blue-Green + SLO gates = secure releases.
Expand-and-contract for DB = zero downtime.
Observability and DORA = manageable improvements.
Regional isolation and compliance = compliance with laws and trust.