Blue-Green and Canary releases

(Section: Architecture and Protocols)

1) Why do we need "safe rollouts"

In modern systems, release is not only code delivery, but also a controlled experiment in sales: we simultaneously minimize risk (do not break users) and reduce feedback time (quickly see the effect). Two classic strategies - Blue-Green and Canary - solve this in different ways, but with a common goal: zero downtime, quick rollback, observability by SLO.

2) Basic definitions

Blue-Green

We keep two full-fledged copies of the production environment: the active (Blue) serves traffic, the passive (Green) is preparing a new version. Switching is atomic (switch/flip) at the balancer/router level. If it got worse, we instantly return to Blue.

Canary

We roll out in parts: first to a small% of traffic (for example, 1-5%), observe metrics/SLO, then step by step increase the share (10% → 25% → 50% → 100%). During degradation - rollback or stop at the previous stable step.

3) When which approach is best

• We need an instant rollback without complex maneuvers.
• Architecture/budget allows for dual infrastructure duplication.
• We want to carry out large-scale migrations or platform updates (OS/JDK/runtime) in isolation.
• Application/connection pools are sensitive to a gradual "mixed" state.

• You need to minimize blast radius and see the behavior on the share of users.
• High release rate, progressive delivery as normal.
• There are mature observability and automatic gates (error budget, latency, conversion).
• The product team wants to test hypotheses: impact on conversion, retention, LTV, etc.

4) General principles for a successful release

Idempotent build artifacts: the same image/package at all stages.
Deterministic configuration: config as code, comparability of environments.
Observability by design: logs, metrics, traces, alerts; SLI/SLO in advance.
Fast, automatic rollback: The rollback button/command is part of the pipeline, not manual magic.
Compatible schema changes: expand-migrate-contract strategy (see § 10).
L7 routing (desirable): Flexibility on API headers/cookies/paths/versions.

5) Blue-Green: Architecture and Process

5. 1 Topology

Two prod stacks: Blue (active) and Green (candidate).
Common external dependencies: CDN, external APIs, queues; DB is a special case (see § 10).
Switch point: balancer/Ingress/Gateway.

5. 2 Step-by-step flow

1. We raise Green under a new artifact (vNext), we conduct smok tests.
2. Autotest run against Green (e2e, contract, regression).
3. Warm up the cache/sessions (if applicable), synchronize background jabs/queues.
4. Switch traffic to Green: atomic flip (DNS TTL low, Route/Listener swap, Ingress weight = 100%).
5. We observe SLO in the first minutes/hours (golden signals: latency, errors, saturation + business metrics).
6. In case of problems - instant return to Blue (flip back).

5. 3 Pros/Cons

Pros: instant rollback, simple mental model, pure isolation.
Cons: doubling infrastructure, difficulty with stateful components and data migrations.

6) Canary Architecture and Process

6. 1 Topology

Single production cluster; several versions of the service (stable and canary) behind a single front.
Traffic is divided by weights (1-5-10-25-50-100%) or by targets (by header/cookie/ID).

6. 2 Step-by-step flow

1. Deploy canary versions to the same cluster/ASG/NSG.
2. Route some traffic (for example, 1-5%) to canary.
3. Automatic checks of SLI/SLO and business metrics; gates in CI/CD (error rate, p95 latency, CPU/RES, conversion, refusal/return).
4. Step-by-step increase in the share of traffic when passing gates.
5. Full rollout to 100% and deactivation of the old version; in case of degradation - auto-rollback.

6. 3 Pros/Cons

Pros: minimal risk for most users, data-driven solution.
Cons: we need mature observability, competent routing, the risk of "version skew" between instances.

7) Traffic routing

Layer L4: balance by IP/ports; simple, but little flexibility.
Level L7: HTTP/S rules - on the path, host, header, cookies, User-Agent, GeoIP, SNI.

• Weighted routing (weights 1-100%).
• Header-based/Cookie-based.
• Session stickiness (important for stateful/cached scripts).
• Shadow/Traffic mirroring (mirror requests to the new version "quietly").

8) Tools and implementations (examples)

Kubernetes: Ingress (NGINX, Contour), Service Mesh (Istio/Linkerd), Argo Rollouts, Flagger.
Облака: AWS ALB/ELB, Route 53 weighted records, ECS/EKS; GCP Load Balancing + NEG; Azure Front Door/App Gateway.
CD platforms: Spinnaker, Argo CD, GitHub Actions + Progressive Delivery plugins, GitLab/CD.

9) Observability, SLI/SLO and gates

Golden signals: Latency (p95/p99), Error rate (5xx/4xx по типам), RPS, Saturation (CPU/Memory/GC), Queue lag.
Business metrics: conversion, authorizations, payments/successes, average check, refusal by funnel steps.

• Error threshold (for example, error rate canary ≤ baseline + X%).
• p95 latency is no worse than baseline by more than Δ.
• Business threshold (e.g. conversion drop
• SLO error budget should not burn out faster.

Step duration: minimum time sufficient for statistical significance (depends on traffic).

10) Database migrations and schema compatibility

The main rule: releases are safe if the back and forth versions are compatible.

1. Expand: add new columns/indexes/tables without breaking the old version.

2. Deploy app vNext (reads/writes to the new scheme, but knows how to work with the old one).

3. Migrate data (background/batch, idempotent, with checkpoints).

4. Contract: delete old fields/features after stabilization.

Anti-patterns: migrations requiring exclusive blocking at the point of Blue-Green switch; inability to downgrade the scheme; "double write" without deduplication.

11) Rollback and emergency plans

Blue-Green: instant flip on Blue; monitor the tails of Green background jobs.
Canary: weight rollback (for example, from 25% back to 5% or 0%); automatic abort on alerts.
Data: a well-thought-out policy of repetition/compensation (idempotency keys, "inbox/outbox" pattern, message deduplication).
Ficheflags: A quick kill switch to shut down partially rolled-out opportunities.

12) Working with state and sessions

Sticky sessions for canaries, or storing sessions externally (Redis/Memcached) so that the versions are interchangeable.
Cache to warm up in advance (Green warm-up) and take into account invalidation when flip.
Background workers: do not allow "races" between versions - queue separation or "leadership" by version.

13) Safety and compliance

Access to Green/Canary - by Zero Trust: service accounts, minimum required roles.
Secrets and keys - via KMS/Secrets Manager; turn on rotation.
Traffic - TLS only; the endpoint versions are clearly marked; Audit routing and release activities.

14) Cost and performance

Blue-Green doubles the infrastructure (at the time of release or constantly) - budget.
Canary is more economical, but requires observability tools and engineering time to automate.
Optimization: autoscaling, ephemeral environment, shortening the window of parallel existence of versions.

15) Checklists

Before release

• Image/build promoted from a single source, signatures verified.
• Test plan, alerts and SLO gates are configured.
• Database migrations - in expand mode, downgrade plans are available.
• Rollback plan - checked in staging/production-like.

At the time of release

• Metrics and logs are compared to baseline.
• For Canary, the steps and thresholds are fixed; for Blue-Green - flip-back readiness.
• The on-call commands are in the know, there is a feedback window.

After release

• SLO did not sink, error budget is normal.
• Post-release migrations/cleanups completed.
• Retrospective and playbook update.

16) Frequent errors and anti-patterns

Rollout without metrics: no data - no managed solution.
Mixing incompatible database schemes, lack of downgrade strategy.
Random traffic mixing: no stickiness, users "jump" between versions.
Hidden stateful dependencies (local disks, in-memory caches).
Long DNS-TTL interferes with fast flip (Blue-Green).
Lack of autogates: manual "by eye" solutions slow down and increase risks.

17) Combined approaches

Blue-Green + Canary: Roll out Green first, then inside Green roll out Canary for individual services.
Shadow/migrating traffic: before Canary, we run mirrored traffic to the new version.
Feature flags (progressive delivery): functionality is included on top of the stable version by "dark" flags by segments.

18) Sample scenarios (sketches)

1. Deploy Green (v2) for the new Listener/Ingress.

2. Warm up the caches, perform readonly checks, smoke.

3. We switch the weight to Green = 100%.

4. Observe SLO for 30-60 minutes; if everything is ok - turn off Blue.

1. Deploy canary vNext (replicas 5%).

2. We include 5% traffic for internal accounts/test segment.

3. Autogate: error rate ≤ baseline + 0. 3%, p95 ≤ +20ms.

4. We raise 10% → 25% → 50% every N minutes when passing gates.

5. We translate ficheflag 100% for all segments; deleting the old version.

19) Variations for different architectures

Monolith: Blue-Green is simpler, Canary is more difficult due to the indivisibility of features; use ficheflags.
Microservices: Canary is natural; monitor consumer-driven contracts.
Stateful services: Prefer Blue-Green with carefully crafted migrations and stickiness.

20) Brief comparison (summary)

Rollback speed: Blue-Green = instantaneous; Canary = fast but with a pullback in weight.
Cost of infrastructure: Blue-Green ↑; Canary ↔︎/↓.
Risk to users: Canary is lower (we control the share).
Implementation difficulty: Blue-Green is easier to start; Canary requires strong observability and automation.
Data/circuit compatibility: critical for both; plan expand-migrate-contract.

21) The bottom line

Blue-Green and Canary are not mutually exclusive strategies, but elements of progressive delivery. The choice depends on cost constraints, observability maturity, and the nature of the changes. Regardless of the approach, the sustainable release rests on four pillars: automation, observability, backward compatibility and fast rollback.