Operations and → Management Operational Infrastructure Scaling

Operational infrastructure scaling

1) Why and what is considered "scaling"

Scaling is the platform's system ability to increase throughput (RPS/TPS, connections, IOPS, throughput) and data volume without losing SLO and at a controlled cost. For iGaming/fintech, this is directly about money: deposit/bet conversion, live games and settlements.

• Keep SLOs at X-fold load growth and seasonal peaks.
• Provide predictable scaling time (minutes, not hours).
• Save Economy: cost/RPS, cost/transaction, cost/1k events.

2) Scalable platform principles

1. Horizontal-first: division into small, stateless services; status - in data clusters.

2. Back-pressure and queues: smoothing bursts, protection against "storms."

3. Caching on all layers: client/edge/service/database.
4. Idempotence and repeatability: safe retreats, outbox, dedup.
5. Constrained dependencies: timeouts, breakers, bulkhead isolation, rate-limits.
6. Observability by capacity signals: headroom, p95/p99, lag, connections, quotas.
7. Auto-scaling with guard rails: HPA/VPA/Cluster Autoscaler + stop conditions.
8. Multi-region by design: independent blast zones, local data, stable fylovers.

3) Capacity planning: how to "calculate how much you need"

Model inputs: target peak TPS, traffic profile (hourly), "critical paths," cache hit ratios, average payloads, SLOs and provider limits.

• RPS → CPU/pods: 'pods = RPS p99_time/effective _ CPU _ in _ pod' (with a margin of 30-50%).
• Queues: 'minimum _ speed _ of _ consumers ≥ peak _ speed _ of _ producers 1. 2`.
• DB connections: 'max _ conns = active _ service _ pools medium _ pool _ size 1. 3`.
• Cache: size = "hot working-set in N minutes" + 20-30% margin.
• Egress/CDN: peak egress = peak requests average response size (consider compression).

Headroom: 20-40% target at peak (by layer). Below 15% → the "capacity uplift" trigger.

4) Layers and scaling patterns

4. 1 Edge / CDN / WAF

Edge caching (TTL + SWR), geo-balance, compression, HTTP/2/3.
Rate-limits on the perimeter by IP/JWT/key, surge protection.
Event fan-out (jackpots, live alerts) through brokers/pub/sub channels.

4. 2 Backend-for-Frontend API Gateway

Horizontal scaling by statles, dedicated pools by downstreams.
HPA by business metrics: RPS, p99, work pool queue - not just CPU.

4. 3 Asynchronous queues/streaming (Kafka/Rabbit/Pulsar)

Scale by parties and consumers; avoid skew (keys and distribution).
Lag alerts + auto-scaling of consumers; DLQ and retry topics.
Retention under SLA reconcilations and replay.

4. 4 Caches (Redis/Memcached)

Cluster modes, replicas, eviction policies (LFU), multiget, pipeline.
Separation of hot-key and background tasks, client limits and max-memory policy.

4. 5 Databases

Read replicas and reading routing, connection pooling.
Sharding by region/tenant/key range.
CQRS: writes to master/leader, reads to replicas.
Indexing and batch-writing workflows (outbox → stream → sink).
Archiving and hot/cold data (tiering).

4. 6 File/Object Stores

Multithreading, multipart downloads, CDN-front, asynchronous transformations.
Provider quotas, tail cleaning and egress budget.

4. 7 Providers (PSP/KYC/Studios)

Multi-vendor and quote/SLO/cost routing.

Circuit breaker + rate-limit for each provider, retray queue, "grace modes."

5) Auto-scaling and guard rails

• HPA: метрики `rps_per_pod`, `queue_depth`, `p99_latency`; `targetAverageValue`.
• VPA: resource guidelines; update out of peak.
• Cluster Autoscaler: spot + on-demand profiles with priorities.
• PodDisruptionBudget/TopologySpreadConstraints: uniform across zones.
• LimitRange/ResourceQuota: protection against "drunk" depleys.

apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: api-gw}
minReplicas: 8 maxReplicas: 200 metrics:
- type: Pods pods:
metric:
name: rps_per_pod target:
type: AverageValue averageValue: "120"
- type: Pods pods:
metric:
name: p99_latency_ms target:
type: AverageValue averageValue: "280"
behavior:
scaleUp:
stabilizationWindowSeconds: 90 scaleDown:
stabilizationWindowSeconds: 300

• "Pause & Rollback," if with canary p99> 1. 3 × baseline 10 minutes.
• "Freeze scale-down" in prime time, scale-up only.
• "Stop retries" at 'open _ circuit = 1' to bottlenecks.

6) Multi-region: asset/asset and asset/liability

Blast-isolated regions: independent clusters, local secrets/quotas.
Global routing: latency-/geo-based, health-probes, manual override.

• Hot - locally + eventual replication (streams).
• Critical transactions - consistent by domain (ledger/balances).
• Failover playbooks: step-by-step source change, TTL, warm-up caches.
• Exercise Regulation (DR): Quarterly workouts with RTO/RPO goals.

7) Network and service patterns

Service Mesh: mTLS, retry/breaker, outlier detection, per-downstream limits.
eBPF/Observability on L4/L7, connection limits, head-of-line protection.
Internal API gateways for S2S, general rate-limit and audit.
VPC/subnets by blast zones, NAT/Egress control, peering with vendors.

8) Performance: Tests and proofs

Load & stress in primetime + worst-case profile.
Soak (long) - memory/descriptor leaks, latency growth.

Chaos/game-days: broker/provider/zone drop, "slow provider."

Perf regressions in CI: a set of reference scenarios and automatic gates.

9) Data and Storage: Growth Strategies

Vertical growth to the ceiling → horizontal/sharding.
Read → replica/cache; records → batch/asynchron/log.
Schema migrations: expand → migrate → contract, no global locks.
Archiving: cold batches to cheap storage + on-demand re-hydration.
Search: individual indexes (OpenSearch/Solr) with incremental updates pipeline.

10) Manage providers and quotas

Quota card (TPS, windows, cost); alerts' usage _ ratio> 0. 9`.
Routing by cost/quality (smart routing).
OLA ↔ SLO agreements and quota increase process.
Pool of alternatives and "hot" switching.

11) Observability and scaling signals

• Capacity headroom по слоям; `queue_lag/backlog growth`; `kafka ISR`; `db connections`/`repl lag`; `redis evictions`; `open_circuit`/`retry_rate`; `quota_usage`.
• Business metrics: success rate/deposit conversion, game start time.
• Cost: cost/RPS, cost/1k calls.

• Capacity Overview (headroom, top risks, burn-rate SLO).
• Stream & Queue Panel (lag/backlog, consumer saturation).
• DB & Cache (p99, connections, hit/evictions).
• Providers & Quotes (TPS, timeouts, cost, switching).
• Change Safety (pre/post release, canary, autogates).

ALERT HeadroomLowAPI
IF capacity_headroom{layer="api"} < 0. 15 FOR 10m

ALERT KafkaBacklogAtRisk
IF (consumer_lag > 5e6 AND rate(consumer_lag[5m]) > 5e4) AND (hpa_desired == hpa_max) FOR 10m

ALERT DBConnectionsNearMax
IF active_conns / max_conns > 0. 85 FOR 5m

ALERT ProviderQuota90
IF usage_quota_ratio > 0. 9 FOR 5m

12) FinOps: Scaling is profitable

Efficiency ratios: cost/RPS, cost/deposit, cost/1k events.
Right-sizing: VPA/recommendations, over-provisioned reports.
Spot/Preemptible for non-critical; Reserved/Committed for baseload.
Egress budget and caching, CDN/edge offload.
Collection and archiving of logs by value level (hot vs cold).
Warning quotas (soft-cap) and auto-tickets for extension.

13) Processes and people

Change Management: canaries, phicheflags, stops in regressions.

Incident-readiness: runbook 'and "where to add capacity," "how to switch region."

Scheduling peaks: match/tournament/campaign calendar and provider windows.
Regular game days and DR exercises.
Ownership matrix: who can "press the button" on the feilover/increase quotas.

14) Implementation checklists

• Map of critical paths and limits (by layer), headroom target ≥ 30%.
• HPA by Business Metrics + Cluster Autoscaler; PDB/SpreadConstraints.
• Queues on hot paths, idempotency-keys, outbox.
• Caches: hit goals ≥ 90%, evictions policy, key indexes.
• DB: read replicas, connection pool, sharding plan.
• Providers: multi-vendor, quotas, breakers/retreats.
• Dashboards "Capacity/Stream/DB/Providers," alerts from § 11.
• Canary and pre-/post-release autogates.
• DR playbook and one partial feilover training.

• Warming up caches, pre-scale HPA/ASG, warm-standby replicas.
• Increase provider quotas, enable smart routing.
• Enables night-mode suppressions for non-critical alerts.
• The "safe mode" feature is ready for instant activation.

15) Anti-patterns

Vertical upgrade "full stop" instead of horizontal.
A common head-of-line pool of threads/connections on all downstreams.
Retrai on bottleneck timeouts, lack of jitter → storm.

There is no hysteresis in alerts and scale policies → "sawing."

Single global database without data sharding and localization.
Blind faith in vendor SDK without timeout/retray/observability control.

Lack of DR exercises: feilover "only on paper."

16) Scalability KPI

SLO compliance at peak (p95/p99, success rate).
Headroom by layer in prime time.
MTTS (Mean Time To Scale) - until additional resources are available.
Backlog/Lag Resolution Time - the time the queues collapse after the peak.
Change Failure Rate for a period of active growth.
Cost/RPS and savings from cache/CDN/edge offload.
DR Readiness: RTO/RPO in exercise.

17) Examples of "quick" templates

partitions(topic="bets") = ceil(peak_msgs_per_sec / target_msgs_per_partition)
consumers = min(partitions, max_pods); rebalance_on: skew > 1. 5x scale_up_if: lag > 5e5 && rate(lag[5m]) > 5e4

max_connections = poolers pool_size 1. 3 read_routing: primary (write), replicas (read majority)
shard_key: tenant_id or region_id

maxmemory-policy: allkeys-lfu cluster-replicas: 1 evict-alert: rate(evictions[5m]) > 0 && used_mem/limit > 0. 8

guardrails:
- metric: api_p99_ms, threshold: 1. 3 baseline_1d, window: 10m, action: pause_and_rollback
- metric: error_rate, threshold: 2 baseline_1d, window: 5m, action: pause max_step: 10%
step_interval: 15m

18) FAQ

Q: What to scale first?
A: Bottlenecks according to dashboards: queues/caches/database reads. Hot paths (deposit/bet/game launch) are a priority.

Q: How to understand that auto-scaling "makes it worse"?
A: See the correlation: scale- up↑, and p99/errors do not improve - perhaps you "scale the problem" (narrow down stream/quota). Include breakers/degradation.

Q: Do you always need a second provider?
A: For critical paths, yes. Otherwise, at least "safe mode" with a simplified script and cache.

Q: Active-active или active-passive?
A: If RTO requirements are low and many regional players are active-active. Otherwise, start with active-passive with a used feilover.