Network Connectivity Resilience

(Section: Ecosystem and Network)

1) Purpose and area

"Network resilience" is the ability of the ecosystem to maintain accessibility and predictable quality of interactions between participants (operators, providers, studios, affiliates, nodes/validators, payment and KYC services) when channels, nodes, regions and external attacks fail. Key objectives are minimizing MTTR, containing cascade failures, controlled degradation, and rapid recovery to target SLOs.

2) Threat model

Network: packet loss/jitter, link congestion, BGP flap, interregional breaks, asymmetric routing.
Transport/sockets: half-open connections, head-of-line blocking (TCP), state exhaustion (NAT/conn-track).
Application layer: traffic spikes, "long-playing" requests, n + 1 RPC, retray storm.
Dependencies: degradation of DNS, KMS/PKI, queues, TURN/relay, third-party APIs.
Security: DDoS L3/L4/L7, bot flood, cache poisoning, Sybil/spam attempts.
Operating system: incorrect feature flags, "hot" releases without limits, incorrect timeouts.

3) Sustainability design principles

1. Redundancy across all layers: paths, regions, providers, relay nodes, DNS, secret storages.
2. Fault isolation: cell-based architecture, circuit-breakers, bulkheads, limits on cross-cell calls.

3. Fail-fast and time-boxing: short timeouts for external calls, prohibition "wait forever."

4. Idempotence and safe retreats: idempotence keys, deadup at the receiver.
5. Default observability: traces, correlation IDs, synthetic samples.
6. Degradation modes: read-only, cache-only, drop-features, priority of critical threads.
7. Chaos engineering: proof of stability by experiment.

4) Topologies and redundancy

Hybrid P2P + super-peers + DHT: local mesh within "contract" groups, super nodes as repeaters and caches, DHT for search.
Anycast/Geo-DNS/SD-WAN: near input, controlled flows, health-based routing.
Multi-relay (TURN/HTTP3-tunnels): independent suppliers, relay budget only if necessary.
Active-Active regions: synchronous for idempotent reads/events; for monetary transactions - final consistency + strict finalization.

5) Protocols, timeouts and retreats

Transport: QUIC/HTTP3 (multiplexing without HoL-blocking, path migration), TCP - as fallback.

• RPC client timeout: p99_latency×1. 5 (but ≤ 2-3 s interregionally).
• Connect timeout: 200-500 ms locally, 700-1200 ms interregionally.
• Backoff: exponential with jitter; max-retries 2-3 for "reading" calls.
• Hedged requests: after p95 delay send a second executor (only idempotent operations).
• Idempotency: header/field 'x-idempotency-key', storage of dedup logs ≥ TTL retrays.
• Queues and outbox: guaranteed delivery of events, repetition in case of network failures, dedup on consoles.

6) Load management and "self-protection"

Rate-limits and quotas: leaky-bucket/token-bucket on RPC/topic.
Adaptive load-shedding - resets low-priority requests when latency increases.
Priorities: money/payouts> gaming events> telemetry.
Backpressure: dynamic window, concurrency restrictions, "credit limits" of peers.
Connection pooling: warm pools, limits on open sockets/NAT states.

7) DDoS and channel security

L3/L4: upstream scrubbing/Anycast, conn-track защита, SYN-cookies, UDP-rate.
L7: WAF/WAAP, proof-of-work/fee-gate for open topics, captcha/wallet pledges against spam.
mTLS/TLS 1. 3 + E2E: encryption "on the go," pinning of super-node keys, rotation of certificates.
Anti-Sybil: trusted peer-ID registry, reputation, KYB/KYC for influencing roles.
Security defaults: "prohibited if not allowed," ACL by topic, minimizing rights.

8) SLO, SLI and resilience metrics

• Uptime of critical endpoints ≥ 99. 95 %/30d
• p99 latency interregionally ≤ 600 ms; error-rate ≤ 0. 2%.
• Success-rate P2P-RPC ≥ 99. 5%; Pub/Sub E2E p95 ≤ 2 с.
• Relay-share ≤ 30%; DHT resolve p95 ≤ 300 мс.
• MTTR SEV-1 ≤ 30 min; MTTA ≤ 5 min.

• Connectivity%, proportion of direct connections, average number of neighbors.
• RTT/Jitter/Loss by traffic class; RPC success/failure taxonomy.
• Queue depth/lag in brokers/relay; DHT hit/miss and age of records.
• Burn-rate by SLO (1h/6h/24h); impact on business KPIs (GTV/MAU losses).

9) Observability and synthetic samples

Tracing: end-to-end trace-IDs, export via OpenTelemetry, semantics of spans for network hops.
Logs/metrics: structural logs, cardinality under control, p95/p99 aggregates.
RUM + synthetics: real user metrics and global sample grid (every 1-5 min) from key regions/providers.
SLO dashboards: "traffic lights" for critical flows, delay/availability maps, degradation reports.

10) Degradation modes

Read-only/cache-only: when clipping a record into backends.
Stale-while-revalidate: we give away an outdated but good cache with a background update.
Feature kill-switch: fast switch of unstable parts.
Limiting fan-out: ban on "fan" requests, fusion in depth.

11) Chaos-engineering (plan)

Network Faults: 1-5% packet-loss, 100-300 ms jitter, blackhole of individual ASNs.
Relay/TURN failure: turning off N% of super-nodes, checking the proportion of direct connections.
DNS/KMS degradation: artificial timeouts/errors, validation of follbacks.
Retray storm: checking protection against cascades (jitter, limits, deadup).
Game-day rules: hypothesis → injection → metrics → improvement → repetition.

12) DR strategy and targets

RPO/RTO: for these configurations and ACL - RPO ≈ 0 (synchronous snapshots), RTO ≤ 15 min; for telemetry, RPO is allowed ≤ 5 minutes.

Catalogs and keys: cold reserves, periodic failed backups, "recovery training."

Regional disasters: Anycast/Geo-DNS switching, cache warming, queue/topic replication.

13) Pseudo-configurations

Client Timeout and Retreat Policy (YAML)

yaml client:
rpc:
connect_timeout_ms: 400 request_timeout_ms: 1500 retries:
max_attempts: 2 backoff: exponential base_ms: 100 jitter: true hedging:
enabled: true threshold_ms: 800  # p95 idempotent_only: true

Circuit-breaker and priorities

yaml resilience:
circuit_breaker:
error_rate_threshold: 0. 02 rolling_window_sec: 60 open_duration_sec: 15 priorities:
payouts: high game_events: medium telemetry: low load_shedding:
target_p99_ms: 600 drop_low_priority: true

ACL and e2e channels

yaml security:
mtls: required e2e_topics: [payouts. status, limits. update]
acl:
operators: [12D3KooA..., 12D3KooB...]
providers: [12D3KooC..., 12D3KooD...]

14) Dashboards: layouts

Ops (hourly/real-time): Connectivity%, RPC p99, error-rate, relay-share, DHT-latency, queue-lag, SLO burn-rate.
Network Health (week): relay-% and RTT trends, lists of "noisy" peers, NAT traversal success, traffic cost.
Strategy (month): SEV, MTTA/MTTR, DR training incidents, correlation with business metrics.

15) Playbook incidents (cheat sheet)

Jump p99 and errors: enable degradation (read-only, cache-only), hedging, increase quotas for critical flows, open tickets on the "hot" path.
Relay-share> threshold: switch STUN/TURN pools, expand super-nodes, strengthen hole-punch, temporarily raise TTL caches.
Retray storm: reduce max-retries, increase jitter, turn on the global backoff flag through the config service.
DDoS L7: enable WAAP rules, signature/speed block, enable PoW/fee-gate on public topics, off. non-essential endpoints.
DNS/KMS problems: use secondary providers, local key caches, switch resolvers.
Region unavailable: failover traffic (Anycast/Geo-DNS), warming another region, recalculating limits.

16) Implementation checklist

1. Record SLO/SLI and owners (by streams/topics).
2. Implement timeouts/retrays/hedging/idempotency.
3. Configure circuit-breakers, bulkheads, and priorities.
4. Run synthetic samples and global dashboards.
5. Enter DR plan (RPO/RTO), regular recovery training.
6. Conduct a quarterly chaos day and revision of parameters.
7. Document degradation modes and communication patterns.

17) Glossary

Bulkhead - isolation of subsystems to prevent cascades.
Circuit breaker - automatically disables an unstable dependency.
Hedging - competitive requests after a threshold delay.
Outbox/Inbox - reliable sending/receiving of events with deduplication.
RPO/RTO - allowable data loss/recovery time.
SLO burn-rate - the rate of "burning" the error budget relative to SLO.

Bottom line: the stability of network connections is not "one feature," but a discipline: redundancy and isolation of failures, competent timeouts and retrays, strict prioritization, observability and regular tests. This approach turns inevitable network failures into managed events with minimal impact on ecosystem business flows.