Sandboxes and test environments

1) Why do we need selected contours

• quickly test hypotheses and integrations without risking production;
• speed up the feedback cycle (PR → preview-link in minutes);
• reproduce errors and incidents on a secure copy;
• Perform contract, integration, load, and chaos tests
• train teams and equip partners on the "playground."

Key principles: isolation, configuration parity, test determinism, data security, default observability.

2) Hierarchy of environments and their purpose

Local (Dev) - local development: Docker Compose/Testcontainers, lightweight provider simulators.
Sandbox is a stand for external integrations (PSP, KYC, game aggregators) with fake data and real protocols.
QA/Test - integration and e2e tests, stable data fixes, regressions.
Stage/Pre-Prod - the outline as close as possible to production (configurations/limits/topology).
Ephemeral Preview - environment "on PR" (lives for hours/days), offline resources and URL, auto-demolition after merge/close.

The parity rule is "settings, policies and infrastructure dependencies in Test/Stage ≈ Prod," the differences are only in secrets and limits.

3) Sandbox types

1. Provider sandboxes: external PSP/KYC/games provide test endpoints; we add a layer of simulators to simulate rare and erroneous cases (timeouts, 5xx, unstable signatures).
2. Functional sandboxes: autonomous instances of domain services (payments, bonuses, achievements) + fixes.
3. Training/demo sandboxes: API "showcase" for partners with DevPortal, keys, quotas and rate limit.

4) Contracts, simulators and moki

Contract-testing (Pact/Buf): consumer/provider agree on schemes; incompatible changes are blocked on the CI.
Provider simulators: play edge cases (double collbacks, clock drift, HMAC signature with expired timestamp).
Event Fixes (Kafka/NATS): Case Library 'payment. authorized`, `kyc. verified`, `game. round. settled`.
Fault injection: controlled delays, drop-rate, out-of-order messages.

X-Timestamp: 1730812800
X-Signature: sha256=hex(hmac_sha256(secret, body + timestamp))

5) Test data, GDPR/PCI and anonymization

Never use real PII/PAN off-production.
Anonymization: synthetics generation + tokenization of sensitive fields; whitelisting for demo accounts only.
Data factories: user/transaction/session factories with predictable IDs and statuses.
Deterministic seeds: Identical fixes between test runs and Wednesdays.
Retention policy: auto-cleaning of preview environments and test databases.

6) Secrets and access

Separate secrets on Wednesdays; temporary credits and restricted roles.
KMS/HSM and rotations; excluded secrets in Git.
RBAC/ABAC for QA/Stage; access audit, break-glass only through negotiation.

7) Observability in non-industrial environments

Logs - structured, without PII, with masking;

Metrics latency p50/p95/p99, error-rate, throughput, DLQ, retrai;

Tracing (OTel): end-to-end 'trace _ id' from the input request to the simulator;

Dashboards as Code - dashboards and alerts are versioned next to the service.

8) Ephemeral preview environments (per-PR)

• PR → CI collects an image, generates migrations, raises namespace 'pr- ' in Kubernetes;
• Preview URLs and tokens of test users are issued;
• enabled tracing/metrics; when the PR is closed, the environment is deleted.

yaml apiVersion: v1 kind: Namespace metadata:
name: pr-4821 labels:
env: preview owner: team-payments

9) Local development: Compose/Testcontainers

yaml version: "3. 9"
services:
api:
build:.
environment:
- DB_URL=postgres://postgres:postgres@db:5432/app? sslmode=disable
- KAFKA_BROKER=kafka:9092 ports: ["8080:8080"]
depends_on: [db, kafka]
db:
image: postgres:16 environment: [POSTGRES_PASSWORD=postgres]
ports: ["5432:5432"]
kafka:
image: bitnami/kafka:latest environment:
- KAFKA_ENABLE_KRAFT=yes
- KAFKA_CFG_PROCESS_ROLES=controller,broker
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 ports: ["9092:9092"]

For automatic lifting of dependencies in tests - Testcontainers with fixes.

10) Load and stability tests

Load profiles: "tournaments," "payment waves," "mass fluffs."

KPI: RPS, p95/p99, resource limits (CPU/memory), TTFB, Time-to-Wallet.
Chaos injections: disconnections of providers, an increase in latency, "flaky" networks.
Circuit breaker/backoff policies are checked on Stage; dips go to DLQ and replicate.

11) Rollback and replay policies

Replay gateway for events from DLQ (manual/auto modes, filters by keys).
Migration bases: clear up/down and dry-run in preview/Stage; protection against disruptive changes.

12) Integration with DevPortal

Catalog of sandboxes and providers, field requirements, examples of queries.
The "Open Preview" button for each PR/branch; SLO/SLA metrics widget.
Generation of SDK and Postman/Insomnia collections from contracts.

13) Sandbox perimeter security

WAF + IP-allowlist for external sandboxes;

quotas and rate limits per key;

individual domains/subdomains; automatic removal of inactive keys;

scan of image vulnerabilities and dependencies on each build.

14) Processes: who uses and how

Developers - locally and preview, fast feedback.
QA - stable Test/Stage with managed data and simulators.
Partners - external Sandbox with DevPortal, quotas and monitoring.
SRE/Platform - load profiles, chaos, SLO verification.

15) Sandbox launch checklist

• Contracts in Registry, simulators cover success/errors/timeouts/repetitions.
• Test data synthetic, deterministic, no PII/PAN.
• Secrets from KMS, roles limited, auditing enabled.
• Metrics/trails/logs are available; alerts to error-budget and DLQ.
• Ephemeral previews go up on PR and auto-demolished.
• Load profiles and chaos scenarios are described by code.
• Migration policies and event replay checked on Stage.
• DevPortal publishes guides and query collections.

16) Implementation Roadmap

M0-M1 (MVP): local environments (Compose), basic PSP/KYC simulator, contract tests in CI, preview spaces in K8s.
M2-M3: fixture catalogs, Dashboards as Code, DLQ + manual replay, load profiles.
M4-M6: full-fledged external Sandbox with keys/quotas, chaos infrastructure, SDK autogen, "two versions in parallel" migration policy.
M6 +: geo-distributed Stage with failover, smart routing of providers via SLA in tests, automated training scripts in DevPortal.

17) Environment maturity model (brief)

1. Basic - there is Test/Stage, manual data, weak isolation.
2. Advanced - simulators, contract tests, observability, partial previews.
3. Expert - per-PR environments, chaos/load as code, DevPortal, strong security and full automation.

Brief conclusion

Properly designed sandboxes and test environments are the "airbag" and "accelerator" of delivery. Isolation, parity with production, provider simulators, deterministic test data, strong observability and automation of preview environments provide a fast and reliable code → check → release cycle, reducing the risk of regressions and simplifying platform scaling.