Sandboxes for experiments

(Section: Operations and Management)

1) Purpose and principles

The sandbox is an isolated environment for safe experiments (features, configs, models, processes) without risk to the production service, money and personal data.

• Default isolation: networks, data, secrets, billing.
• Reproducibility: fixtures/seating, artifact versions, deterministic pipelines.
• Ethics and Safety: Protecting PII, guardrails and SoD.
• Observability: metrics/logs/trails as in prod, but with medium marking.
• Efficiency: quick lifting/demolition (ephemeral), cap on cost.

2) Sandbox taxonomy

Dev Sandbox: local development + test keys; minimal data.
Feature Sandbox (Preview): branch/PR environment with its own URL, static fixes.
Integration Sandbox: a fully functional stand with test external integrations (PSP/KYC/content providers).
Data Science/ML Sandbox: access to anonymized slices, experiment tracker, model registers/feature.
Chaos/Resilience Sandbox: injections of failures, slowdowns, limits.
Partner Sandbox: isolated showcase and API for external partners/tenants with test certificates.

3) Architecture and isolation

Network perimeters: individual VPC/NSG, closed subnets, egress through gateways with allow-list.
Identities and accesses: separate IdP groups/roles for sandbox, JIT rights with short TTL.
Secrets: individual vaults/namespaces in Vault/KMS; prohibition of shared secrets with prod.
Data: "zones of trust" (public → synthetic → anonymous → masked replica). Direct access to prod-PII is not allowed.
Artifacts: register of images/packages/configs with signed releases (DSSE), semantic versioning.

4) Data for experiments

Synthetic (generative profiles, distributions close to combat).
Anonymized (masking/tokenization, k-anonymity, diff-privacy for aggregates).
Fixes (ready-made cases: "payment> limit," "invalid KYC," "affiliate dispute").
Seeds/scenarios: deterministic generators with 'seed _ id', directory of edge cases.
Freshness rules/TTL: life of kits, ban on exfiltration.

5) Ephemeral environments

Automatic media lift to PR/branch (IaC), dedicated domains/certificates.
Auto demolition by TTL/Merge; limits on CPU/RAM/egress, storage quotas.
Auto-heating of fixtures/seeds; Status bar for QA/product/partners.
Signed snapshots of the environment for reproducibility of bugs.

6) Experiments: types and techniques

A/B/n and feature-flags: percentage rolling, targeting by segment/region.
Shadow traffic: a copy of real requests to the sandbox without side effects (write-drop).
Canary/Blue-Green: Small percentage of real traffic per experimental route.
Chaos injections: delays, errors, dependency failure, short timeouts.
Datadrift/models: backtesting, offline metrics, online-guardrails.

7) Guardrails and Ethics

Policies-as-code: OPA/ABAC - where/what data can be used.
Guardrails experiments: latency limits, error-rate, payout/egress limits, ban on manipulative UX patterns.

SoD: "launches experiment" ≠ "approves" ≠ "analyzes and makes a decision."

Ethics: Transparency for affected users (where appropriate), respect for vulnerable groups.

8) Observability and metrics

Trails/metrics/logs with tags: '{environment = sandbox, experiment_id, variant, seed_id}'.
SLI: availability, p95, error-rate, correctness of calculations, quote↔checkout match.
Experiment KPIs: conversion, retention, complaints, cost/1k, egress.
Guardrails: negative signals (growth of fraud/chargers, RG-triggering) - instant stop.

9) Cost and FinOps

CPU/RAM/egress per-sandbox quotas; budgets/cap-alerts 80/90/100%.
Downsampling and shelf life of metrics/logs; artifacts - into cheap layers after T + N days.
"Save button": stop inactive sandboxes, automatic archive of snapshots.

10) Registry of experiments and reproducibility

Experiment Registry: '{id, hypothesis, owner, design, date, metrics, guardrails, artifacts, solutions}'.
Protocol autogeneration: timeline, artifact versions, data samples, screenshots/trails.
Links to PR/tickets/dashboards; "planned/running/frozen/closed" statuses.

11) Safety and compliance

Separate roles and keys; MFA/FIDO2 for admin panels.
PII - synthetic/masked only; request for access to aggregates - through approval of the data owner.

Webhooks: test endpoints signed/TTL/nonce; prohibition of sending to

WORM logs for critical experiments (finance/responsible play).
Regional restrictions (data/key localization) are also observed in sandboxes.

12) Integrations with external providers

Test accounts/certificates of PSP, KYC, content providers.
Environment marker in headers/metadata ('X-Sandbox: true'), individual limits and reporting.
Simulators with controlled latency/errors.

13) Processes and RACI

14) Sandbox SLO

The rise time of the ephemeral medium ≤ 10 minutes (p95).
Availability of core services in Integration Sandbox ≥ 99. 5%.
Coincidence of schemes/contracts with sales: 100% (validation at CI).
Completeness of experiment artifacts (protocol/versions/metrics) = 100%.
Sandbox cost per unit time ≤ N (budget).

15) Typical playbooks

Shadow traffic gives errors: disable writing, enable write-drop, increase simulator timeouts.
Price discrepancy (quote≠checkout): reconciliation 'fx _ version/tax _ rule _ version', cache disability, catalog fix.
Surge egress: turn on compression, cut the logging level, recalculate quotas.
Provider failure: switch simulator/backup provider, fix latency.
PII flag: immediate sandbox isolation, artifact removal, Privacy/Legal notification.

16) Specificity of iGaming/fintech

RTP & Limits: synthetic/simulation only, no real payout calculation; guardrails on display.
Payments/PSP: test merchants, quarantine of "gray" transactions, manual clearing into a simulator.
Affiliates/webhooks: test signatures, sandbox escrow logic, receipt reconciliations.
Responsible game: simulators of RG events, checking UX messages and limits.

17) Implementation checklist

• Describe the sandbox levels and responsibilities (Dev/Preview/Integration/ML/Chaos/Partner).
• Deploy IaC templates for ephemeral environments (PR auto-lift).
• Configure Identities/Secrets/Networks: complete isolation from prod.
• Create catalogs of fixtures/sides and synthetic datasets; Enter TTL.
• Connect observability and experiment tags; start guardrails and alerts.
• Enable Experiment Registry and protocol templates.
• Set up external provider test loops and simulators.
• Enter quotas/budgets and FinOps-dashboard.
• Ethics/Compliance Procedures (PII, Notifications, SoD).
• Conduct GameDay: leaked test keys, egress surge, PSP simulator failure.

18) FAQ

Is it possible to use "impersonal food"?
Only after strict anonymization/mask and with a separate perimeter; synthetic data are preferred.

Do I need separate domains/certificates?
Yes, to eliminate intersections with sales and simplify security policies.

How to quickly close sandboxes so that the score does not grow?

TTL, auto demolition by merge/inactivity, budget alerts, "save button."

How is Preview different from Integration?
Preview - for feature/PR, fixed fixes; Integration is a fully functional stand with test providers.

Summary: Sandboxes are a manageable infrastructure for safe and honest experimentation. Isolate networks/data/secrets, use synthetic kits, raise ephemeral environments over PR, enable observability and guardrails, keep a register of experiments and control costs. This will accelerate innovation without risking production business, players and compliance.