Shadow traffic and comparison

1) What is Shadow traffic and why it is needed

Shadow traffic (aka traffic mirroring/dark launch) is a secure "run" of real requests/events to a new version of the service in parallel with the production version without affecting users. The results of the new version are not returned to the client and do not produce external side effects, but are collected in a comparison system.

• Checking compatibility: schemes, contracts, business logic.
• Performance evaluation: latency, resistance under real load.
• Drift detection: differences in responses, distributions, error rate.
• Preparing for canary releases: Reducing risk before actually switching traffic.

• Rewriting the kernel/algorithms, migrating the database/cache, switching to another runtime/SDK, changing the provider of the external API.

2) Architectural patterns of Shadow traffic

2. 1 L7 proxy/gateway (HTTP/gRPC)

The proxy accepts the request → gives a combat response from the old version → asynchronously mirrors a copy of the request into a "shadow."

Suitable for synchronous APIs.
Share/mirroring filter control: on the way, header, user, tenant.

yaml route:
route:
cluster: prod-v1 request_mirror_policies:
- cluster: shadow-v2 runtime_fraction:
default_value:
numerator: 10 # 10% denominator: HUNDRED trace_sampled: true

nginx location /api/ {
proxy_pass http://prod_v1;
mirror/shadow; # request copy
}
location = /shadow {
internal;
proxy_pass http://shadow_v2; # answer ignored
}

2. 2 Event buses (Kafka/Threads)

• The producer writes as usual → prod consumers read.
• In parallel, the shadow-pipeline reads the same stream into a separate sandbox.

Options: MirrorMaker/Replicator, dual-write (caution), source → prod + shadow bridges.

2. 3 Replayer (record/play)

Snapshot of real requests/trails (PCAP/NGINX access, gRPC taps) → playing to the "shadow" at a controlled pace.

2. 4 "Synthetic Shadow"

Generating a load profile from production logs + phase of filling edge cases is useful for confidentiality restrictions.

3) Isolation of state and side effects

The golden rule: the shadow does not change the outside world.

Reed-on access to the database/cache or a separate sandbox (snapshot/replica).
Prohibition of outgoing side effects: payments, letters, fluffs, webhooks → stub/blackhole/record-only.
Command/POST idempotence: Shadow should not be registered as a repeat of the original.
PII/secret masking, test provider tokens.

Example: masking in a mirror

yaml shadowFilter:
headers:
redact:
- Authorization
- X-Api-Key body:
jsonPaths:
- replace "$ .email" # with token
- "$.card. number"

4) Sampling strategies and safe loading

Traffic share: 1-10% at the start; increase if v2 is within budget.
Selection criteria: by route, user, request size, operation type (GETs are safer).
Perf budget: mirroring should not increase p95/p99 combat service. The shadow is always asynchronous.
Back-pressure: when the shadow chain overheats, drop the shadow, not combat requests.
Time: Minimum 24-72 hours to cover per diem and peak patterns.

5) Comparison of results: methods and levels

5. 1 Comparison levels

1. Byte diff: The body of a one-in-one response/event. Simple but fragile (timestamps, key order).
2. Semantic diff: normalize and sort fields, ignore epemerides (traceId, timestamps, counters).
3. Business invariants: whether the same amounts, statuses, quantities, boundaries.
4. Statistical Analysis: Do Metric Distributions Match? (p50/p95, KS test, categorical χ ²).

5. 2 Comparison policy

Masks/ignore lists of fields (e.g. 'updatedAt', 'etag').
Accuracy: absolute/relative error for numbers (e.g. ± 1e-6).

Tolerance bands: "price difference ≤ 0. 01," "no more than + 0 errors. 1% relative to prod."

Comparator pseudocode

pseudo compare(prod, shadow, policy):
a = normalize(prod, policy)
b = normalize(shadow, policy)
diffs = deepDiff(a, b, ignore=policy. ignore, floatTol=policy. floatTol)
invariants_ok = checkInvariants(a, b, policy. invariants)
return Result(diffs, invariants_ok)

5. 3 Comparison of zapros↔otvet

Correlation-ID is required.
Link spans: shadow track gets link to battle.

6) Observability and comparison artifacts

• `shadow_requests_total{route,...}`
• `shadow_discrepancies_total{type=byte|semantic|invariant}`
• `shadow_error_ratio` и `shadow_slo_breach_total`
• Perf: 'shadow _ latencies _ ms {p50, p95, p99}'
• Diffuse logs: compact JSON deltas by key.
• Reporting: daily report with top N discrepancies, heat maps by routes/tenants.
• UI "diff explorer": filters by type, route, field, export in CSV.

7) Special occasions and subtleties

7. 1 Consistency and consistency

Shadow requests can come later/earlier; normalize to versions/hours (Lamport/vector) or window tolerances.
Read-after-write: a shadow with read-replica without synchronous replication will give different answers - compare through lag windows.

7. 2 Cache/recommendations

Do not mix prod and shadow caches.
For ML/recommenders, compare online metrics and offline metrics separately; watch for drift input features.

7. 3 External providers

Wrap clients in record-only/stub mode.
For settlement services (taxes, rates) - fix a snapshot of directories for both parties.

8) Canary juxtaposition/blue-green

Shadow: zero risk to users, but no real side effects; great for logic and perf.
Canary: small percentage of real answers from the new version; requires a ready-made rollback strategy and SLA.
Blue-green: instant switching after validation; Shadow is often used in front of it.

9) Implementation plan (GitOps-style)

1. Goals and metrics: which invariants and tolerances check which SLO for discrepancies.
2. Trace: Correlation-ID, distributed trace links.
3. Proxy configuration: mirror policy, sampling, redaction.
4. Isolation: database sandbox/cache, side stubs, test keys.
5. Comparator: normalization, ignore-maps, invariants, reports.
6. Load plan: start from 1-5%, growth up to 20-50% with green metrics.
7. Observability: "discrepancy/perf/volume" dashboards.

8. Exit criteria: "0 critical discrepancies 48 h," "errors not worse than prod," "perf within ± 5%."

9. Move to canary: Include real answers with safe share and automatic garda rules.

10) Configuration examples

10. 1 Istio (Traffic Mirroring)

yaml apiVersion: networking. istio. io/v1beta1 kind: VirtualService spec:
hosts: ["svc. example"]
http:
- route: [{ destination: { host: svc, subset: v1 } }]
mirror:
host: svc subset: v2 mirrorPercentage:
value: 0. 1 # 10%

10. 2 Kafka Tee (sketch)

text source-topic -> prod-consumer-group
-> shadow-consumer-group (isolated sink/db)

10. 3 Comparison rules (yaml policy)

yaml ignoreFields:
- $.traceId
- $.meta. generatedAt floatTolerance:
default: 1e-6 fields:
$.price: 0. 01 invariants:
- name: "nonNegativeTotal"
expr: "$.total >= 0"
- name: "statusMapping"
expr: "map($.status in ['ok','fail'], true)"

11) Anti-patterns

Shadow writes out: real payments/notifications out of the shadows.
Shared cache/shared queues: cross-impact and contamination.
Lack of normalization: byte diffuses are "red" due to clock/key order.
Too high percentage on the go: degradation of the pen prod.
Long "eternal shadow": the second system lives separately and diverges from reality.

12) Shadow mode launch checklist

• The proxy is configured with a mirror with a share and a redaction.
• Shadow isolated: DB/caches/external integrations - readonly/stub only.
• Correlation-ID is thrown everywhere; traces are linked.
• The comparator can ignore/normalize and check invariants.
• Dashboards and alerts for discrepancies and load.
• Sampling by routes/tenants; limits and back-pressure.
• Green light tolerances and criteria are defined.
• Transition to canary/blue-green and rollback plan.

13) FAQ

Q: How is Shadow different from A/B?
A: In A/B, both versions return answers to users (split experiment), in Shadow the new version does not affect the user - its answers are only analyzed.

Q: Can POST/PUT be shadowed?
A: Yes, if side effects isolation (stub) and idempotence are guaranteed. Often start with GET, then expand.

Q: How do I compare responses when the order of items is not fixed?
A: Sort by stable key before comparing or compare as sets/by keys.

Q: What to do with database replica delays?
A: Enter "comparison windows" and reference book snapshots; Aggregate results by version rather than by wall hour.

14) Totals

Shadow traffic is a "painless rehearsal of production": real load, zero risk for users, detailed analytics of discrepancies. Success is determined by isolation, correct sampling, quality comparator, and observability. Following the proposed plan, you will get a reproducible practice that confidently bridges the path to canary/blue-green releases and accelerates the evolution of architecture.