Service Mesh and Traffic Policy

1) Why Service Mesh

• Default security: mTLS, automatic issuance/rotation of certificates, service identities.
• Traffic policy: L7 routing, canary/AV tests, degradation and stability.
• Observability: metrics, logs, traces, gold signals on each call.
• Operations: uniform policies for all languages ​ ​/frameworks.

Difference from API gateway: gateway - north-south perimeter; mesh - east-west within the cluster/organization. Often work together.

2) Architecture: Planes and Patterns

Data Plane: sidecar proxy (Envoy/Linkerd-proxy/haproxy), which intercepts pod/VM traffic.
Control Plane: distributes configurations (routes, policies, certificates), stores status, publishes service drives.
Identity: usually SPIFFE ID and automatic X.509 certificates (SPIRE/built-in CA).
Entry/exit points: ingress/egress-gateway to monitor boundary flows.
Implementation modes: sidecar for each under; per-node/ambient-modes in new implementations.

3) Security policy and zero-trust

1. mTLS by default - servis↔servis encryption and mutual authentication.

• AuthN: trust only identities issued by CA mesh'a (SPIFFE).
• AuthZ: declarative "who can to whom and how" rules (RBAC/ABAC).

3. Isolation and egress control:

• Allowed domains/subnets; forced egress via egress-gateway.
• Blocking direct outcomes from the hearths.

4. Secret rotation: short-lived certificates, automatic proxy reconfiguration.

yaml apiVersion: security. istio. io/v1beta1 kind: PeerAuthentication metadata: { name: default, namespace: istio-system }
spec:
mtls: { mode: STRICT }

yaml apiVersion: security. istio. io/v1beta1 kind: AuthorizationPolicy metadata: { name: billing-allow, namespace: prod }
spec:
selector: { matchLabels: { app: billing } }
rules:
- from:
- source: { principals: ["spiffe://corp. local/ns/prod/sa/inventory"] }
to:
- operation: { ports: ["8080"], methods: ["POST"], paths: ["/proto. Billing/"] }

4) Traffic policy: persistence and routing

4. 1 Timeouts and Retreats

Timeouts: must be set on the call (connect/read/overall).
Retries: for idempotent operations only; backoff + jitter; limits per-try timeout.

yaml apiVersion: networking. istio. io/v1beta1 kind: VirtualService metadata: { name: orders }
spec:
hosts: ["orders"]
http:
- route:
- destination: { host: orders, subset: v1, port: { number: 8080 } }
timeout: 5s retries:
attempts: 2 perTryTimeout: 2s retryOn: "5xx,connect-failure,reset"

4. 2 Circuit Breaker и outlier detection

CB: restricts simultaneous requests/connections, protecting the upstream.
Outlier: throws out "bad" instances by error/latency.

yaml apiVersion: networking. istio. io/v1beta1 kind: DestinationRule metadata: { name: orders }
spec:
host: orders trafficPolicy:
connectionPool:
http: { http1MaxPendingRequests: 1024, maxRequestsPerConnection: 100 }
outlierDetection:
consecutive5xxErrors: 5 interval: 5s baseEjectionTime: 30s maxEjectionPercent: 50

4. 3 Canary and by conditions

Weighted - Traffic is divided by v1/v2 weights.
Header-based: flags/cookies/tenant → be transferred to the new version.
Session affinity: hash by key (neatly scaled).

yaml http:
- match: [{ headers: { "x-experiment": { exact: "new" } } }]
route: [{ destination: { host: orders, subset: v2 } }]
- route:
- destination: { host: orders, subset: v1, weight: 90 }
- destination: { host: orders, subset: v2, weight: 10 }

4. 4 Fault Injection and Degradation

Delay/error injection for robustness test and SLO.

yaml fault:
delay: { fixedDelay: 300ms, percentage: { value: 10 } }
abort: { httpStatus: 503, percentage: { value: 1 } }

5) Network boundaries: ingress/egress and external services

Ingress-gateway: the only entry point for external clients in mesh; integration with WAF/OIDC/ratelimits.
Egress-gateway: central output with a list of allowed hosts, TLS inspection, telemetry record.
ServiceEntry: declares external SNI/host as part of mesh (policies and mTLS-origination apply).

yaml apiVersion: networking. istio. io/v1beta1 kind: ServiceEntry metadata: { name: payments-external }
spec:
hosts: ["api. payments. com"]
ports: [{ number: 443, name: https, protocol: TLS }]
resolution: DNS location: MESH_EXTERNAL

6) Multi-cluster, multi-network and hybrid

Shared PKI/trust domain: single SPIFFE identities between clusters.
Endpoint discovery: service balls between regions; local priority and failover.
Zonal isolation: per-regional policies, limits and priorities.
VM in mesh: connecting legacy/Stateful systems to proxies and the same policies.

7) Observability, SLO and operation

Метрики: `requests_total`, `request_duration_ms{p50,p95,p99}`, `5xx_rate`, `retry_attempts`, `cb_state`, `mTLS_authz_denied`.
Access logs: structural, with 'traceparent', 'user/tenant', 'response _ flags'.
Tracing: auto-injection of headers (W3C Trace Context), sampling, spans at the hop level.
SLO: targets by p99/errors on the route (service→service).
Alerts: '5xx' spike, 'reset' rise, 'outlier _ ejections', mTLS degradation (handshake failures).

8) Performance and cost

• Graininess: include politics where needed; do not turn on heavy filters everywhere.
• Pools: split paths (critical/background) into different routes and limits.
• Profiling: p99 on "cold" routes, telemetry volume (rate-limit logs/trails).
• Consider ambient/sidecarless modes if supported and appropriate.

9) Safety and compliance

Minimum required access: Explicitly allow directions and methods.
Policies on neimspaces/tenants: network/authorization boundaries.
Key rotation/AC: planned and emergency; short TTL certificates.
PII/secrets: masking in logs/tracks; encryption on the wire/at rest.
Audit: who, when and what policy changed; two-stage uproars.

10) Integration with K8s layer

Mesh policies complement, not replace, NetworkPolicy.
In ingress/egress-gateway, you can hang PodSecurity/PSA at a level below.
NRA/autoscaling: consider retrays/CBs - they change load.
Release plans: canary weights via VirtualService + automatic SLO promotion.

11) Implementation checklist

• Trust bounds defined and STRICT mTLS enabled.
• AuthZ policies enabled: who to whom, on which ports/methods.
• Timeouts/retries and outlier detection are configured, idempotent paths are defined.
• Canary routes and rollback plan are registered; fault-injection - only in non-prod.
• External dependencies are derived through egress-gateway and ServiceEntry.
• Metrics, logs, traces are configured; dashboards and alerts on the p99/5xx/CB.
• Quotas/limits per-tenant/namespace are available.
• Prepared runbooks: certificate leak, CA failure, upstream degradation, mass 503/RESET.
• Multi-cluster plan (shared PKI, local priorities, DR scenarios).
• Test days (game days): drop control plane, stop sidecar, network break, poisonous upstream.

12) Anti-patterns

Mesh "everywhere and at once" without route inventory and SLO → expensive complexity.
Default retrays for all methods → effect duplicates and traffic avalanches.
A disabled mTLS "temporarily" → remains permanently.
Egress without gateway → data leakage/unaccounted dependencies.
One global policy for all services → false security and false positives.
Zero observability: included mesh, but do not collect metrics/trails - lose meaning.

13) Quick recipes

Linkerd: enable mTLS and policy by server

yaml apiVersion: policy. linkerd. io/v1beta1 kind: Server metadata: { name: billing, namespace: prod }
spec:
podSelector: { matchLabels: { app: billing } }
port: 8080 apiVersion: policy. linkerd. io/v1beta1 kind: ServerAuthorization metadata: { name: billing-allow-inventory, namespace: prod }
spec:
server: { name: billing }
client:
meshTLS:
identities: ["inventory. prod. serviceaccount. identity. linkerd. cluster. local"]

Consul (L7 intent + splitter)

hcl
Kind = "service-router"
Name = "orders"
Routes = [{
Match { HTTP { PathPrefix = "/v1" } }
Destination { Service = "orders" }
}]
Kind = "service-splitter"
Name = "orders"
Splits = [
{ Weight = 90, ServiceSubset = "v1" },
{ Weight = 10, ServiceSubset = "v2" }
]

14) FAQ

Does mesh need a small team?
If 3-5 services - more often not. Start with good ingress and resilience libraries. Connect mesh when there is a need for mTLS-default, uniform policies, and tracing without code changes.

How to control the cost?
Measure overhead (CPU/RAM/latency) on critical paths, turn off unnecessary filters, reduce the volume of logs/trails, use modes without sidecar where it is safe.

Is it possible to interfere with mesh and manually configured proxies?
Yes, but avoid double routing/duplicate retrays. A single place is true - the control plane.

What is more important: security or performance?
The default is security (mTLS, AuthZ). Performance is achieved by tuning connection pools, outlier detection, target routes.

15) Totals

Service Mesh turns the network between services into a programmable layer with uniform policies: encryption and identities, fine routing and resilience, telemetry and quotas. Start with critical paths, turn on STRICT mTLS and explicit AuthZ, set timeouts/retrays/CB, control egress, measure p99 and 5xx, spend game days. Then mesh will become an amplifier of reliability and speed of releases, and not a source of surprises.