Key and token rotation

1) Why rotation is needed

Keys and tokens inevitably "age": exposure in logs/backups, insider risks, library vulnerabilities, leaks from partners. Rotation reduces the "risk lifetime" and gives controllability in incidents. The goal is to build predictable rotation cycles and rapid recall mechanisms without downtime.

2) Area: what exactly we rotate

Signature/encryption keys: JWT (JWS/JWE), OAuth/OIDC, SAML, webhooks (HMAC), licenses.
Integration secrets: API keys, client secret, tech passwords. users.
TLS/mTLS: server/client certificates, root/intermediate CAs.
Data keys: KEK/CMK in KMS/HSM, DEK (envelope encryption).
Токены: access/refresh, service-to-service (mTLS, HMAC), short-lived session.

3) Storage, versions, labels

KMS/HSM/Vault as a source of truth. It is forbidden to store private keys in git/ENV/image files.
Versioning: 'key _ id '/' version' + labels: 'purpose = jwt-sign', 'env = prod', 'alg = ES256', 'created _ at', 'rotates _ at'.
Access policies: the principle of minimum necessary rights (least privilege), separation of duties (SoD).
Audit: who created/read/signed; immutable logs.

4) Basic rotation patterns

4. 1 Overlapping windows (graceful rollover)

We publish the new key → in JWKS/distribute the certificate.
Overlap window: validation with old and new keys, signature - only with new ones.
After the grace period expires, delete the old one from the trusted set.

4. 2 Dual-run

A short period when part of the instances signs old, part - new (for large fleets).
Requires a strictly synchronized JWKS and monitoring of the percentage of validations by 'kid'.

4. 3 Rotate-on-schedule vs rotate-on-use

Scheduled: once every N days/weeks (signature keys, TLS).
When using: refresh tokens - one-time, for each exchange release new ("sliding" rotation).

5) JWT/JWKS: Practice

5. 1 Headings and identifiers

Use 'kid' in the JWS header to select a verification key.
Minimum clims, short 'exp', correct 'aud/iss/nbf'.

json
{ "alg": "ES256", "kid": "jwt-2025-10", "typ": "JWT" }

5. 2 JWKS publication

JWKS must contain all active validation keys (old + new in grace window).
Client JWKS caching: short TTL (e.g. 5-15 min).
If compromised, remove the compromised key from the JWKS (abruptly), force-disability of the cache.

json
{
"keys": [
{ "kty":"EC","crv":"P-256","kid":"jwt-2025-10","use":"sig","alg":"ES256","x":"...","y":"..." },
{ "kty":"EC","crv":"P-256","kid":"jwt-2025-07","use":"sig","alg":"ES256","x":"...","y":"..." }
]
}

5. 3 Cadence and timing

JWT signature: key rotation every 3-6 months (or more often for high-risk).
'exp'access-token: 5-30 min; refresh - 7-30 days (with "rotate-on-use").
Forced "bonding" with PoP/DPoP (see § 8) to reduce theft risk.

6) HMAC rotation (webhooks/signatures)

Keep active and canary secrets; accept signatures from both.
Titles: 'X-Signature' + 'X-Timestamp'; window limitation ± 300s.
Complete disconnection of the old one - after the sender has switched confirmed.
For partners, publish the switch date-time and endpoint checks.

7) TLS/mTLS and trust chains

ACME/auto-renew for public server certificates (Let's Encrypt or enterprise CA).
mTLS: short client certificates (7-30 days), automatic rotation through channels (SPIFFE/SPIRE/mesh).
Intermediate/root CA rotation - only through overlapping trust bundle anchors and long canary.
Keep an eye on OCSP/CRL and clock-skew. In the logs - the reasons for the validation failure.

8) PoP/DPoP and client token↔klyuch bundle

DPoP (Demonstration of Proof-of-Possession): the token is bound to the public-key of the client; reduces the risk of replay.
Client key rotation = release of a new DPoP key, tokens - for a short time.
For service-to-service, mTLS is preferred (the device/worker "carries" the key in the HSM/TPM).

9) Refresh tokens: rotate-on-use

One-time refresh tokens: each exchange → a new refresh + access.
List of recalled 'jti '/' sid' store with TTL = lifetime refresh.
Reuse detection (re-play): immediate session/device recall, alert.

10) Recall and block lists

JWT without introspection: use short 'exp' + 'blacklists' 'jti' for critical cases (locally/in Redis, hash sharding).
OAuth introspection: centralized status server cache "active = false/true" with a short TTL.
API keys: store key hash (like passwords), owner/tenant labels, scope, creation/last access date; recall - instant.

11) Data keys: envelope encryption

CMK/KEK (KMS/HSM) protects DEK; CMK rotation occurs without data re-disclosure: re-wrap DEK.
DEK for each object/tenant/party; KDF/HKDF for derived keys.
Crypto-shredding policies: deleting KEK = unreadable data when compromised.

12) Incident procedures (compromise)

1. Freeze: disable token issuance on a compromised key, transfer issuance to a new one.
2. Revoke: remove 'kid' from JWKS, revoke certificates (OCSP/CRL), block API keys from the list.
3. Reduce TTL: temporarily reduce 'exp' tokens, strengthen PoP/DPoP checking.
4. Forced logout: disable sessions (revoke 'sid '/' jti').
5. Forensics and reporting: timelines, coverage, who/what suffered; update playbooks.

13) Pipeline and rollout

13. 1 Generation and publication

Generate keys in HSM/KMS; private key export - prohibited.
Automatic publication of JWKS/certificates with verification and tests.
Canary release: 1-5% of clients → 100%.

13. 2 Health control

Metrics: the proportion of validations by 'kid', signature/certificate errors, clock drift.
Alerts: 401/403 spike due to signature, OCSP/CRL unavailable, expiring certificates (T-30/T-7/T-1).

14) Configs and examples

14. 1 Vault/KMS Policy Example (Pseudo)

hcl path "transit/keys/jwt-prod" {
capabilities = ["read," "update," "list"] # signature/rotation
}
path "transit/keys/jwt-prod/rotate" {
capabilities = ["update"]
}

14. 2 JWT Rotation Plan Example

T0: create a new version of the key (kid = jwt-2025-10), add to JWKS
T0 + 15m: start signing with a new kid; validate with old and new
T0 + 7d: remove old kid from JWKS
T0 + 30d: delete old private key from KMS (schedule purge)

14. 3 Envoy: force JWKS update (pseudo)

yaml jwt_authn:
providers:
oidc:
issuer: https://auth. example. com/
remote_jwks:
http_uri:
uri: https://auth. example. com/.well-known/jwks. json cluster: jwks_cluster timeout: 2s cache_duration: 300s # короткий TTL

15) Observability and audit

Метрики: `jwt_verify_fail_total{reason}`, `jwks_refresh_total`, `jwks_kid_share{kid}`, `token_revoked_total`, `refresh_rotations_total`, `dpop_fail_total`.
Логи: `kid`, `jti`, `sid`, `reason`, `client_id`, `tenant`, `trace_id` (без PII).
Dashboards: 'kid' share card, expiring certificates, recall frequency, invalid signatures by region.

16) Antipatterns

Long-lived JWTs with no recall and no short 'exp'.
Absence of'kid 'and "manual" selection of the verification key.
Storing secrets in ENV/k8s-Secret without KMS and without encryption at the etcd level.
Non-rotational refresh tokens; reuse refresh without detection.

A single global API key "for all."

"Quiet" release of new keys without JWKS publication and monitoring.
Zero overlap windows (instant replacement) → mass 401/403.

17) Specifics of iGaming/Finance

Regulators and audit: unchangeable logs of rotations/recalls; provability of time and actors.
Partner PSP/KYC: separate keys per partner/jurisdiction; rapid recall for SLA/safety violations.
Multi-lease: per-tenant API keys with scope; brand/region key isolation.
High risk: PoP/DPoP for critical operations, short'exp ', mTLS between internal services.
Backoffice: SSO/OIDC, short sessions, hardware tokens (FIDO2), ubiquitous rotate-on-schedule.

18) Prod Readiness Checklist

• All private keys in KMS/HSM/Vault; export prohibited.
• JWKS is published and cached with short TTL; there is a 'kid' in JWT headlines.
• Rotation plan with overlapping window and automatic rollout.
• Refresh tokens are disposable; list of revoked 'jti' s with TTL.
• HMAC secrets: active + canary; reception by both; T-switch time declared.
• TLS/mTLS: auto-renew, T-30/T-7/T-1 alerts, trust bundle for CA change.
• Envelope encryption: KEK/CMK rotated without downtime, DEK per object/tenant.
• Metrics/alerts by signature, JWKS, feedback; dashboards' kid '-deals.
• Incident playbook (compromise) and regular drills.
• Tests of canary and validation replicas with new keys/CA.

19) TL; DR

Keep keys in KMS/HSM, sign JWT with 'kid' and post JWKS. Rotate keys and certificates with overlap, monitor validation shares by 'kid'. Refresh - rotate-on-use and short 'exp'; for critical operations - PoP/DPoP and mTLS. For data, use envelope encryption with KEK rotation without downtime. Implement metrics/alerts, incident playbooks, and regular canary rotations.