Idempotence and keys

What is idempotency

Idempotency is a property of an operation in which repeating with the same identifier does not change the final effect. In distributed systems, this is the main way to make the result equivalent to "exactly one processing," despite retrays, duplicate messages, and timeouts.

Key idea: each potentially repeatable operation should be marked with a key by which the system recognizes "this has already been done" and applies the result no more than once.

Where it matters

Payments and balances: write-offs/credits by 'operation _ id'.
Reservations/quotas/limits: same slot/resource.
Webhooks/notifications: repeated delivery should not duplicate the effect.
Import/Migration - Re-run files/packages.
Stream processing: duplicates from broker/CDC.

Types of keys and their scope

1. Operation key - ID of the specific attempt of the business transaction

Examples: 'idempotency _ key' (HTTP), 'operation _ id' (RPC).
Scope: service/aggregate; is stored in a deduplication table.

2. Event key - unique identifier of the event/message

Examples: 'event _ id' (UUID), '(producer_id, sequence)'.
Area: consumer/consumer group; protects projections.

3. Business key - natural domain key

Examples: 'payment _ id', 'invoice _ number', '(user_id, day)'.
Area: aggregate; used in uniqueness/version checks.

TTL and Retention Policy

TTL keys ≥ a possible redo window: log retention + network/process delays.
For critical domains (payments) TTL - days/weeks; for telemetry - hours.
Clean dedup tables with background jobs; for audit - archive.

Key stores (deduplication)

Transactional database (recommended): reliable upsert/unique indexes, joint transaction with effect.
KV/Redis: fast, convenient for a short TTL, but without a joint transaction with OLTP - careful.
State store stream processor: locally + changelog in broker; good at Flink/KStreams.

• idempotency_keys

`consumer_id` (или `service`), `op_id` (PK на пару), `applied_at`, `ttl_expires_at`, `result_hash`/`response_status` (опц.) .

Indexes: '(consumer_id, op_id)' - unique.

Basic implementation techniques

1) Effect + Progress Transaction

Record result and capture read/position progress in one transaction.

pseudo begin tx if not exists(select 1 from idempotency_keys where consumer=:c and op_id=:id) then
-- apply effect atomically (upsert/merge/increment)
apply_effect(...)
insert into idempotency_keys(consumer, op_id, applied_at)
values(:c,:id, now)
end if
-- record reading progress (offset/position)
upsert offsets set pos=:pos where consumer=:c commit

2) Optimistic Concurrency (unit version)

sql update account set balance = balance +:delta,
version = version + 1 where id=:account_id and version=:expected_version;
-- if 0 rows are updated → retry/conflict

3) Idempotent sinks (upsert/merge)

sql insert into bonuses(user_id, op_id, amount)
values(:u,:op,:amt)
on conflict (user_id, op_id) do nothing;

Idempotency in protocols

HTTP/REST

'Idempotency-Key: <uuid'hash>' header.
The server stores the key record and returns the same response again (or code '409 '/' 422' in case of invariant conflict).
For "insecure" POST, 'Idempotency-Key' + stable timeout/retray policy is required.

gRPC/RPC

Metadata 'idempotency _ key', 'request _ id' + deadline.
Server implementation - as in REST: a table of deduplication in a transaction.

Brokers/Streaming (Kafka/NATS/Pulsar)

Producer: stable 'event _ id '/idempotent producer (where supported).
Consumer: dedup by '(consumer_id, event_id)' and/or by business version of the aggregate.
Separate DLQ for non-idempotent/corrupt messages.

Webhooks and external partners

Demand 'Idempotency-Key '/' event _ id' in the contract; re-delivery must be safe.
Store 'notification _ id' and sending statuses; at retray - do not duplicate.

Key design

Determinism: Retrays must send the same key (generate in advance on the client/orchestrator).
Scope: Form 'op _ id' as' service: aggregate: id: purpose '.
Collisions: use UUIDv7/ULID or hash from business parameters (with salt if necessary).
Hierarchy: the general 'operation _ id' at the → front is translated into all suboperations (idempotent chain).

UX and Product Aspects

A repeated key request must return the same result (including body/status) or an explicit "already executed."

Show the user the statuses "operation is being processed/completed" instead of trying again "for good luck."

For long operations - polling by key ('GET/operations/{ op _ id}').

Observability

Log 'op _ id', 'event _ id', 'trace _ id', outcome: 'APPLIED '/' ALREADY _ APPLIED'.
Metrics: repetition rate, dedup table size, transaction time, version conflicts, DLQ rate.
Trace: the key must pass through the command → event → projection → external call.

Safety and compliance

Do not store PII in keys; key - identifier, not payload.
Encrypt sensitive fields in deduplication records with long TTL.
Retention policy: TTL and archives; right to be forgotten - through crypto-erasure of responses/metadata (if they contain PII).

Testing

1. Duplicates: run one message/request 2-5 times - effect exactly one.
2. Drop between steps: before/after recording the effect, before/after fixing the offset.
3. Consumer restart/rebalance: no dual use.
4. Competition: parallel queries with one'op _ id '→ one effect, the second -' ALREADY _ APPLIED/409 '.
5. Long-lived keys: Checks for TTL expiration and retries after recovery.

Anti-patterns

Random new key for each retray: the system does not recognize replays.
Two separate commits: first the effect, then the offset - the fall between them duplicates the effect.
Trusting only the broker: no deduplication in the bruise/aggregate.
Missing aggregate version: repeated event changes state a second time.
Fat keys: the key includes business fields/PII → leaks and complex indexes.
No repeatable responses: Client cannot safely retract.

Examples

Payment POST

Customer: 'POST/payments' + 'Idempotency-Key: k-789'.
Server: transaction - creates a'payment' and an entry in 'idempotency _ keys'.
Redo: returns same '201 '/body; in case of invariant conflict - '409'.

Bonus accrual (sink)

sql insert into credits(user_id, op_id, amount, created_at)
values(:u,:op,:amt, now)
on conflict (user_id, op_id) do nothing;

Projection from events

The consumer stores' seen (event_id) 'and' version'of the unit; repeat - ignore/idempotent upsert.
Read progress is captured in the same transaction as the projection update.

Production checklist

• All unsafe operations have an idempotent key and its scope defined.
• There are deduplication tables with TTLs and unique indexes.
• The effect and progress of reading are committed atomically.
• Optimistic competition (version/sequence) is included in the write model.
• API contracts capture 'Idempotency-Key '/' operation _ id' and repetition behavior.
• Metrics and logs contain 'op _ id '/' event _ id '/' trace _ id'.
• Tests for duplicates, falls and races - in CI.
• TTL/Archive policy and PII security are followed.

FAQ

How is' Idempotency-Key'different from' Request-Id '?
'Request-Id '- trace; it can be changed on retrays. 'Idempotency-Key' is the semantic identifier of the operation, which is required to be the same during repetitions.

Is it possible to do idempotence without a database?
For a short window - yes (Redis/in-process cache), but without a joint transaction, the risk of duplicates increases. In critical domains, it is better in one database transaction.

What to do with external partners?

Negotiate keys and repeatable responses. If the partner does not support - wrap the call in your idempotent layer and store "already applied."

How to choose TTL?
Sum the maximum delays: log retention + net/rebalance worst-case + buffer. Add stock (× 2).

Total

Idempotency is a discipline of keys, transactions, and versions. Stable operation identifiers + atomic fixation of effect and reading progress + idempotent sinks/projections give "exactly one effect" without transport-level magic. Make keys deterministic, TTL realistic and tests malicious. Then the retrays and duplicates will become routine, not incidents.