Auditing and immutable logs

Auditing and immutable logs

1) Why do you need it

The purpose of the audit is to capture "who, what, where, when and why" with provable integrity in order to maintain security, investigations, financial reporting and compliance.
Immutable log - a format and storage where events are recorded only by append-only, and subsequent change/deletion is either impossible or detectable by cryptographic means and storage policies.

2) Threat model and control objectives

• Intentional editing/deletion of events by an insider.
• Time/source substitution (replay/backdating).
• "Quiet" logging shutdown on the node.
• Loss of part of the log during accidents/migrations.
• Excessive PII collection and discord with privacy.

1. Integrity: proven by protection against modifications/deletions.

2. Completeness: closure of key flows (control plane, data plane, access, money).

3. Time accuracy: reproducible, synchronized time.

4. Accessibility: reading and searching within SLO.

5. Privacy: PII minimum, tokenization/encryption, legality of use.

3) Taxonomy and event priorities

• Control Plane: authentication/authorization, configuration changes, key operations (KMS), secret management, policy changes.
• Data Plane: access to objects/records/checks/payments; read/create/delete.
• Admin and DevOps: SSH/console, CI/CD, infrastructure/IaC changes, rights escalation.
• Product: transactions, billing, customer operations.
• System/network: kernel/agents/proxies/balancers, brokers, databases.
• Security: IDS/IPS/EDR, WAF, anti-DDoS, antifraud, DLP.

For each class, we fix: criticality, scheme, mandatory fields, shelf life, requirements for invariability.

4) Required fields and format

Correlation IDs are 'trace _ id', 'span _ id', 'request _ id', 'actor _ id' (user/service), 'tenant _ id', 'resource _ id'.
A&A context: authentication method, roles/policies at the time of action.
Time: RFC3339/UTC, milliseconds/nanoseconds; synchronization source.
Action and result: type of operation, purpose, status, number of affected objects.
Integrity: local HMAC records, sequence number, hash-prev.
Schema: JSON with a stable model (for example, compatible with common event dictionaries).

Prohibited: secrets, keys, tokens, full PAN, passwords, private keys. PII - only when necessary, with masking/tokenization.

5) Time and synchronization

Time source: at least two independent sources (NTP/PTP) + bias monitoring.
Critical time signatures: Use trusted time stamp (TSA) services or an internal time-sealing service for batches of events.
Rules: no local time zones, UTC only; log and offset/quality of time.

6) Log stream architecture

Agents → Buffer → Transport → Landing → Hash Chain/Signature → Cold/Archive → Index to search.

Collection on the node: light agents (daemonset/sidecar) with a buffer on the disk (back-pressure).
Transport: protected channel (TLS/mTLS), guaranteed delivery (at-least-once), idempotent-ingest.
Landing zone: object storage in a "raw" form (batches by date/tenant/type).
Index: search/analytics engine for online queries (hot layer).
Archive (WORM): immutable buckets/tapes with retention policies and legal hold.
Anchor/Seal: periodic "sealing" of packs of hash chains (see below).

7) Cryptographic immutability

7. 1 Chains of hashes (append-only)

Each entry contains: 'hash _ curr = H (record)', 'hash _ prev' from the previous entry, 'seq'. Any edit breaks the chain.

prev = GENESIS for record in stream:
payload = canonical_json(record_without_integrity_fields)
h = H(payload          prev.hash          record.seq)
store(record + {hash_prev: prev.hash, hash_curr: h})
prev = {hash: h}

7. 2 Signature of packs and time stamp

Add a TSA timestamp and publish the "Transparency Log."

Every N seconds/MB we form a block: Merkle root of all 'hash _ curr'.
We sign the block with the audit key (stably stored in KMS/HSM).
Optional: periodically anchor the root hash to an external provable space (for example, an independent journal or public unchangeable storage).

7. 3 Audit Key Management

Signature keys - in KMS/HSM, rotation on schedule, multifactor access, dual-control for export.
Key revocation → new trust branch; old signatures remain verifiable.

8) Retention and WORM policies

WORM/immutability: include immutable containers/buckets with Retention and Legal Hold policies for P0/P1 classes.
Versioning: enabled; deletion - only for procedures with a delay (prohibition of instant purge).
Retention: hot (7-30 days), warm (3-6 months), cold/archive (1-7 years or more - depending on the regulator/contract).
Multi-tenancy: separate namespaces/accounts/encryption keys per tenant; log access reporting.

9) Privacy and minimization

Collection according to the principle of necessity: we do not log unnecessary.
Tokenization/pseudonymization of sensitive fields, salt hash for identifiers.
Producer-side field encryption (AEAD) when stored in shared object storage.
The right to delete (where applicable): is implemented through crypto-erasure of field/part keys, without violating the invariability of the container (planned during design).

10) Access, roles and audit of the audit itself

Split: producers ≠ readers ≠ admins.
Read-only from WORM; change of retention policies - through separate roles and procedure with approval.
All read/export operations are logged to a secondary log (meta audit).
Export for investigation/compliance - in encrypted form with a hash block directory and a trust chain.

11) Observability and SLO

Metrics: injection rate, lag to index,% lost/repeated, fraction of unsynchronized time, signing/anchoring errors, buffer filling.
SLO: ≥99. 9% of events delivered ≤ X seconds to hot index; 0 unexplained "holes" in sequences; 100% of blocks are signed and time-stamped.
Alerts: injection pause> N minutes, invalid-hash growth, chain divergence, signature/timestamp failure, time offset beyond the threshold.

12) Testing and verification

Red/Blue tests: an attempt to edit/delete a record at different stages; detection check.

Chaos: disabling the agent on the node, breaking the network, buffer overflow, "time spoofing."

Crypto checks: regular validation of chains, reconciliation of Merkle roots and stamps.
Forensics: playing investigative scripts from end-to-end logs.

13) Operation and procedures

Runbook "integrity check" (on-demand and scheduled).
Procedure for legal hold and temporary "freezing" of parties.
Discovery and export procedure while maintaining the chain of trust.
Plan for rotation of audit keys and response to compromise (new trust branch, re-signing of blocks, notifications).

14) Mini recipes

records = read_partition(ts_window)
leaves = [H(canonical_json(r)) for r in records]
root  = merkle_root(leaves)
sig   = KMS.sign(root          ts_now)
tsa   = TSA.timestamp(sig)
store_block({root, sig, tsa, count=len(leaves), window})
append_transparency_log(H(root          sig          tsa))

for i in 1..N:
assert rec[i].hash_prev == rec[i-1].hash_curr assert rec[i].hash_curr == H(canonical_json(rec[i]_no_hash)          rec[i].hash_prev          rec[i].seq)

• Control/Data plane P0: hot 30 days → warm 6 months → archive 7 years (WORM).
• DevOps: hot 14 days → warm 3 months → archive 1 year.
• Securiti signals: hot 90 days (for investigations), then 1-3 years.

15) Frequent errors

"Logs are text without a schema." Without a schema, there is no deterministic integrity and search; canonical JSON and fixed fields are required.
No correlation. The absence of 'trace _ id' breaks down investigations.
Local time. UTC only and offset control.
Writes to modifiable volumes. Without WORM, any immutability is a fiction.
Do not log reads. Reading sensitive data is important to fix no less than writing.
Secrets in the logs. Turn on sanitizers before sending, "red lists" of patterns.
One key for everything. Signing and encryption keys - separately, with a role and rotation.

16) Compliance and regulation

Requirements for shelf life/immutability depend on the domain (finance, payment, telecom, etc.).
Provability: availability of WORM/retention protocols, circuit validation reports, log access logs, legal hold and export procedures.

17) Checklists

Before selling

• Event taxonomy and schema approved (required fields).
• Agents, buffers, protected transport, back-pressure configured.
• Included: hash chains, block signature, timestamp, transparency log.
• WORM/presentation storage is enabled; test for the inability to overwrite/delete.
• Masking/tokenizing sensitive fields.
• Time synchronization and offset monitoring.
• Rotation plan and storage of audit keys in KMS/HSM.

Operation

• Weekly validation of chains and blocks (+ report).
• Circuit Break/Signature Error/Time Offset Alerts.
• Periodic Red-team substitution/deletion tests.
• Regular review of retentions and costs.

18) FAQ

Q: Is just "just adding" at the database level enough?
Oh no. We need cryptographic guarantees (chains of hashes, block signatures, time stamps) and WORM policies.

Q: What about the right to delete data?
A: Design crypto-erasure (key removal) for fields/parties, keeping the media unchangeable and logs provable.

Q: Do I need a separate key to sign the blocks?
Oh, yeah. Separate block signing keys from storage encryption keys; store in KMS/HSM, with rotation and auditing.

Q: Is it possible to "anchor" into a public space?
A: Optional. This enhances verifiability and will cut off "rewriting history" within the circuit.

• "At Rest Encryption"
• "In Transit Encryption"
• "Secret Management"
• "Key Management and Rotation"
• "Sign and Verify Requests"