Logging and Logging

1) Why do we need logs and protocols

Logs are the organization's "black box": they provide evidence for audits and investigations, reduce operational and regulatory risk, allow you to restore the course of events and confirm the execution of policies (access, retention, encryption, KYC/AML, PCI, etc.).

• Tracing actions (who/what/when/where/why/what).
• Incident detection and containment (detective and preventive controls).
• Evidence base for regulators/auditors (immutability).
• SLA/SLO performance and compliance analytics.

2) Taxonomy of logs (minimum coverage)

Accesses and Identities (IAM/IGA): Authentication, Role Reversal, SoD, JIT Accesses.
Infrastructure/cloud/IaC: API calls, configuration drift, KMS/HSM events.

Applications/Business - Transactions, PI/Financials, Query Life Cycle (DSAR)

Security: IDS/IPS, EDR, DLP/EDRM, WAF, vulnerabilities/patches, antivirus.
Network: firewall, VPN/Zero Trust, proxy, DNS.
CI/CD/DevSecOps: builds, depla, SAST/DAST/SCA, secret scan.
Data/analytics: lineage, storefront access, masking/anonymization.
Operations: ITSM/tickets, incidents, change-management, DR/BCP tests.
Vendors/3rd-party: webhooks, SSO federation, SLA events.

3) Regulatory requirements (guidelines)

GDPR/ISO 27701: minimization/masking PI, retention on schedule, Legal Hold, DSAR tracing.
SOC 2/ISO 27001: audit trails, log access control, evidence of control execution.
PCI DSS: logging access to media/card data, log integrity, daily review.
AML/KYC: traceability of checks, sanction/PEP screening, STR/SAR protocols.

4) Reference architecture of logging

1. Producers: applications, cloud, network, host agents.
2. Bus/collectors: back-pressure, retry, TLS mTLS, deduplication.
3. Normalization: single format (JSON/OTel), enrichment (tenant, user, geo, severity).

• Hot (search/SIEM): 7-30 days, fast access.
• Cold (object): months/years, cheap storage.
• Archive-evidence (WORM/Object Lock): immutability, hash receipts.
• 5. Integrity and signature: hash chains/merkley-tree/timestamps.
• 6. Access and security: RBAC/ABAC, segmentation by jurisdiction, case-based access.
• 7. Analytics and alerts: SIEM/SOAR, correlation ID, playbooks.
• 8. Catalogs and schemas: event type registry, versioning, schema tests.

5) Policies-as-code (YAML examples)

Retention and Legal Hold

yaml id: LOG-RET-001 title: "Access logs retention"
scope: ["iam. ","app. access"]
retention:
hot_days: 30 cold_days: 365 worm_years: 3 legal_hold: true # when Legal Hold is active, block privacy removal:
pii_mask: ["email","phone","ip"]
review: "annual"

Integrity and signature

yaml id: LOG-INT-001 title: "Signature and commercial fixation"
hashing: "SHA-256"
anchor:
cadence: "hourly"
store: "s3://evidence/anchors"
verification:
schedule: "daily"
alert_on_failure: true

6) Log quality requirements

Structuring: JSON/OTel only, no raw text.
Time synchronization: NTP/PTP, drift control; 'timestamp ',' received _ at'entry.
Correlation IDs: 'trace _ id', 'span _ id', 'request _ id', 'user _ id' (alias).
Field semantics: data dictionary and test schema contract.
Localization/language: fields - English keys, values ​ ​ - unified (enum).
Volume and drop policy: prohibition of uncontrolled drop; queues/quotas/risk sampling.
Sensitive data: masking/tokenization; prohibition of keeping secrets/cards entirely.

7) Privacy and minimization

PII hygiene: log hashes/tokens instead of values; strict mask for email/phone/IP.
Context: Do not pay with personal data without reason.
Jurisdictions: storage and access by country (data residency), traceability of copies.
DSAR: search labels and export by case; ability to print reports with depersonalization.

8) Immutability and evidence

WORM/Object Lock - prevent deletion/overwriting in the period.
Crypto signature: signature of batches; Merkli roots with daily anchoring.
Chain of custody: access log, hash receipts, quotas in reports.
Verification: periodic integrity checks and out-of-sync alerts.

9) Log access control

RBAC/ABAC: read/search only roles vs export/sharing.
Case-based access: access to sensitive logs - only as part of an investigation/ticket.
Secrets/keys: KMS/HSM; rotation, split-knowledge, dual-control.
Access audit: a separate magazine "who read which logs" + alerts to anomalies.

10) Metrics and SLO logging

Ingestion Lag: 95th percentile of reception delay (target ≤ 60 seconds).
Drop Rate: percentage of events lost (target 0; alert> 0. 001%).
Schema Compliance:% of events validated by schema (≥ 99. 5%).
Coverage:% of systems under centralized logging (≥ 98% critical).
Integrity Pass: successful hash chain checks (100%).
Access Review: monthly claim of rights, delay - 0.
PII Leak Rate: detected "clean" PIs in the logs (target 0 critical).

11) Dashboards (minimum set)

Ingestion & Lag: volume/speed, lag, drop, hot springs.
Integrity & WORM: anchoring status, verifications, Object Lock.
Security Events: critical correlations, MITRE card.
Access to Logs: Who and What Read/Exported; anomalies.
Compliance View: retention/Legal Hold statuses, audit reports, DSAR exports.
Schema Health: parsing errors/schema versions, percentage of legacy agents.

12) SOP (standard procedures)

SOP-1: Log source connection

1. Source and criticality registration → 2) selection of the/OTel → 3) TLS/mTLS scheme, tokens →

2. dry-run in staging (validation of schemes, PII masks) → 5) connection in production →

3. adding to directories/dashboards → 7) verification of retention/WORM.

SOP-2: Incident response (logs as evidence)

Detect → Triage → case-scope → Legal Hold →

Hash Capture and Anchoring → Analytics/Timeline → Report and CAPA → Lesson Release.

SOP-3: Reg Request/Audit

1. Open the case and filters by request ID → 2) export to the required format →

2. Legal/Compliance verification → 4) hash summary → 5) sending and logging.

SOP-4: Log access revision

Monthly certification of owners; auto-roar of "orphaned" rights; SoD report.

13) Formats and examples

Access Event Example (JSON)

json
{
"ts": "2025-10-31T13:45:12. 345Z",
"env": "prod",
"system": "iam",
"event": "role_grant",
"actor": {"type": "user", "id": "u_9f1...", "tenant": "eu-1"},
"subject": {"type": "user", "id": "u_1ab..."},
"role": "finance_approver",
"reason": "ticket-OPS-1422",
"ip": "0. 0. 0. 0",
"trace_id": "2a4d...",
"pii": {"email": "hash:sha256:..."},
"sign": {"batch_id":"b_20251031_13","merkle_leaf":"..."}
}

Detection rule (pseudo-Rego)

rego deny_access_if_sod_conflict {
input. event == "role_grant"
input. role == "finance_approver"
has_role(input. subject. id, "vendor_onboarder")
}

14) Roles and RACI

15) Vendor and Supply Chain Management

In contracts: the right to audit logs, formats, SLA storage and access, WORM/immutability.
Sub-processors: source register and "end-to-end" retention.
Export/offboarding: confirmation of destruction and hash summary report.

16) Antipatterns

Logs in "free text," without diagrams and correlation.
Storage without WORM and hash fixation is a dispute in the audit.

Sensitive data in logs "as is."

There is no synchronization of time and normal trace_id.
Event drop at load peaks; lack of back-pressure.
Universal access to logs without case control.
"Eternal" rights to read logs, without re-certification.

17) Checklists

Starting the logging function

• Source taxonomy and criticality identified.
• Retention schemes and policies/Legal Hold declared (as-code).
• TLS/mTLS, tokens, auto-update agents.
• PII masks/tokens tested.
• WORM/Object Lock and anchoring are enabled.
• Dashboards/alerts/metrics are established.
• Access revision and SoD are configured.

Before Audit/Reg Request

• "audit pack" collected: schemas, policies, integrity reports, samples.
• Checks integrity and access logs for the period.
• DSAR/Legal Hold statuses confirmed.
• A hash summary of uploads and send confirmation has been generated.

18) Maturity model (M0-M4)

M0 Manual: scattered logs, no schemes and retentions.
M1 Centralized collection: basic search, partial taxonomy.
M2 Managed: schemas and policies-as-code, dashboards, retention/WORM.
M3 Integrated: OTel tracing, SOAR, anchoring/merkly, case-based access.
M4 Assured: "audit-ready by button," predictive detections, automatic integrity control and legally significant receipts.

19) Related wiki articles

Continuous Compliance Monitoring (CCM)

KPIs and compliance metrics

Legal Hold and Data Freeze

Policies and Procedures Lifecycle

Communication of compliance solutions

Compliance Policy Change Management

Due Diligence and Outsourcing Risks

Total

A strong logging function is not a "message warehouse," but a managed system: structured events, strict schemes and permissions, invariability and signature, default privacy, tight access control and replication into evidence. Such a system makes investigations fast, audits predictable and risks manageable.