Event-Driven kernel

What is the Event-Driven Kernel

The Event-Driven Core (EDC) is the "spine" of architecture, in which business facts are captured and distributed as immutable events, and the rest of the functionality (reading, integration, analytics, caches, notifications) is built on top of the flow of these events. The kernel sets the event contract, delivery rules, and order/idempotency invariants, providing weak connectivity and scalability.

The key idea: first write down the fact (core), and then independently enrich and project it into the desired models. This reduces connectivity and increases resilience to partial failures.

EDC Goals and Properties

Truth of facts: Each event is an immutable record of "what happened."

Weak connectivity: Producers don't know consumers; system expansion - by adding subscribers.
Scaling: horizontal growth by party/topic, independent consumers.
Observability and auditing: end-to-end identifiers, reproducibility, retentions and replay.
Managed evolution: schema versions, compatibility, deprecation.

Architectural components

1. Bus/event broker: Kafka/NATS/Pulsar/SNS + SQS - channels, parties, retentions.
2. Schema Registry: JSON Schema/Avro/Protobuf for compatibility and evolution.

3. Outbox/CDC contour: atomic fact fixing + publishing without "double write."

4. Projections/Reads (CQRS): materialized views for quick queries.
5. Sagas/orchestration: coordination of long-lived processes through events/teams.
6. Enrichment: individual topics' .enriched '/' .derived'without affecting the critical path.
7. Observability: tracing, logging, metrics by events and lags.

Event model

Event types

Domain Events: Business Facts ('payment. authorized`, `kyc. approved`).
Integration Events: focused on external systems (stable, slowly changing).
Change Data Capture (CDC): technical changes to the record (use for migrations/integrations).
Audit/Telemetry: actor actions, security, SLA.

Required Attributes (Core)

json
{
"event_id": "uuid",
"event_type": "payment. authorized. v1",
"occurred_at": "2025-10-31T11:34:52Z",
"producer": "payments-service",
"subject": { "type": "payment", "id": "pay_123" },
"payload": { "amount": 1000, "currency": "EUR", "method": "card" },
"schema_version": 1,
"trace_id": "abc123",
"partition_key": "pay_123"
}

Recommendations: 'event _ id' is globally unique, 'partition _ key' sets the order for the entity, 'trace _ id' provides correlation.

Delivery semantics and idempotency

At-least-once (the default for many brokers): Consumers are required to be idempotent.
At-most-once: acceptable only for secondary telemetry.
Exactly-once: achieved at the flow and storage level through transactions/idempotent keys/watering cans (more expensive, need a good reason).

Consumer idempotency pattern

Dedup table by 'event _ id '/' (event_id, consumer_id)' with TTL ≥ topic retention.
Upsert instead of insert; versioning projections by 'sequence '/' occurred _ at'.
Transactions within the transaction: mark "saw" + state change.

Order and partitioning

Guaranteed order within the party.
Select 'partition _ key' so that all events of the same aggregate entity fall into the same batch ('user _ id', 'payment _ id').
Avoid "hot keys": hash with salt/sub-keys if you need to distribute the load.

Schemes and evolution

Additive-first: new optional fields, prohibition on changing types/semantics without a major version.
Compatibility: BACKWARD/FORWARD in the schema registry; CI blocks incompatible changes.
Naming: 'domain. action. v{major}` (`payment. authorized. v1`).
Migrations: publish pairs' v1'and' v2'in parallel, provide dual-write through outbox, shoot' v1'after the transition.

Outbox и CDC

Outbox (recommended for transactional services)

1. In one database transaction: save the domain record and event to the outbox.

2. The background publisher reads the outbox, publishes to the broker, marks "sent."

3. Guarantees: no "loss" of fact in falls, no desynchronization.

CDC (Change Data Capture)

Suitable for existing systems/migrations; source - DB replication log.
Requires filtering/transcoding to domain events (do not translate raw tables outside).

CQRS and projections

Commands change state (often synchronously), events form projections (asynchronously).
Projections are designed for queries (search, lists, reports), updated by subscribers.
Temporary desynchronization is the norm: show a stable UX ("data will be updated in a few seconds").

Sagas: Process Coordination

Orchestration: One coordinator sends commands and waits for events.
Choreography: participants react to each other's events (simpler, but requires discipline in contracts).
Rules: clear compensations and timeouts, repeatable steps, idempotent handlers.

Observability

Trace/Span: Trace 'trace _ id '/' span _ id' through the headers when generating events.
Metrics: consumer lag, publishing/consumption rate, dead-letter rate, deduplication share.
DLQ/parking lot: unsuccessful messages - to a separate topic with alert; provide reprocessing.

Security and compliance

Data classification: the core contains only the required minimum of PII/financial data (reverse pyramid model), details - in enrichment.
Signature/hash of critical attributes, integrity control.
Encryption in-flight and at-rest, partitioning rights by topic/consummer (IAM/ACL).
Retention policies and rights to be forgotten: clearly defined for each topic.

Performance and stability

Backpressure: consumers have limited competition, the broker has quotas/limits.
Batch and compression-Group records to reduce overhead.
Retrai with jitter and DLQ instead of endless attempts.
Rebalance-resistance: store offsets transactionally/externally, speed up a cold start with snapshots.

Typical event templates

Payment Core

`payment. initiated. v1` → `payment. authorized. v1` → `payment. captured. v1` → `payment. settled. v1`

Refusals: 'payment. declined. v1`, `payment. refunded. v1`

Partitioning: 'payment _ id'

SLA: core lag ≤ 2c p95; Consumer idempotence is mandatory.

CCM/verification

`kyc. started. v1` → `kyc. document. received. v1` → `kyc. approved. v1`/`kyc. rejected. v1`

PII - minimal; document details - in'kyc. enriched. v1 'with restricted access.

Audit/Security

`audit. recorded. v1 'with attributes' actor ',' subject ',' action ',' occurred _ at ',' trace _ id '.

Continuous retention/archiving; Enhanced Integrity (WORM)

Antipatterns

Fat Event: overloaded payloads needlessly, PII leaks.
Hidden RPC through events: waiting for synchronous "here and now" responses.
Raw CDCs outward: Close connectivity to DB schema.
No idempotency in consumers: doubles lead to double side effects.
One common topic "for everything": conflict of interest, problematic order, complex evolution.

Step-by-Step EDC Implementation

1. Domain mapping: Highlight key aggregates and lifecycles.
2. Catalog of events: names, meanings, invariants, required fields.
3. Schemas and Registry - Select a format, enable compatibility rules.
4. Outbox/CDC: For each producer, define a mechanism for publishing facts.
5. Partitioning: Select keys and evaluate hot keys/repartition.
6. Idempotency: deduplication pattern + consumer transactionality.
7. Projections-Define materialized models and update SLAs.
8. Observability: tracing, lags, DLQ, alerts.
9. Security/PII: data classification, encryption, ACL.
10. Guide to evolution: version policy, deprecate, dual-write for migrations.

Production checklist

• Each event has' event _ id ',' trace _ id ',' occurred _ at ',' partition _ key '.
• Schemes in the registry, compatibility checks enabled.
• Consumer identity implemented and tested.
• DLQ/parking lot and alerts are configured for publish/consume errors.
• Projections are recreated from the log (replay) with an acceptable time.
• Limited access to PII; minimum payload's in the kernel.
• SLAs by lags/delivery are measured and visible on dashboards.
• There is a plan for migrating event versions and deprecate windows.

FAQ

How is EDC different from "just a bus"?
The core is not only the broker, but also the event contract, order/idempotency rules, evolution processes, and observability.

Can we only build on the CDC?
CDC is suitable for integrations/migrations, but domain events express meaning more clearly and experience database changes more stably.

What about consistency?
We accept the eventual consistency and design UX/processes for it (indicators of update, retray, compensation).

When is exactly-once needed?
Rare: when doubling is strictly unacceptable and impossible to compensate for. More often, at-least-once + idempotency is enough.

Total

The Event-Driven kernel turns the flow of business facts into a reliable foundation for the system. Clear event contracts, delivery discipline, and observability yield scalability, resilience, and rate of evolution - without the fragile synchronous connections and "storm" of regressions under change.