Workflow Engine

1) Why do you need an engine

• Predictable: explicit steps, statuses, SLAs and responsible.
• Reliable: idempotency, retrays, compensations, deadlines.
• Transparent: metrics, tracing, audit, provability for regulators.
• Effective: automation of routine + a person connects according to the rules.

2) Key principles

Orchestrate the critical, choreograph the rest: critical chains (payments/outputs/settle) - under centralized orchestration; non-critical events - through choreography (pub/sub).
Idempotency is everywhere: each step takes' idempotency _ key'and stores the results.
SLA-awareness: time per step and overall deadline are fixed; escalation by timers.
Compensate, don't rollback DB: for external effects - sagas/compensation.
Human-in-the-loop: formalized "narrow gates" (appruves, 4-eyes, SoD).
Policy-as-Code: routing, priorities, branch conditions - in policies.
Observability: Each task has an SLI/SLO, trails and audit.

3) Domain model

3. 1 Underlying entities

Process: Long-lived orchestration (minutes/hours/days).
Task: atomic operation (service/human).
Activity: process step with type (service/human/decision).
Signal/Event: external events (PSP webhook, KYC response, custom action).
Timer: deadlines, reminders, periodicals.
Context: secure payload of the process (tenant, region, KYC-id, limits, risk rate).

3. 2 Task states

`scheduled → running → (succeeded | failed | timed_out | cancelled | compensated)`

4) Architectural patterns

Process orchestrator: the central engine stores state, timers, queues, routing.
Workers: stateless services subscribed to domain task queues (Payments, KYC, Risk, Games).
Sagas: For each "strong" operation, there is an inverse (compensatory).
Outbox/Inbox: guarantees of "exactly-once" integration with external systems.
Command/Callback: tasks are initiated by commands; results - by sausages/webhooks.
Feature flags: dynamic branch selection (e.g. alternative PSP).
Tracing: Process' trace _ id'correlation with all calls.

5) Guarantees and sustainability

At-least-once task execution + handler idempotency.
Retrai with jitter and limited budgets (per-task, per-process).
Timeouts: 'task _ timeout' <step SLA; 'process _ deadline' <regulatory period.
Hysteresis and backoff: storm protection.

Circuit-breakers: stop retrays when the dependency is "red."

Grandfather Letter (DLQ): for manual disassembly of rare glitches with full context.

6) Catalog of typical processes (iGaming)

1. Deposit: init → 3DS/auth → capture → ledger → bonus credits → notice → antifraud check (asynchronously).

Compensations: cancellation/cancel, reversal, rebate return.

2. Withdrawal: request → risk scoring → 4-eyes app → payment gateway → payment register → notification.

Compensation: withdrawal cancellation, re-route, account freeze.
3. KYC/AML: document collection → provider 1 → fallback provider 2 → manual check → result/TTL.
4. Bet/Settle: Reservation → Factor Fix → Confirmation → Settle/Settlement → Payout.
5. Bonus campaign: targeting → coupon issue → activation → budget monitoring → expiration/cancellation.
6. Incident-process: detection → classification of P1-P4 → var-room → actions → closure of post-mortem →.

7) Task Spec

IDempotent key: 'task _ id' + business key (e.g. 'within _ id').
Preconditions: launch conditions (data, limits, flags).
Action RPC/HTTP/gRPC/queue command.
Result processing successful/partial/error/timeout.
Retrai: strategy (exp backoff + jitter), maximum attempts.
Compensation: reverse action/transition to a safe state.
Audit: what, by whom/what, when and why; before/after.

8) Human-in-the-loop

Built-in human-tasks: checklist, attachments, tips (runbook), RACI.
SoD/4-eyes: incompatible roles, two apps for P1/P2.
SLA: escalation during inactivity (timers, group change, auto-decline/approve in low-risk).
Communication: notifications to the desired channels, status page on P1/P2 through Comms Lead.

9) SLA, prioritization and scheduler

Priorities are P1 (immediate) → P2 → P3 (background).

Quotas: per-tenant/region/provider; protection against queue "capture."

Deadlines: one step and process; omission of deadline → compensation/escalation.
Periodicals: cron processes (closing registers, expiration of bonuses, reports to regulators).
Queues by QoS class: real time (A), operational (B), analytical (C).

10) Policies and DSL

Policy-as-Code: Rego/YAML/JSON-DSL for branches, PSP routing, SoD requirements, limits.
Versioning: migrating v1→v2 processes without interrupting active instances.
Canary policies: part of the traffic on the new branch; rollback by SLI.

11) Data, privacy and compliance

Minimizing context: in the process - only the necessary fields; PII - tokenized.
Geo-aware storage: by jurisdiction (GDPR and local rules).
TTL and retention: different for magazines, artifacts and documents.
Export: only by workflow with encryption, ticket and SoD.
Audit: non-replaceable logs (WORM), event connectivity.

12) Observability and quality control

SLI/SLO process: percentage of completions, average/95th duration, SLA violations.
Task metrics: success/error/retrays/timeouts, age in queue.
Traces: spans by steps, correlation with payments/game events.
Dashboards: Exec (SLA/error budget, bottlenecks), Ops (queues/lag, retrays, DLQ), Risk/Payments (PSP-branches, apps).
Anomalies: STL/CUSUM/CPD on duration and errors; auto-scale/feilover.

13) Cost (FinOps Workflow)

$/process instance, $/task, $/retray.
Optimizations: batching low-priority steps, aggregation of events, limits on long processes, cleaning old data.
Quotas: for launching/storing per-tenant; showback/chargeback.

14) Safety

IAM/ABAC: access to processes/tasks by roles and attributes (tenant/region/environment).
PAM/JIT: temporary privileges for manual steps.
Signature of webhooks and requests: HMAC/mTLS.
Protective actions: auto-block export PII in case of anomaly; dual control to sensitive branches (PSP routing, payment limits).

15) Integrations

Payment providers (PSP): commands/webhooks, fallback routing.
KYC/AML: providers, manual queues, regulatory deadlines.
Game providers: settle/reporting, processing channel delays.
Incident-platform/status-page: automatic creation/updating of maps.
Release-gates: blocking dangerous releases during "red" processes.

16) Template directory (DSL fragments)

yaml type: http id: payments_auth retry:
max_attempts: 5 backoff: exponential_jitter timeout: 2s idempotency_key: ${process. deposit_id}
on_fail: compensate: cancel_auth

yaml type: human id: withdrawal_approve sod: true approvers: [Risk, Finance]
sla: 2h on_timeout: escalate: L2

yaml saga:
do:  [reserve_funds, capture, ledger_post]
undo: [ledger_revert, refund_capture, release_funds]

17) Implementation Roadmap (8-12 weeks)

• Inventory of processes (deposit/output/CCM/settle), SLA goals, risk classes.
• Engine/approach selection (orchestrator + queues + state store).

• MVP: deposit and withdrawal as two sagas; idempotent handlers; DLQ; baseline metrics/trails.

• Human-tasks (4-eyes) for conclusions; Policy-as-Code for PSP routing timers and deadlines.

• Observability (SLO/dashboards), anomalies by duration, auto-scale workers; integration with incident platform/status page.

• Compliance: privacy/TTL/WORM audit; export-workflow; SoD/ABAC.

• Cost optimization, peak perf tests, tabletop exercises, template library.

18) KPI/KRI functions

SLA process execution, MTTP (mean time to process).
Proportion of automatic completions without manual involvement.
Retried/Task ratio, DLQ rate, Compensation rate.
Time of applications (human-tasks) and% of delay.
Cost: $/process, $/task, $/retray.
Risk signals: withdrawal/deposit anomalies, SoD inconsistencies.

19) Antipatterns

One monolithic process for "everything" is → difficult to scale and change.
Retrays without idempotency → duplicate payments/actions.
There are no deadlines/escalations → hanging conclusions/CCL.
PII storage in the context of a process without TTL and masking.
Compensation "on paper" without automation.
Lack of tracing and auditing → it is impossible to prove correctness.

Total

The workflow engine is a system for managing the lifecycle of business operations: orchestration of critical paths, sustainability (idempotency, retreats, sagas), formalized human participation, security and compliance policies, end-to-end observability and value control. This contour makes the iGaming platform predictable in spikes, fast in incidents and convincing for regulators and partners.