Time synchronization and drift

1) Why time is an architectural component

• break Kerberos/OAuth/JWT ('iat/nbf/exp' fields);
• distort metrics/trails and alerts;
• drop brokers/clients (timeouts, retrays, exponential backoff);
• disrupt order and idempotency in distributed scenarios.

• Offset - difference of local time from the reference.
• Skew - difference of offsets between nodes.
• Drift - Clock drift rate (ppm) in the absence of correction.
• Jitter - delay/measurement variability.

2) Sources and time protocols

2. 1 NTP (Network Time Protocol)

Strata (Stratum 1 - direct from GNSS/radio, Stratum 2 - from Stratum 1, etc.).

• slew (smooth frequency adjustment, safe for applications);
• step (time jump; undesirable on proda).
• Implementations: chrony, ntpd, systemd-timesyncd. For servers, it is preferred to chrony.

2. 2 NTS (NTP over TLS)

Authenticated synchronization (MITM protection and time spoofing).
Recommended for external time servers.

2. 3 PTP / IEEE 1588

Hardware timestamps in NIC/ToR, millisecond and microsecond precision.
Modes: boundary/transparent clock, telecom/enterprise profiles.
Use for hard SLOs on p99-order, HFT/telecom/industry.

2. 4 GNSS (GPS/GLONASS) and PPS

Local receivers give the PPS (pulse-per-second) reference for Stratum 1.
It is important to consider spoofing/jamming - install antennas and monitor integrity.

2. 5 Clouds

Cloud sources (internal stratum pools) reduce offset and jitter within the VPC.
For hybrid environments, combine local and cloud references.

3) Time in OS and hardware

TSC/HPET/RTC: modern CPUs hold TSC as a fast monotone counter; fix the frequency (invariant TSC).
Virtualization/containers: drift and "jump" more often. On the hypervisor - a strict time service; away - chrony.
Power saving can interfere with timer monotony - check BIOS/UEFI options.

4) Monotonous and "wall" watches

Wall-clock (real time, TZ/UTC) - for logs, event tags, people.
Monotonic clock - for measuring intervals/timeouts.

• Linux: `CLOCK_MONOTONIC`.
• C++: `std::chrono::steady_clock`.
• Go: built-in monotone parts' time. Time 'in intervals.
• Java: `System. nanoTime () 'for durations, not for calendar.

Rule: deadlines and retreats - on monotonous watches; serialization/logging - at UTC.

5) Leap second/" leap smear" and calendar traps

Leap second can cause "00:59:60" or repeat second → loops in timers/metrics.

• Smear (smooth smearing of a second in N hours).
• Step (undesirable).
• Never rely on local TZ/daylight saving time for logic; store UTC, show in the user's TZ.
• Update the TZDB (time zone base) - political changes happen.

6) Coordination of order without trust in the "wall"

Lamport clocks and Vector clocks are causal relationships without a physical clock.
Hybrid Logical Clocks (HLC) - combine physical time and counter, resistant to small skew.
TrueTime-like models - return the interval '[earliest, latest]' and require commit-wait for serialization.

7) Impact of time on protocols and systems

Security: Kerberos allows a small skew (usually ± 5 minutes), TLS/certificates are sensitive to 'notBefore/notAfter', JWT to 'exp/nbf/iat'.
Brokers/queues: task deadlines/visibility timeout depend on the correct time.
DBMS/clusters: version conflict by 'updated _ at '/ts - enter HLC/versions, not compare "raw" wall-timestamps.
Streaming: distinguish between event time and processing time; configure watermarks and lateness.
Kron/planners: drift leads to "sticking "/double starts. Use monotone intervals and dedup keys.

8) Observability and time SLO

8. 1 Metrics

`time. offset_ms' (offset to reference), 'time. jitter_ms`, `stratum`, `root_delay`, `root_dispersion`.
Для PTP: `path_delay`, `grandmaster_offset`, `gm_identity`, `clock_class`.
Alerts: offset> threshold (for example, 100-500 ms), source loss, step correction.

8. 2 Diagnostics

`chronyc tracking/sources/sourcestats`

`ntpq -p`, `ntpstat`

PTP: 'pmc', vendor NIC/ToR utilities.

8. 3 SLO/erroneous budget

SLO example: "median offset ≤ 1 ms, p99 offset ≤ 25 ms, no step on production nodes; PTP grandmaster failover ≤ 2 s».

9) Configuration Practices (Linux/containers/K8s)

9. 1 chrony (recommended)

pool time. example. org iburst maxsamples 9 nts makestep 0. 5 1 # one step at big error at start rtcsync # synchronize hardware clock leapsectz right/UTC # leap seconds from tzdata driftfile/var/lib/chrony/drift

• `maxsources`, `minsamples/maxsamples`, `maxslewrate`.
• For isolated DCs - local reference + GPS/PPS.

9. 2 Containers and assemblies

Synchronize on the host; containers use a core.
In K8s - DaemonSet with chrony or node-level time agent; Prevent applications from adding time.

9. 3 PTP stack

NIC with hardware timestamping, PTP daemon, boundary clocks on ToR.
PTP domain diversity (profiles), protection against "bad" grandmaster.

10) Time security

NTS/authenticated NTP, filters and rate-limit (NTP-gain - DDoS vector).
PTP security: L2 isolation, multicast ACL, GM spoofing monitoring.
GNSS: antennas with good visibility, spoofing/jamming detecta, fallback sources.

11) Engineering patterns and code

11. 1 Deadlines/Timeouts

Store deadlines as a "monotone start + delta" rather than an absolute wall-timestamp.
Always add stock to skew (for example, 2 × of the expected p99-skew to the TTL token).

11. 2 Version comparison

• versioning/ETag;
• HLC/seq;
• optimistic blockages.

11. 3 Logs and traces

Always UTC; include the'time _ offset _ ms' field of the host in the agent logs.
Glue event-time in trace events.

11. 4 Processing leap second

Select a policy (smear/step) uniformly across all nodes.
Test: Metrics should not "break" in a second.

12) Impact on domains

Auth: tokens - consider "clock skew allowance" (for example, ± 2-5 minutes).
Payments/Time segments: Round off intervals, not absolute time.
Brokers: Retray schedules - on monotonous watches.
DB/TTL: TTL in Redis/DB - relies on local clocks: lay down stock.
Analytics - Time Aggregation - Use a single UTC and ingestion synchronization.

13) Test playbooks (Game Days)

Drift injection: artificially take the clock to +/ − Δ; check auth, brokers, SLO.
NTP outage: disable sources, trace drift and auto-switch.
Leap second/smear: simulation of leap occurrence, evaluation of schedules/timers.
PTP GM failover: check switch time and offset after.
VM suspend/resume: make sure there are no "jumps" and step on guests.

14) Anti-patterns

Compare events of different nodes in wall time without HLC/seq.
Put "string locales" of time (with TZ) in the database instead of UTC.
Allow applications to do'date -s '/' timedatectl set-time'.
Include step corrections on the product without planning.
Ignore TZDB updates and daylight saving time rules.
Use wall-clock for backoff/timeouts/token-TTL without skew margin.
Trying to "heal order" with physical time instead of a logical clock.

15) Implementation checklist

• Single policy: NTP (with NTS) or PTP; list of trusted sources.
• Nodes are configured for slew, step only at start.
• Single leap second (smear/step) policy across clusters.
• Monitoring of offset, jitter, stratum/PTP indicators; alerts.
• Applications use monotone hours for intervals/deadlines.
• For order/conflicts - HLC/version, not wall-timestamps.
• Stocks on skew in TTL tokens, certificates, schedules.
• K8s/VM: synchronization on hosts, containers without rights to change time.
• Documentation and runbooks on time failures, game days in the CI/CD calendar.
• Regular TZDB updates, checking behavior on DST/leap events.

16) FAQ

Q: When is PTP needed instead of NTP?
A: When SLO requires microsecond-tens of microseconds (telecom/HFT/industry) and there is support for hardware labels in the network/cards.

Q: How much to put on clock skew?
A: For typical NTP in DC - tens to hundreds of ms (p99); Lay down 2 stock ×. With PTP - units-tens of μs.

Q: How to survive leap second?
A: Use smear and the same policy everywhere; Test schedules/aggregators and timers.

Q: Can you rely on wall-clock for deadlines?
A: No. Monotone hours only + stock per skew.

Q: How to store "time" in the database?
A: In UTC ('timestamptz'), plus/HLC versions for conflict resolution; Do not store local zones in data.

17) Totals

Reliable time is protocol + policy + discipline in code. Synchronize nodes (NTP/NTS or PTP), use monotone hours for intervals, UTC for data, HLC/versions for order, lay stocks on skew, monitor offsets and regularly spend game days. This will avoid "mystical" authentication bugs, event discrepancies and unstable SLOs.