SLO-burn alerta on Payments/Bets
操作路線圖
1）為什麼需要它
操作路線圖（Ops Roadmap）將不同的SRE/平臺/支持任務和域命令轉變為一個透明的計劃：我們在每個季度都會對SLO/成本/事件產生什麼影響,以及付出什麼代價（人員、時間、預算）。這減少了混亂，簡化了技術債務，加快了向企業交付價值。
目標是：- 結合圍繞可測量結果（SLO、MTTR、Cost/RPS、Risk）的舉措。
- 協調平臺、域和外部提供商之間的優先級。
- 預算資源並記錄「我們不做什麼」（明確交易）。
- 保持關於績效和風險的統一真相。
2）路線圖原則
1.Outcome-first：每個計劃都與結果度量掛鉤（不是「實施X」,而是「將MTTR降低20%」）。
2.SLO-aware：影響關鍵途徑SLO（存款/投註/遊戲/KUS）的舉措優先級更高。
3.數據驅動：依靠事件、驗屍、異形、Capacity/FinOps面板。
4.時間框和可逆的：小的插值，假設驗證，快速回滾。
5.單一真相來源：一個人工制品，定期的咆哮和公共地位。
6.沒有隱藏的工作：在地圖之外-根據規定，只有「火災」。
3）車架Roadmap： 水平和人工制品
願景（12-18個月）：3-5個操作主題（可信度、規模、成本、安全、自動化）。
支柱（6-12個月）：主題計劃塊（例如「100％關鍵路徑的SLO覆蓋」，「2個地區的主動活動」）。
季度計劃（Q）：具有指標，所有者，從屬關系，預算的具體計劃。
叠代（2-3周）：任務/史詩和實際進展。
ID: OPS-23
4) Prioritization: How to compare the incomparable
4. 1 RICE (Reach, Impact, Confidence, Effort)
Reach: affected users/transactions/geo.
Impact: expected contribution to SLO/MTTR/Cost.
Confidence: Confidence in estimates (data/pilots).
Effort: man-weeks/calendar window/dependencies.
4. 2 WSJF (Scaled)
Cost of Delay = (SLO Risk + Revenue Impact + Compliance + Incident Rate)
/ Job Size = duration/force.
Suitable for mixed initiatives (technical debt, security, platform features).
The rule: initiatives with high SLO risk and high cost of delay come first, even if the effect is "invisible" on UI.
5) Relationship with OKR, SLO and incidents
Platform-level OKR:
KR1: "Reduce Change Failure Rate from 18% to 12% by the end of Q2."
KR2: "Increase Pre-Incident Detect Rate from 35% to 60%."
SLO-matrix: for each domain - target p95/p99/Success Rate/Availability.
Incident analytics: the top 3 reasons for the last quarter should have counteraction initiatives in the current one.
6) Resource and budget planning
FTE-matrix: by squads and competencies (SRE, Observability, Data, Integrations).
Provider calendar: maintenance/quota windows (impact on dates).
CapEx/OpEx: licenses/cluster extensions vs command hours.
Buffer: ~ 15-20% for unplanned "fires" and regulatory tasks.
What-don't-do policy: A list of rescheduled/postponed initiatives with reasons.
7) Managing dependencies and risks
Dependency map: who blocks whom (service/provider/data/command).
Risk register: risk, probability/impact, owner, mitigation plan/plan B.
Change freeze: periods of prohibition of major changes (prime time events/tournaments).
Ficheflags/canaries: Mandatory for initiatives affecting traffic.
8) Quarterly cycle (rhythms)
Q-0 (preparation, 2 weeks): data collection (SLO, incidents, costs), revision of topics, preliminary prioritization.
Q planning: protection of initiatives by owners, reconciliation of resources/risks, fixing the Q plan and "not doing" the list.
Weekly sync: status, blockers, adjustments; maximum 30 minutes.
Monthly review: checking effects on metrics, possible re-scope.
Q retro: compare plan/fact, update principles/patterns.
9) Roadmap view formats
Outcome View: grouped by purpose (SLO, Cost, Risk).
Domain View: Payments/Bets/Games/KYC/Platform.
Timeline View: quarterly, with dependency and frieze markers.
Budget View: FTE/CapEx/OpEx by Initiative and Topic.
Example of a quarterly slice (summary):
Initiative Outcome Metrics Term Owner Risk
-------------------- ----------------------- -------------------- ----- ------------- -------
Active-Active Games RTO≤5 min Availability 99. 95% Q1–Q2 platform-sre High
SLO-burn на Payments − 30% of late incidents Pre-Incident↑, MTTR↓ Q1 observability Average
Kafka Lag Guardrails − 50% of lag storms Lag p95↓, DLQ↑ Q1 streaming Average
FinOps Right-sizing −15% cost/RPS Cost/RPS↓ Q2 finops Low
10) Roadmap Success Metrics (KPIs)
Delivery Predictability: percentage of initiatives completed on time (target ≥ 80%).
SLO Coverage:% of critical paths with active SLOs/alerts.
Incident Trend: − X% of P1/P2 QoQ incidents
Change Failure Rate: Target decline by quarter.
Cost Efficiency: Cost/RPS, Cost/transaction - downward trend.
Risk Burn-down: the number of "red" risks and their total weight.
Stakeholder NPS: satisfaction of domain teams with the quality of the Roadmap.
11) Roadmap launch checklist
[] Defined themes/pillars and 3-5 target outcomes per year.
[] Catalog of initiatives linked to metrics and owners.
[] Prioritization methodology (RICE/WSJF) and scales adopted.
[] Checked resources: FTE, provider windows, budgets.
[] Fixed Q-plan + "not doing."
[] Set up Outcome/Domain/Budget panels, alerts by shifts.
[] Review Schedule: weekly/monthly/quarterly.
12) Anti-patterns
List of tasks without outcomes: "make X" instead of "achieve Y by metric."
Hidden initiatives and private arrangements outside of a single artifact.
Eternal epics: no time-box, no verifiable milestones.
Priority "in terms of volume": resources are spent on the "loudest" request, and not on the most valuable one.
No "what not to do": expectations are unmanageable, trust is falling.
Lack of a link with incidents/SLO: "cosmetic" improvements instead of real ones.
13) Templates (fragments)
Initiative Template (YAML):
yaml id： OPS-42標題：「Guardrails for Canaries發行」
theme: "Reliability"
quarter: "2025-Q1"
owner: "platform-release"
stakeholders: ["payments", "bets", "games"]
結果： 「將發布後的回歸降低40%」
metrics:
name: change_failure_rate target: "<= 12%"
name: post_deploy_regression_rate target: "-40% QoQ"
slo_impact: ["api_p99 <= 300ms@99.9", "availability >= 99.95%"]
effort_weeks: 6 rice:
reach： 5000,000#事務/空間影響：3。0 confidence: 0.7 effort: 6 dependencies: ["observability-baseline", "feature-flags-core"]
risks:
-名稱： 「誤報門」
mitigation： 「基線/調諧,10%流量的飛行員」
budget:
fte: 3 capex: 0 milestones:
name: design eta: "2025-01-20"
name: pilot-10%
eta: "2025-02-10"
name: rollout-100%
eta: "2025-03-05"
Quarterly report template (Markdown):
Q1 Ops Roadmap-報告
結果總數： SLO Coverage 92%（+7個百分點），MTTR − 18%，Cost/RPS − 9%
已執行： 8/10倡議（80%）
轉換： OPS-31 → Q2（依賴PSP-X提供程序）
事件： P1=2（− 1 sq/sq），主要原因：提供商時間軸上的轉發
Follow-ups： 調音破解者，備份PSP-Y配額
14）與流程集成
事件管理：每個驗屍人員在Roadmap中→計劃/改進。
更改/版本：主要計劃僅帶有旗幟/金絲雀。
Capacity/FinOps：每月一次通過頭部和成本趨勢進行同步。
安全/合規性：季度要求和審計檢查點。
15） 30/60/90（快速啟動）
30天：收集事件/公制基礎，形成主題，以YAML格式描述10-15個舉措，選擇RICE/WSJF，記錄Q計劃。
60天：啟動Outcome/Domain/Budget面板，進行首次中季度審查，調整數據優先級。
90天：總結Q，更新原則和比額表，重新計算年度支柱。
16）溝通和透明度
Stakholders每月回顧：30分鐘，重點關註結果和風險。
異步升級：帶有「前/之後」度量的短條目。
單個Roadmap通道：狀態，更改，優先級解決方案。
紅牌規則：任何團隊都可以通過附加數據（SLO/事件/成本）來啟動優先級審查。
17) FAQ
Q：如果一切「燃燒」並且沒有時間在Roadmap上,該怎麼辦?
答：包括15-20%的「收縮緩沖」和3 個覆蓋事件主要原因的舉措的最低Q計劃。任何新的「熱門」工作都是通過重新排列優先事項。
問：如何證明「無形」倡議的價值（可觀察性，自動登機）？
答：計數變更失誤率、MTTR、事件前詳細率、回滾和「夜間佩奇」。顯示之前或之後的動態。
問：如何處理技術債務？
答：債務也是一項主動行動，包括："− X％的N類事件"，"− Y％費用/RPS"，"+Z p.p.SLO Coverage».如果沒有可衡量的結果，債務就不會進入計劃。