Meta-analytics and metrics metrics

1) What is meta-analytics

Meta-analytics is the management of the measurement itself: formulas, sources, freshness, stability and the cost of obtaining metrics. The goal is for each metric to be a testable unit of knowledge: reproducible, timely, comparable between teams, and economical.

2) Metric Skeleton (Metric Entity Model)

• Identifier: 'metric _ key', owner (Product/Data Owner), criticality (Gold/Silver/Bronze).
• Meaning: definition, units, aggregation, windows (day/wow/mom/rolling).
• Formula and layer: SQL/DSL, semantic layer (dimensions, filters), valid segments.
• Sources and lineages: tables, versions, dependencies (features/showcases/models).
• SLO метрики: latency p95, freshness, coverage, accuracy, stability.
• Trust Signals: last checks, DQ status, formula CI tests.
• Economy: bytes scanned, cost-per-chart (CPC), cost-per-insight (CPI)
• Access control: RLS/CLS, sensitivity.
• Versioning: semver formulas ('ggr @ 2. 3. 0 '), change log.

3) "Metrics metrics": what to measure

Quality and trust

Accuracy: discrepancy with reference/accounting (MAE/APE).
Consistency: coincidence between sources/dashboards (canonical formula).
Freshness: age of data vs SLO (sec/min).
Completion: proportion of completed segments/dates.
Stability: variance/volatility under constant conditions (Levene/variance ratio).
Explainability: presence of formula/links/CI/ranges in the card.

Performance and cost

Latency p95/p99 calculation/render.
Bytes scanned/query.
CPC/CPI: chart/insight cost.
Cache hit-rate and materialization.

Risk and use

Adoption: Proportion of dashboards/solutions using the metric.
Breakage rate: rate of test drops/freshness breakdowns.
Drift score-Shifts distributions after formula/source changes.
Support load: cases/incidents by metric.

4) Semantic layer and "Metric Store"

Unified grammar: measures, dimensions, filters, aggregation and compatibility rules.
API/Metrics catalog: search, cards, versioning, lineage graph.
Intermediate calculations: materializations (daily/weekly), canonical blizzards.
Publishing policy: only from the Metric Store to production boards/AI visualization.

5) Versioning and compatibility

SemVer for metrics: 'MAJOR' - breaking change in meaning/aggregations; 'MINOR' - new section/attribute; 'PATCH' - optimizations without changing the value.
Depression flow: warning, parallel mode v1/v2, shutdown date, migration guide.
Contract tests: equality/inequality, invariants (sum of subsegments = integer).

6) Lineage and metrics audit

Upstream: sources, DQ rules, versions.
Transform: SQL/DBT/DAG node, schema compatibility checks.
Downstream: dashboards/models/reports who consumes the metric.
Audit: who changed the formula and when, which incident or release affected.

7) Metric Observability

Time profiles: seasonality, calendar effects, promo.
Abnormalities: STL/ESD/BOCPD; alerts with criticality sensitivity.
Check gates: before release - comparison of the old/new formula on the "golden" slice.
Drift monitoring: PSI/JS by segment; source change markers.

8) Metrics Economics and FinOps

Quotas/limits: max bytes scanned, runtime, off-peak rebuild.
Chargeback: Teams pay "virtually" for tough reports.
Cache/materializations: tile and TTL profiles.
Optimization: column formats, ZSTD, sorting/clustering, preaggregates.

9) Change Mgmt

RFC metrics: goal, risk, expected shift, rollback plan.
A/B formulas: parallel calculation and comparison of v1 vs v2 metrics.
Communication: changelog in card, banner on linked dashboards.
Rollback: quick switch to previous materialization.

10) Roles

Metric Owner: meaning, formula, releases, communications.
Data Engineer: pipeline reliability, performance.
Analyst/Scientist: validity and causal interpretation.
FinOps: cost and quotas.
Compliance/Privacy: access, masking, reporting.

11) Antipatterns

SELECT in the metric formula.
Two "official" formulas of the same metric.
Manual edits in the dashboard instead of changing in the Metric Store.
No comparison window/basis.
Zero lineage and no owner.
Hidden filters (different countries/currencies) → disparate numbers.

12) Implementation Roadmap

1. Inventory: list of top 50 metrics, owners, criticality, current formulas.
2. Metric Store MVP: cards, SemVer, API, lineage, CI tests.
3. Observability: freshness/latency/bytes scanned/anomalies, SLO panels.
4. FinOps: limits, cache, materializations, value reports.
5. Governance: RFC/degradation/kickbacks, deprecation policy.
6. Scale: automatic "metrics metrics," integration with AI visualization/context.

13) Metric checklist (before publication)

• The metric card is full (definition, units, segments, window).
• Formula is versioned; consistency/invariant tests are green.
• Freshness/Latency SLOs are set and monitored.
• DQ sources are green; lineage is transparent.
• Request cost within limits; cache/materialization configured.
• Access policies (RLS/CLS) and masking applied.
• Communication of changes is prepared; there is a rollback plan.

14) Mini templates

14. 1 Metric Card (YAML)

yaml metric:
key: ggr version: 2. 3. 0 owner: "product-data@company"
definition: "Bet amount minus win amount"
unit: "currency"
aggregation: "sum"
window: ["day","wow","mom"]
dims_allowed: ["country","device_os","provider","game"]
lineage:
sources: ["fact_bets","fact_payouts"]
transforms: ["ggr_daily. sql"]
slo:
freshness_s: 600 latency_p95_ms: 1500 accuracy_mae_pct: <=0. 5 finops:
max_bytes_scanned_mb: 2048 cache_ttl_min: 60 access:
sensitivity: "internal"
rls: ["tenant","region"]

14. 2 Formula tests (dbt-style)

sql
-- invariant: GGR = bets - wins select count () as violations from (
select bets - payouts as ggr_calc, ggr from mart_daily
) t where abs(ggr_calc - ggr) > 1e-6;

14. 3 Alert Metrics Policy

yaml alerts:
- name: ggr_freshness when: freshness_s > 600 severity: high action: [page:oncall-data, degrade:use_last_materialization]
- name: ggr_stability when: variance_ratio_week > 2. 5 severity: medium action: [open:investigation, add:banner_on_dashboards]

14. 4 Comparison of formula versions

sql select dt, country,
ggr_v1, ggr_v2,
(ggr_v2 - ggr_v1) as delta,
100(ggr_v2 - ggr_v1)/nullif(ggr_v1,0) as delta_pct from compare_ggr_v1_v2 order by dt desc, abs(delta_pct) desc limit 100;

14. 5 "Metrics metrics" report for dashboard

yaml meta_dashboard:
tiles:
- metric: freshness_s target: <=600
- metric: latency_p95_ms target: <=1500
- metric: bytes_scanned_mb target: <=2048
- metric: anomaly_rate_7d target: <=0. 5%
- metric: adoption_rate target: >=80%

15) The bottom line

Meta-analytics make metrics reliable engineering objects: they have owners, versions, SLOs, tests and cost. When metric cards, semantic layer, observability, and FinOps work together, the organization gets comparable, verifiable, and cost-effective numbers rather than "different truths." This is the foundation for fast analytics, correct solutions and scalable growth.