Sampling and representativeness
Sampling and representativeness
Sampling is the selection of a subset of observations for measurements, experiments, and model training. Representativeness means that the conclusions from the sample are scaled to the target population without bias. Below is how to design a sample, evaluate its quality and correct offsets.
1) Basic concepts
Population: target set of objects (all users/sessions/transactions).
Sampling frame: list/mechanism from which we actually select (it is important to identify coatings and "holes").
Unit of selection: user, session, event, transaction, device.
Representativeness: matching key attribute distributions in sample and population (with random error tolerance).
Random vs bias: variance of estimates versus bias (bias).
2) Types of sampling
2. 1 Probabilistic (preferred)
SRS (simple random): equal probability of selection.
Stratified: stratified population (country/channel/platform), then SRS within strata → lower variance and better coverage.
Cluster: selection of groups (days/servers/stores); cheap but higher intracluster correlation.
Systematic: every kth element after a random start ("asynchronous" data required).
PPS (probability proportional to size) - probability ∝ unit size (for example, partner traffic volume).
2. 2 Incredible (with caution)
Amenities/volunteers: quick polls, "clicking" logs → the risk of selection bias.
Quota/snowball: useful in niche groups, but subsequent calibration is needed.
3) Sources of displacement
Coverage: part of the population is not in the frame (for example, iOS data when tracking is prohibited).
Selection: the selection mechanism is associated with the target variable (active more often fall).
Nonresponse: The unresponsive are systematically different.
Survivors: Ignoring the departed/blocked.
Offset label: noisy tags, proxy labels, "manual" adjustments.
Faces: the use of future information in the formation of the sample/feature.
4) Sample size and power
4. 1 Approximations
For fraction (p) with precision (e) and confidence (1-\alpha):[
n \approx \frac{z_{\alpha/2}^2, p(1-p)}{e^2}
]We take conservatively (p = 0 {.} 5). Adjust for final population if necessary.
For medium with known (\sigma):[
n \approx \left(\frac{z_{\alpha/2},\sigma}{e}\right)^2
]4. 2 Design effect and effective size
Design Effect: (\mathrm {Deff} = 1 + (m-1 )\rho), where (m) is the cluster size, (\rho) is the intra-cluster correlation.
Effective size: (n_\text{eff}=n/\mathrm{Deff}). Cluster/weighted designs often require more (n).
4. 3 A/B and MDE
For a binary metric with symmetric groups:[
n_{\text{на группу}} \approx \frac{2(z_{1-\alpha/2}+z_{1-\beta})^2, \bar{p}(1-\bar{p})}{\mathrm{MDE}^2}
]where (\bar {p}) is the base level, MDE is the minimally detectable effect.
Consider seasonality and interference (spillover), apply CUPED/covariates to reduce variance.
5) Weights and calibration (we sample "similar" to the population)
Design weights: (w_i=1/\pi_i) (inverse probability of selection).
Post-stratification and raking: we adjust the weighted marginal (country/platform/gender-age, etc.) to the known fractions.
Calibration weighting: minimize the deviation of the weights when exactly coinciding with the control totals.
Bootstrap/replication: correct estimation of variances at weights.
Diagnostics: ESS (effective sample size), weight spread (CV, p95/p5), before/after comparison by key features.
6) Class imbalance and rare events
Stratified selection by target: oversample is a rare class, but always use threshold/weight adjustment in training.
Cost-sensitive learning: weighted losses instead of synthetics.
SMOTE/ADASYN: caution - risk of leaks/artifacts; keep strict time/group splits.
Rating: PR-AUC, Recall@FPR≤x%; probability calibration.
7) Streaming and Big Data
Reservoir sampling: Hold a representative subset from a stream of unknown length.
Event sampling: proportional to frequency/importance; for rare - trigger buffers.
Counters and sketches: Bloom/Count-Min for frequencies/uniqueness; for analytics - combine with periodic accurate counting.
De-dup and idempotency - event keys, deduplication windows.
8) Temporal and spatial aspects
Time-based sampling: fixed windows (rolling), point-in-time correctness.
Cluster/geo-sampling: clustering by node/region; consider spatial autocorrelation.
Seasonality/rhythm: representative coverage of hours/days of the week/holidays.
9) Off-policy/log data and causality
10) Representativeness diagnostics
Marginal comparison: tables population vs sample by key attributes.
Balance of covariates: SMD (standardized mean difference), Love plots.
Densities/quantiles: KS tests, quantile-plots, PSI.
Model qualification: stabilization of metrics on out-of-time and out-of-domain slices.
ESS and weights: Low ESS signals high variance in scores.
11) Anti-patterns
"Sample only active yesterday" → loss of cold/sleepers.
Average percentages "segment average" without weights.
Mix aggregation levels (events and users) in one sample.
Random selection after target-dependent filters (selection on the outcome).
Cross-val without group/time splits with dependent data.
Apply SMOTE before train/val (leak).
12) Private scenarios (cases)
Player survey: stratified by country/platform/age; post-stratification to MAU shares; nonresponse offset control.
Logs for EDA: systematic 1:N event sample + complete coverage of rare types.
Fraud detection: target stratification, cost-sensitive loss, PR-AUC score and Recall@FPR≤x%.
A/B with restricted traffic: MDE computation, power rule renewal, CUPED.
Off-policy evaluation promo: IPS/DR with weight limitation (trimming), support overlap check.
13) Artifact patterns
Sampling plan (template)
Target/Population:...
Sampling box: sources, covers/holes
Unit of selection: user/session/event
Design: strata (country, platform), shares, selection method
Size (n): calculations, assumptions (α, power, MDE), Deff
Weight scheme: design weights, post-stratification (control totals)
Calendar: day/hour/holiday coverage
Quality: coverage tests, nonresponse plan, contact procedures
Risks: selection, interference, privacy/PII
Owners and control: who counts/stores/versions
Weighing passport (template)
Basis weights: (w_i=1/\pi_i)
Calibration: goals (country, platform, age), method (raking), tolerances
Restrictions: trimming w ∈ [w_min, w_max]
Diagnostics: ESS, CV (w), SMD before/after
Usage: which reports/models apply weights
Versions: v1→v2, date, owners
14) Privacy and ethics
Privacy by Design: field minimization, aggregation, pseudonymization.
Differential privacy: randomization/subsampling as privacy amplification.
Fairness: Check error/weight difference against sensitive attributes; do not allow groups to be "invisible."
15) Pre-start checklist
- The sampling box is described; coating holes identified and documented
- Selected design (strata/clusters), calculated (n), Deff, MDE
- Set up design weights and calibration plan (control totals agreed)
- Time windows/seasonality defined; there is a plan by nonresponse
- Validation splits consider time/groups; no leaks
- Quality metrics: ESS, SMD, PSI, bootstrap intervals
- Documentation and versions; access rights and privacy-contours checked
Mini Glossary
Deff: multiplier increasing variance of scores due to design.
ESS: effective sample size after weighting.
IPS/DR: weighting methods for off-policy/log data.
SMD: standardized mean difference (covariate balance).
Reservoir sampling: maintaining a random sample from the stream.
Total
Representativeness is not "lucky with the sample," but a designed process: a correct frame, thoughtful selection design, sufficient size, weighing and calibration, honest splits and rigorous diagnosis. By following the practices described, you reduce bias, increase the portability of conclusions and get reliable solutions for product, marketing, risk and ML.