Before Trusting a Metric Change: An Experiment Checklist

A metric can move because of a product change, a different mix of users, a tracking error, or ordinary variation. An experiment is useful when its design and measurement let you distinguish those explanations well enough for the decision at hand.

Before collecting data, state the question, the target population, the assignment unit, the primary outcome, and the smallest effect that would change a decision. Keep guardrail metrics for important harms. A long list of possible improvements without a primary question makes selective interpretation too easy.

Align assignment and analysis

If teams are randomized, their users may share behavior and exposure. Treating every user as an independent randomized unit can understate uncertainty. Choose an analysis that respects the assignment and plausible correlation structure. Also consider interference: a treated user can affect an untreated user in a shared marketplace or collaboration tool.

Check whether assignment was implemented correctly and whether measurement differs by group. A feature that changes how an event fires can improve the dashboard while leaving the actual user outcome unchanged. Validate a few real event sequences before interpreting the aggregate.

Plan the stopping and comparison rules

Repeatedly checking a conventional fixed-horizon p-value and stopping as soon as it crosses a threshold changes the procedure’s error behavior. Choose a fixed horizon or an appropriate sequential method before looking at results. Account for the family of comparisons when many metrics, variants, or slices are tested.

Low power does not establish no effect. Report the estimated effect and uncertainty in terms meaningful to the product. A wide interval covering both a useful gain and a serious loss calls for a different decision than a narrow interval around a negligible change.

Inspect missingness and exclusions

Track attrition by assignment. Removing participants based on behavior the treatment itself changes can distort the comparison. Define inclusion rules before assignment where possible, and disclose any post-assignment exclusions with a clear explanation of what quantity is being estimated.

Run practical sensitivity checks: do conclusions depend on one unusually active team, one day, or an instrumentation incident? Inspect subgroup results that matter for the product, while acknowledging when those subgroups are too small for a precise conclusion.

Keep a decision record

• State who was measured and who was not.
• Record the assignment, measurement, and stopping rules.
• Show the effect estimate, uncertainty, and guardrail outcomes.
• Separate causal conclusions from descriptive observations.
• Keep concrete user outcomes beside the aggregate metric.

A trustworthy experiment report makes its limitations easy to see. It should help a reviewer decide what the evidence supports and what still needs to be tested.

Reference tables to keep nearby

• Experimental Design and Inference Pitfalls — Common threats to experiment validity, with a practical design adjustment and reporting consequence.

Primary sources

These are the specifications and project documentation behind the examples. Check the relevant implementation and version before applying a pattern.

• NIST design of experiments handbook
• NIST statistical assumptions