Experimental Design and Inference Pitfalls
Common threats to experiment validity, with a practical design adjustment and reporting consequence.
Common threats to experiment validity, with a practical design adjustment and reporting consequence. Each row includes a practical application and a failure to check before using it.
Use it when
- Review a product experiment before collecting data.
- Interpret a metric change without overstating causality.
Watch out for
- Design guidance does not replace a task-specific statistical analysis.
- Randomization reduces some biases but does not fix poor measurement.
The data
10 rows
| Pitfall | Example | Adjustment | Report |
|---|---|---|---|
| Selection bias | Only enthusiastic early users enter the experiment. | Define the target population and recruitment process. | State who is outside the measured population. |
| Confounding | A new feature and a pricing change launch together. | Randomize assignment or separate the interventions. | Distinguish association from an identified causal effect. |
| Peeking | Stop when a running p-value first falls below a threshold. | Use a planned fixed horizon or valid sequential method. | Describe the stopping rule. |
| Multiple comparisons | Test many metrics and publish only the best. | Predefine primary outcomes and account for multiplicity. | Show the family of analyses performed. |
| Low power | A small test misses a practically important effect. | Plan sample size around a meaningful effect and variance. | Report uncertainty rather than declaring no effect. |
| Attrition | One group loses more participants before measurement. | Measure and investigate missingness by assignment. | Report dropout and sensitivity analyses. |
| Unit mismatch | Randomize teams but analyze users as independent. | Use an analysis that respects the assignment unit. | State the number of independent units. |
| Interference | Treated users change the experience of control users. | Consider cluster assignment or an interference model. | Describe spillover limitations. |
| Metric gaming | A proxy improves while the real user outcome worsens. | Use guardrail metrics and inspect concrete outcomes. | Explain what the primary metric fails to capture. |
| Post-treatment filtering | Remove participants based on behavior affected by treatment. | Define inclusion before assignment or justify the estimand. | Disclose exclusions and their timing. |
Fields
The shape of this dataset, read from the data itself: every column, its type and a real value. Agents can use this to plan a query without fetching the whole document.
| Section | Field | Type | Example |
|---|---|---|---|
| data | pitfall | string | Selection bias |
| data | example | string | Only enthusiastic early users enter the experim… |
| data | adjustment | string | Define the target population and recruitment pr… |
| data | report | string | State who is outside the measured population. |
Fetch the same bytes
The static files contain the same reference rows as the API, with no API rate limit. Live metadata, such as view counts, may differ. Use the API for content negotiation; use the static files for a cacheable snapshot.
curl "https://yjtoon.com/api/dataset/experimental-design-pitfalls?format=toon"
const res = await fetch( "https://yjtoon.com/static-data/dataset/experimental-design-pitfalls.toon" ); const toon = await res.text();
Rate limit: 120 requests per minute per IP, no key and no signup. API reference →
Sources and editorial notes
Original editorial reference notes by YJTOON, informed by the linked primary documentation. External specifications retain their own licenses.
Editorial review: . Source documents keep their own licenses.