Hypothesis Testing
Modes of Inference in Randomized Experiments
Randomization provides the “reasoned basis for inference” in an experiment. Yet some approaches to analyzing experiments ignore the special structure of randomization. Simple, familiar approaches like regression models sometimes give wrong answers when applied to experiments. Approaches exploiting randomization deliver more reliable inferences than methods neglecting it. Randomization inference should be the first method we reach for when analyzing experiments.
The Reasoned Basis for Inference in Experiments
In his 1935 book, “Design of Experiments”, Ronald Fisher described randomization as the “reasoned basis for inference” in an experiment. Why do we need a “basis” at all, let alone a reasoned one?
Testing with Many Variants
This is a long drive for someone with nothing to think about.
Multiple Comparisons
The simplest kind of A/B test compares two options, using a single KPI to decide which option is best. The more general theory of statistical experiment design easily handles more options and more metrics, provided we know how to incorporate the multiple comparisons involved. To see why this is important, read on!
Contingency Tables Part IV: The Score Test
The score test can be used to calculate p-values and confidence intervals for A/B tests. The score test considers the slope of the likelihood function at the parameter value associated with the null hypothesis.
Unshackle Yourself from Statistical Significance
Don’t be a prisoner to statistical significance. A/B testing should serve the business, not the other way around!
Fisher's Exact Test
Simulation-based inference sits on a rigorous foundation.
A/B Testing, Part 2: Statistical Significance
Results can’t always be taken at face value.
