Hypothesis Testing on Adventures in Why

Interpretable and Validatable Uplift Modeling

bob@ConvexAnalytics.com (Bob Wilson) — Tue, 27 Oct 2026 00:00:00 +0000

Modes of Inference in Randomized Experiments

bob@ConvexAnalytics.com (Bob Wilson) — Sat, 13 Jan 2024 00:00:00 +0000

Randomization provides the “reasoned basis for inference” in an experiment. Yet some approaches to analyzing experiments ignore the special structure of randomization. Simple, familiar approaches like regression models sometimes give wrong answers when applied to experiments. Approaches exploiting randomization deliver more reliable inferences than methods neglecting it. Randomization inference should be the first method we reach for when analyzing experiments.

The Reasoned Basis for Inference in Experiments

bob@ConvexAnalytics.com (Bob Wilson) — Sun, 04 Jun 2023 00:00:00 +0000

In his 1935 book, “Design of Experiments”, Ronald Fisher described randomization as the “reasoned basis for inference” in an experiment. Why do we need a “basis” at all, let alone a reasoned one?

Testing with Many Variants

bob@ConvexAnalytics.com (Bob Wilson) — Sun, 05 Sep 2021 00:00:00 +0000

This is a long drive for someone with nothing to think about.

Multiple Comparisons

bob@ConvexAnalytics.com (Bob Wilson) — Sat, 23 Jan 2021 00:00:00 +0000

The simplest kind of A/B test compares two options, using a single KPI to decide which option is best. The more general theory of statistical experiment design easily handles more options and more metrics, provided we know how to incorporate the multiple comparisons involved. To see why this is important, read on!

Contingency Tables Part IV: The Score Test

bob@ConvexAnalytics.com (Bob Wilson) — Tue, 24 Nov 2020 00:00:00 +0000

The score test can be used to calculate p-values and confidence intervals for A/B tests. The score test considers the slope of the likelihood function at the parameter value associated with the null hypothesis.

Unshackle Yourself from Statistical Significance

bob@ConvexAnalytics.com (Bob Wilson) — Mon, 12 Oct 2020 00:00:00 +0000

Don’t be a prisoner to statistical significance. A/B testing should serve the business, not the other way around!

Fisher's Exact Test

bob@ConvexAnalytics.com (Bob Wilson) — Sun, 25 Mar 2018 00:00:00 +0000

Simulation-based inference sits on a rigorous foundation.

A/B Testing, Part 2: Statistical Significance

bob@ConvexAnalytics.com (Bob Wilson) — Mon, 05 Mar 2018 00:00:00 +0000

Results can’t always be taken at face value.