Multiple Comparisons on Adventures in Why

Interpretable and Validatable Uplift Modeling

bob@ConvexAnalytics.com (Bob Wilson) — Tue, 27 Oct 2026 00:00:00 +0000

Interpretable and Validatable Uplift Modeling

bob@ConvexAnalytics.com (Bob Wilson) — Sat, 15 Jun 2024 00:00:00 +0000

In this note, we introduce a method for interpreting and validating the results of uplift modeling. We propose two novel strategies for controlling the Familywise Error Rate in this setting.

The Winner's Curse: Why it Happens and What to Do About It

bob@ConvexAnalytics.com (Bob Wilson) — Tue, 04 Jan 2022 00:00:00 +0000

When running an A/B test with many variants, say, more than 5, we often run into a phenomenon known as the Winner’s Curse, where the winning variant performs worse when we adopt it universally than it had during the test itself. In this post, we discuss why this phenomenon occurs and what to do about it.

Testing with Many Variants

bob@ConvexAnalytics.com (Bob Wilson) — Sun, 05 Sep 2021 00:00:00 +0000

This is a long drive for someone with nothing to think about.

Scheffe's Method for Multiple Comparisons

bob@ConvexAnalytics.com (Bob Wilson) — Sun, 01 Aug 2021 00:00:00 +0000

I’ve written previously about using the Bonferroni correction for the multiple comparisons problem. While it is without a doubt the simplest way to correct for multiple comparisons, it is not the only way. In this post, I discuss Scheffé’s method for constructing simultaneous confidence intervals on arbitrarily many functions of the model parameters.

Multiple Comparisons

bob@ConvexAnalytics.com (Bob Wilson) — Sat, 23 Jan 2021 00:00:00 +0000

The simplest kind of A/B test compares two options, using a single KPI to decide which option is best. The more general theory of statistical experiment design easily handles more options and more metrics, provided we know how to incorporate the multiple comparisons involved. To see why this is important, read on!