Potential Outcomes on Adventures in Why

Principal Stratification and Mediation

bob@ConvexAnalytics.com (Bob Wilson) — Sun, 05 Jan 2025 00:00:00 +0000

This post explores principal stratification and mediation analysis as tools for understanding causal effects, decomposing them into direct and indirect components. It covers scenarios like non-compliance, missing outcomes, and surrogate indices, highlighting the importance of assumptions such as no direct effects and no Defiers. Practical methods, including multiple imputation, regression, and matching, are discussed for estimating effects even when key quantities are unobserved. Real-world examples, like marketing lift studies and product funnels, illustrate the relevance of these techniques for addressing complex causal questions.

Attributable Effects

bob@ConvexAnalytics.com (Bob Wilson) — Sat, 29 Jul 2023 00:00:00 +0000

In a previous post, we discussed why randomization provides a reasoned basis for inference in an experiment. Randomization not only quantifies the plausibility of a causal effect but also allows us to infer something about the size of that effect.

The Reasoned Basis for Inference in Experiments

bob@ConvexAnalytics.com (Bob Wilson) — Sun, 04 Jun 2023 00:00:00 +0000

In his 1935 book, “Design of Experiments”, Ronald Fisher described randomization as the “reasoned basis for inference” in an experiment. Why do we need a “basis” at all, let alone a reasoned one?

Tests with One-Sided Noncompliance

bob@ConvexAnalytics.com (Bob Wilson) — Sun, 26 Feb 2023 00:00:00 +0000

Introduction

Tech companies spoil data scientists. It’s so easy for us to A/B test everything. We can alter many aspects of the product from a configuration UI. We have the sample size to get a good read in as little as a few days. We have the data infrastructure to analyze and report results quickly.

Violations of the Stable Unit Treatment Value Assumption

bob@ConvexAnalytics.com (Bob Wilson) — Sun, 10 Jan 2021 00:00:00 +0000

We have previously mentioned the Stable Unit Treatment Value Assumption, or SUTVA, a complicated-sounding term that is one of the most important assumptions underlying A/B testing (and Causal Inference in general). In this post, we talk a little more about it and why it is so important.

Contingency Tables Part I: The Potential Outcomes Framework

bob@ConvexAnalytics.com (Bob Wilson) — Sat, 24 Oct 2020 00:00:00 +0000

“Why can’t I take the results of an A/B test at face value? Who are you, the statistics mafia? I don’t need a PhD in statistics to know that one number is greater than another.” If this sounds familiar, it is helpful to remember that we do an A/B test to learn about different potential outcomes. Comparing potential outcomes is essential for smart decision making, and this framework is the cornerstone of causal inference.

Counterfactuals and Causal Reasoning

bob@ConvexAnalytics.com (Bob Wilson) — Sat, 26 May 2018 00:00:00 +0000

What does ‘Why?’ mean anyway?