BoostSampleSize
  • Articles

Covariate balance in experimental design

How much precision you can buy by choosing the random split carefully, when the sample is too small to grow.

Here are two facts that changed how I think about small experiments.

The first is that how you split the sample into control and treatment groups matters. Consider an experiment with 400 pupils, where the aim is to test whether an intervention improves test scores. If you pair up pupils with similar test scores before the experiment, then randomly assign control/treatment within each pair, the results will be as precise as one run on 1,200 pupils where the pupils are split at random and analysed with a two-sample t-test.

The second is how you analyse the results matters. In the example above, if you analysed the data from the paired experiment design with a two-sample t-test instead of a paired t-test, the gain from the design would be lost. The precision comes from the pairs, and an ordinary two-sample test cannot see them, so it reports an interval too wide and misses effects it should have caught.

This site goes through some techniques for improving the design of small experiments, and explains how to analyse them correctly. Pairwise matching splits the sample into pairs that are as alike as possible and randomly assigns control/treatment within each pair. Rerandomization draws multiple ordinary random splits, scores each on how balanced it is, and keeps the most balanced one. The Mahalanobis distance is the distance metric both of them use to decide how alike two units are.

All the code is Python, is available at this repository.

NoteSeries status

Pairwise matching, Rerandomization and The Mahalanobis distance are written. Stratification exists as an outline and is not yet published.

Pairwise matching

Pairing units before you randomise: when it beats rerandomization, why the pairing has to be solved properly, and how to analyse the result.

Rerandomization

How much precision a stricter acceptance threshold buys, how to run it on a fixed budget, and how to analyse the experiment afterwards.

The Mahalanobis distance

The distance the other two methods are built on: why correlated columns are counted once, how to handle categorical features, and where it fails silently.

No matching items

Alexander Bolton

 

Built with Quarto