| R² = 0.01 | R² = 0.33 | R² = 0.66 | R² = 0.99 | |
|---|---|---|---|---|
| covariates balanced | ||||
| 5 | under 1% smaller | 18% smaller | 41% smaller | 85% smaller |
| 20 | under 1% smaller | 14% smaller | 31% smaller | 55% smaller |
| 50 | under 1% smaller | 11% smaller | 22% smaller | 37% smaller |
Rerandomization
How much precision a stricter acceptance threshold buys, how to run it on a fixed budget, and how to analyse the experiment afterwards.
You randomise a few hundred people into two arms, but the treatment and control groups might differ in some important way. Rerandomization is a solution to this. You draw a split, measure how unbalanced it is, and if it’s bad, draw another. The result is still a random assignment, drawn from a smaller pool of good ones. A small cost in computing time buys a better baseline table and a more precise estimate of the treatment effect.
How much more precise is worth stating up front, because it is the only reason to bother. An 18% reduction in the standard error is equivalent to having 50% more data, and with strongly predictive covariates you can do considerably better than that. With covariates that predict nothing you get the tidier baseline table and no precision at all. The next section gives the formula and a calculator for your own \(k\) and \(R^2\).
But is it worth doing? How do you run rerandomization? And how do you analyse the experiment afterwards? This article answers those questions.
What it does
Imbalance is scored with the Mahalanobis distance, which is a distance measure that avoids double-counting correlated covariates. With \(N\) units, \(k\) covariates and two equal arms:
- Compute the covariate covariance matrix \(S\) once, from all \(N\) units.
- Draw a candidate split. Take the two arms’ covariate mean vectors, \(\bar X_T\) and \(\bar X_C\).
- Score the imbalance as the squared Mahalanobis distance between those two mean vectors, scaled by \(N/4\): \[ M \;=\; \frac{N}{4}\, (\bar X_T - \bar X_C)^{\!\top}\, S^{-1} (\bar X_T - \bar X_C). \]
- If \(M\) is small enough, keep the split. Otherwise discard it and go back to step 2.
Note that \(S\) describes the pool of units, not the split, so it never changes as you redraw. You can compute it once outside the loop.
The \(N/4\) term puts \(M\) on a \(\chi^2_k\) scale, which makes thresholds interpretable.
Does it buy you enough to bother?
It mostly depends on whether your covariates predict the outcome, but a little on how many times you rerandomize.
You gain a smaller standard error on the treatment effect, shrunk by a factor of \(\sqrt{1 - (1 - v_a)R^2}\), where \(R^2\) is the share of outcome variance your balanced covariates explain and \(v_a\) is the fraction of imbalance the rerandomization leaves behind, given by \[ v_a \;=\; \frac{P(\chi^2_{k+2} \le a)}{P(\chi^2_{k} \le a)} , \] (there is a calculator below). The formula comes from Morgan and Rubin (2012), §3.2, Precision of the estimated treatment effect, which also states that the variance of the treatment effect falls by a factor \(1 - (1-v_a)R^2\). Their derivation assumes an additive treatment effect and an approximately linear outcome-covariate relationship.
At \(R^2 = 0.01\) rerandomization buys nothing but the baseline table. At \(R^2 = 0.99\) it is worth more than any sample size you could realistically afford. The covariate count matters far less than whether the covariates predict the response.
Try your own numbers
The one on the difference in means between the two arms, which is what an unpaired two-sample \(t\)-test reports, with \(R^2\) measured between that outcome and the covariates you balanced.
For example, it would not automatically apply to a difference-in-differences interaction term. The standard 2×2 DiD is a difference in means applied to the change in outcome, so the same factor applies, but \(R^2\) then has to be the share of variance your covariates explain in the change rather than in the level. That might be much smaller.
For estimators further from a difference in means, you would need to do a tailored calculation or simulation to see how much rerandomization helps.
Balance the handful of covariates you believe predict the outcome, then keep rerandomizing and retain the best of 10,000 draws. You could also threshold at an acceptance rate of 1 in 10,000, using the \(\chi^2_k\) quantile at \(p_a = 10^{-4}\).
Do not throw in covariates because you have them. Every one you add has to be balanced simultaneously, so the balance on important metrics gets diluted.
Rerandomization narrows the true sampling distribution of your estimate whatever you do next. The estimate varies less from one experiment to the next, so the test statistic is larger on average and power rises.
The reported standard error does not shrink with it. It is computed from the within-group variances and the sample sizes, and uses nothing about the acceptance rule, so it comes out at the value it would have had under plain randomisation. The interval is therefore too wide and the \(p\)-value too large. The test is still valid, since its coverage is conservative rather than wrong, but you are leaving precision unclaimed.
For an experiment powered at 80% under plain randomisation, with covariates at \(R^2 = 0.66\):
| power | |
|---|---|
| plain randomisation | 80% |
| rerandomization, unadjusted \(t\)-test | 93% |
| rerandomization, adjusted analysis | 99.8% |
Most of the benefit survives even if you cannot adjust the analysis afterwards, but you’re leaving some precision on the table if you can do the adjustment but don’t.
A budget is simpler than a threshold
Picking an acceptance rate commits you to an unbounded loop. You will probably accept within a few thousand draws, but nothing guarantees it. Fixing a budget instead, drawing 10,000 and keeping the best, runs in known time and needs no threshold.
Morgan and Rubin recommend exactly this, in §5.2 of their 2015 tiers paper:
In practice we expect to see little difference between rerandomizing until an allocation with \(M \le a\) is obtained, and choosing the allocation with the smallest \(M\) from \(1/p_a\) randomizations, where \(p_a = P(M \le a)\).
The two schemes are different though. The winner of \(B\) draws is not guaranteed to clear the 1-in-\(B\) bar, and misses it about \((1 - 1/B)^B \approx 37\%\) of the time. Best-of-\(B\) is always the weaker of the two.
| threshold at 1 in 10,000 | best of 10,000 | ratio | |
|---|---|---|---|
| covariates k | |||
| 2 | 0.00005 | 0.00010 | 2.04x |
| 10 | 0.07326 | 0.08151 | 1.11x |
| 20 | 0.19594 | 0.20692 | 1.06x |
| 50 | 0.39496 | 0.40645 | 1.03x |
The ratio looks alarming at \(k = 2\), twice the imbalance, but the absolute difference is tiny: \(0.00005\) against \(0.00010\). Both schemes have annihilated the imbalance. The gap could only matter at large \(k\), where a fraction of the imbalance survives, and that is where the two schemes converge: 6% apart at twenty covariates, 3% at fifty.
Morgan and Rubin’s expectation holds, then, for a reason their sentence does not give. The discrepancy is largest where both schemes are already far better than they need to be. Pick a budget you can afford and keep the best draw.
What you are actually selecting
Below is a plot of the acceptance region for two correlated covariates, height and weight. Each dot is one candidate split, placed at its pair of covariate mean differences. The cloud tilts because height and weight are correlated, so a split over-representing tall people over-represents heavy ones too, and the acceptance region tilts with it.
At a 5% acceptance rate that region is a small ellipse at the origin. Drag to zoom in on it.
Zoomed in, the green dots are the splits you would keep. They sit inside a tilted ellipse. A larger height imbalance is tolerated when weight is imbalanced the same way, and not tolerated when the two disagree. It’s because the criterion is the Mahalanobis distance rather than the sum of squared z-scores.
How to account for it afterwards
Rerandomization changes the design, so it changes what “by chance” means. The sampling distribution of a difference in means is no longer normal. Li, Ding and Rubin (2018) show it picks up a truncated term, because the design removed the part of the imbalance the covariates explain.
Include the design covariates with the predictors of whatever model estimates your treatment effect.
Li and Ding (2020) show that once the analysis adjusts for the design variables, asymptotic normality comes back and conventional standard errors are justified again.
What “adjust” means in practice. The default is ordinary least squares: regress the outcome on a treatment indicator plus the balancing covariates, and read the treatment coefficient. Two refinements worth knowing:
- Centre the covariates and interact them with treatment. Lin (2013) showed this version can only help. It never does worse asymptotically than the unadjusted estimator, even if the linear model is wrong, and it recovers more when the treatment effect varies with the covariates. Plain additive adjustment is usually fine; the interacted form is the safer default.
- It does not have to be linear. The requirement is that the fitted model includes the balancing covariates. Logistic, Poisson and other M-estimators are covered by the same argument. For a difference-in-differences, adjust the DiD specification itself rather than running a separate procedure.
You are not restricted to the balancing covariates. Including others is allowed and breaks nothing. It is only worth doing for covariates you had a reason to collect, though: adding predictors because they happen to be available costs degrees of freedom and invites specification search.
If you used a draw budget rather than a threshold and want to be certain, randomisation inference always applies. Regenerate assignments with the same procedure you actually ran, refit the model each time, and read the null distribution off the results. It depends on no formula and no asymptotics, so it covers any acceptance rule and any estimator.
The analysis must use the pre-randomisation values of the balancing covariates rather than their latest values.
Suppose you ran a development experiment across villages and balanced on village population. By the end of the study populations have moved, and if the intervention worked it may well have caused some of that movement, through migration or falling child mortality. Adjusting for end-of-study population would be conditioning on something the treatment affected. That is post-treatment bias. It removes part of the effect you are measuring, and it can flip the sign of an estimate.
References
- Morgan, K. L. and Rubin, D. B. (2012). Rerandomization to improve covariate balance in experiments. The Annals of Statistics, 40(2), 1263–1282.
- Morgan, K. L. and Rubin, D. B. (2015). Rerandomization to balance tiers of covariates. JASA, 110(512), 1412–1421.
- Li, X., Ding, P. and Rubin, D. B. (2018). Asymptotic theory of rerandomization in treatment–control experiments. PNAS, 115(37), 9157–9162.
- Lin, W. (2013). Agnostic notes on regression adjustments to experimental data. The Annals of Applied Statistics, 7(1), 295–318.
- Li, X. and Ding, P. (2020). Rerandomization and regression adjustment. Journal of the Royal Statistical Society Series B, 82(1), 241–268.