The dirty secret of multivariate testing is that most teams running it never had the sample size to learn anything. This post does the arithmetic in the open: real power calculations at realistic conversion rates, the multiple-comparisons correction almost everyone skips, and the fractional designs that make MVT viable at moderate traffic.
This article is a stub.
The outline below is what the finished post will cover. Want it sooner? Tell us.
01The baseline arithmetic
- Power, MDE, and significance in plain language
- A worked 2-arm calculation at a 3% conversion rate
02What a factorial design multiplies
- Cells, not factors, drive the bill
- 2x2, 2x3, and 3x3 worked examples
- The multiple comparisons correction and why skipping it manufactures winners
03Fractional factorials
- Trading interaction visibility for sample efficiency
- Choosing which interactions you can afford to confound
04Practical stop rules
- Fixed-horizon vs sequential designs
- Why 'run it two weeks' is not a stop rule
05A calculator you can steal
- Formulas and a worked spreadsheet layout
Research brief
“Rigorous sample size calculation for full-factorial multivariate tests: multiple comparison corrections, fractional factorial designs as mitigation, power analysis formulas with worked examples for 2x2 and 3x3 designs at typical email and web conversion rates.”
The deep-research question this article will answer, sources cited in the finished piece.