Every experimentation vendor ranks on this comparison, and most of them answer it with a sales pitch. The honest answer depends on three numbers: your traffic, your baseline conversion rate, and how much you believe your page elements interact. This post gives you the decision framework, with the math shown.
This article is a stub.
The outline below is what the finished post will cover. Want it sooner? Tell us.
01What each design actually measures
- A/B: one factor, total effect of a bundled change
- Full factorial MVT: main effects plus interaction effects
- Why 'test one thing at a time' is folk wisdom, not statistics
02The sample size penalty, quantified
- Worked power calculations for 2-arm, 2x2, and 3x3 designs
- Why MVT needs 4-8x the recipients and what that costs in calendar time
- Fractional factorials as the middle path
03Interactions A/B testing structurally cannot see
- Published case studies where the winning combination lost every individual test
- CTA color x placement as the canonical example
04A decision framework by traffic volume
- Under 10k conversions/month: sequential A/B
- 10k-100k: fractional factorial on 2-3 factors
- Above that: full factorial, then bandits
05How TraqLyte models this
- Factors and variants as first-class objects, cohorts as the cross-product
- Fractional designs without changing your integration
Research brief
“Compare A/B testing and full-factorial multivariate testing: minimum sample sizes at 80% power, how interaction effects change conclusions, real published case studies where MVT found interactions A/B missed, and decision criteria by traffic volume.”
The deep-research question this article will answer, sources cited in the finished piece.