Interaction effect

When one factor's effect depends on which variant of another factor is present — exactly what A/B testing structurally can't see.

An interaction effect is when the effect of one factor depends on which variant of another factor is present. A new headline might do nothing on its own and a new CTA might do nothing on its own, while the combination of both lifts conversion sharply — or the reverse, where two changes that each help independently cancel each other out when shipped together.

A sequence of isolated A/B tests cannot see this by construction: each test only ever varies one factor while holding the others fixed, so it never observes what happens when both move at once. Only a full factorial design — testing every combination at once — can estimate an interaction term directly.