If every user gets some variant, you can say which variant won, but not whether the campaign itself did anything. That question, incrementality, needs a cohort that sees nothing. This post covers the holdout designs that answer it and the org pressure that makes teams skip them.
This article is a stub.
The outline below is what the finished post will cover. Want it sooner? Tell us.
01Attribution vs incrementality
- Why last-click numbers always look better than lift
- The holdout as the only honest denominator
02Holdout designs
- Per-campaign controls vs a global holdout
- Sizing: how small a control can you get away with
- Published lift-study methodology from the ad platforms
03The politics of holding out revenue
- Why teams quietly shrink or drop controls
- Framing the holdout as insurance, with the cost quantified
04Controls as first-class cohorts
- A control is an arm with no treatment, not an absence of data
- How TraqLyte flags control cohorts and blocks publishing without one
Research brief
“Best practices for holdout and control groups in marketing campaign measurement: global holdouts vs per-campaign controls, incrementality measurement vs attribution, published lift-study methodologies from Meta, Google, and independent researchers.”
The deep-research question this article will answer, sources cited in the finished piece.