A visitor browses anonymously, gets assigned to variant B, then logs in and gets assigned again as a known user, this time to variant A. Which cohort do their purchases count toward? Most stacks answer 'both', and their lift numbers are quietly wrong. This post walks the failure modes and the design that avoids them.
This article is a stub.
The outline below is what the finished post will cover. Want it sooner? Tell us.
01The double-assignment problem
- Anonymous and identified keys as separate users
- Cross-cohort contamination after login
- Why this biases lift toward zero, so you kill real winners
02How the major platforms resolve identity
- Merge-on-login, alias tables, and identity clusters
- What Segment, Amplitude, and GrowthBook each actually do
- Where each approach rewrites history, and why that is the dangerous part
03Never re-key an assignment
- Assignments as immutable facts about what a user was shown
- Linking as a recorded relationship, resolved at read time
- TraqLyte's rule: /assign searches the whole identity cluster, writes nothing
04What to audit in your own stack
- A checklist to detect double-counting in historical data
- Estimating how much contamination has cost you
Research brief
“How do experimentation platforms handle anonymous-to-identified identity resolution? Document failure modes: double-counting, cohort contamination after login, session stitching errors, and how Amplitude, Segment, Optimizely, and GrowthBook resolve identity clusters. Include published incident writeups.”
The deep-research question this article will answer, sources cited in the finished piece.