When identity linking corrupts experiments

01The double-assignment problem

Every identity system starts with two identifiers that have nothing to do with each other: a browser or device handle created before anyone logs in, and a stable account-level ID created the moment they do. Cookies, localStorage, and app/device identifiers issued by a public-facing tracker persist the first. A login or signup event mints the second. Until something explicitly ties them together, an experimentation platform has no way to know they are the same human, and it will happily treat them as two.

That gap produces the most common identity failure in experimentation: double-counting. A visitor is bucketed into variant B while anonymous, converts, logs in, and the platform sees a brand-new user with no history who then gets assigned again, sometimes into variant A. Amplitude documents this directly in its own "merged users" behavior notes, where duplicate internal IDs exist until a later merge resolves them, and raw exports can still show the pre-merge rows even after resolution happens downstream. GrowthBook surfaces the same symptom from the other direction, as a "multiple exposures" health-check warning whenever one experimental unit shows up under more than one variation.

The subtler and more damaging version isn't double-counting, it's cohort contamination: the same person ends up attributed to both arms of the same experiment. If assignment happens on the anonymous ID and the platform later re-evaluates on the account ID, a user can flip from control to treatment mid-journey. Amplitude has a name for this, variant jumping, and lists inconsistent identity input between assignment and exposure as a direct cause. Optimizely's Feature Experimentation docs go further and simply forbid it: never change the user ID inside an existing user context, because doing so changes variation assignment, inflates visitor counts, and skews results.

The statistical damage compounds. Split exposure dilutes both arms toward a shared average, which biases measured lift toward zero: the exact opposite of a false positive. Teams don't ship experiments that look neutral; they kill them. The winners most likely to get discarded because of an identity bug are the ones the underlying design earned. Nobody investigates a null result for a plumbing defect, because a null result looks like the expected outcome of a mediocre idea, not a corrupted measurement.

02How the major platforms resolve identity

There's real architectural divergence here, and it maps to how much each vendor trusts itself to guess who a person is. Segment is the most graph-native of the group: Unify builds an online/offline identity graph across user_id, anonymous_id, email, device IDs, and custom external IDs, with an explicit create/add/merge decision tree per incoming event. Segment is also explicit that matching is deterministic only (it does not do probabilistic fuzzy matching), and it exposes real levers to the operator: identifier priorities, blocked values, per-identifier time limits, and a merge-limit cap (100 profiles, 1,000 mapped identifiers by default). That configurability is a strength and a liability in the same breath: a graph powerful enough to merge everything correctly will merge everything incorrectly if you feed it a shared identifier without limits. Segment's own docs use a shared-iPad example to make exactly that point.

Amplitude sits in the middle. It auto-merges anonymous device history into a known user around three signals: device_id, user_id, and an internal amplitude_id, but it deliberately does not auto-merge across an invented new user ID for an existing person; that requires the separate User Mapping API, and even then user properties don't merge across the mapped IDs. Amplitude compensates for the parts it can't merge automatically with strong diagnostics: dedicated variant-jumping and sample-ratio-mismatch guides, and assignment-event deduplication via a derived insert_id that yields roughly one assignment per user per result per day.

Optimizely and GrowthBook both punt identity resolution back to the application almost entirely. Optimizely hashes a caller-supplied visitor or user ID deterministically (MurmurHash on web, a 10,000-bucket split in Feature Experimentation) and offers "bring your own ID" so you can override its default optimizelyEndUserId cookie; useful because Safari's Intelligent Tracking Prevention can cut that cookie's life to seven days and constrain local storage too, silently reallocating returning visitors as new ones. There is no merge graph; the rule is simply never mutate the user ID inside a live user context. GrowthBook is the most warehouse-native of the four: its core primitive is a hashAttribute plus an optional fallback attribute for sticky bucketing across the login boundary, backed by pluggable storage (cookie, localStorage, Redis, custom). Being open source, some of its most instructive incident material lives in GitHub issues rather than a vendor knowledge base, including a documented case where lost exposures during a redirect, not duplicate identifiers, drove a severe sample-ratio mismatch.

The common thread across all four: every "rewrite history" operation is the dangerous one. Segment states plainly that adding a new identifier to its resolution rules only affects new events going forward, never retroactively re-merging old profiles. Changing an Optimizely user ID in place rebuckets live users. Changing a GrowthBook hash attribute, or letting it diverge from whatever identifier the warehouse uses at analysis time, produces multiple-exposure warnings even when the SDK behaved exactly as configured. None of the vendors reviewed treat "merge after the fact and pretend it always looked this way" as safe, because it isn't: it silently changes what an experiment measured without changing what actually happened to the user.

03Never re-key an assignment

TraqLyte's rule is the one Optimizely arrives at by warning and Segment arrives at by admission: an assignment is an immutable fact about what a specific user_key was shown, and linking identities must never rewrite it. `assignment_id` is derived as sha256(campaign_id || user_key || campaign_version || salt) and doubles as the primary key. Rewriting `user_key` on an existing row would desynchronize the PK from its own derivation. A later unlink or an anonymize request would recompute the same PK and collide irrecoverably with whatever took its place. Structurally, re-keying isn't just risky here, it's disallowed.

So `POST /identity/link` doesn't touch `Assignment` at all. It records a relationship, an anonymous key now points at a canonical known key, and nothing else. Resolution happens entirely at read time: `POST /assign` takes the presented key, resolves it to its canonical identity, and then searches the assignment table across the whole cluster (the canonical key plus every anonymous key linked to it). An assignment made while the visitor was still anonymous is found, not moved. That single design choice removes the window, present in merge-graph systems, where an assignment briefly belongs to no one during a merge, and it makes a link trivially reversible: deleting one relationship row undoes it, with no assignment history to repair.

Identity is also namespaced per account: `UserIdentity`'s primary key is `(account_id, user_key)`, because `user_key` is caller-chosen and an unscoped namespace would let one tenant's identifiers collide with another's. Two accounts can use the identical `user_key` string and never observe each other's data. This is also why `POST /identity/link` gives back the same response whether the key belongs to a different account or was never seen: a distinguishable error there would leak which keys exist.

The one case genuine merge-graph systems handle better is the assignment conflict: both the anonymous and the known key are already independently assigned in the same campaign and version, and `Assignment`'s uniqueness constraint on `(campaign_id, user_key, campaign_version)` means the two rows can't collapse into one. TraqLyte's policy is that the identified assignment wins at read time for every key in the cluster: nothing is deleted, the superseded row is reported in `conflicts[]` and written to the audit log, and it keeps its own outcome history. The honest caveat, and it applies to every vendor in this piece, not just TraqLyte: when that conflict happens, one human really does hold two assignment rows accruing outcomes in two different arms until the warehouse-level dedup work resolves it, and cohort counts mean "assignments," not "people," until then.

04What to audit in your own stack

Most identity corruption isn't a merge-logic bug, it's a delivery bug. The most common anti-pattern across every vendor's own troubleshooting docs is the same shape: assign a variation on the anonymous ID, log the user in, mutate the in-memory context to the new user ID, redirect immediately, and only fire the exposure or identify call after the navigation has already started. Depending on which platform receives which event first, that produces a missing exposure, a split identity, or a silent rebucketing. If your login flow redirects before your tracking calls flush, assume you have this bug somewhere.

Run this checklist against your own instrumentation before trusting a borderline experiment result:

• Pull a sample of converted users and check whether any have more than one assignment row for the same campaign and version: that's double-counting or a live conflict, not noise.
• Check whether your exposure/assignment events carry an idempotency key (Segment's messageId, Amplitude's insert_id, or an equivalent you built yourself). No key means retries and duplicate client/server sends are indistinguishable from real double exposure.
• Confirm exposure calls use sendBeacon, keepalive, or a server-side call, not a client call that a redirect can interrupt mid-flight.
• Decide, explicitly, what logout means for your product: same person continuing anonymously, or a fresh anonymous identity. Then verify your code actually implements that decision instead of leaving it to whatever the SDK defaults to.
• Verify the identifier your assignment logic hashes on is the same identifier your analysis queries group by. A mismatch here produces sample-ratio mismatches and multiple-exposure warnings that look like a tracking bug but are actually an analysis-unit bug.
• Test what happens on a shared device or shared browser deliberately: don't assume it's rare. It's a normal operating condition, not an edge case, for any product with meaningful mobile or kiosk-style traffic.
• If you changed identity-resolution rules recently, confirm you understand they were not retroactive: old profiles were not re-merged, only new events were affected from that point forward.

Estimating the cost of contamination you already have is more tractable than it sounds: take any experiment you shipped as "no significant difference" and re-run the analysis after excluding users with more than one assignment or more than one exposure event in the window. If the effect size moves meaningfully once those units are excluded, you were measuring identity noise, not the treatment. That's usually the fastest way to convince a skeptical team that this is worth fixing before it's worth debating.