It feels harmless: the test is running, one variant has a typo, you fix it. But the users assigned before and after your edit saw different things under the same label, and your results are now a blend of two experiments neither of which you can analyze. This is why TraqLyte locks a campaign version the instant the first assignment lands.
This article is a stub.
The outline below is what the finished post will cover. Want it sooner? Tell us.
01What mid-flight edits do to your data
- Variant edits: two treatments, one label
- Weight changes: Simpson's paradox in traffic ramps
- Adding variants: unequal exposure windows and seasonality bias
02Peeking, the softer version of the same sin
- Why checking significance daily inflates false positives
- Sequential testing as the legitimate alternative
03Versioning as the fix
- Draft, locked, published: the lifecycle
- Snapshots: every assignment permanently tied to the config it saw
- Changing your mind means publishing a new version, not rewriting history
04What to look for in your platform
- Does it stop you, warn you, or silently allow it?
- Audit trails as the test of seriousness
Research brief
“Find documented cases and statistical explanations of mid-experiment configuration changes invalidating results: peeking, changing traffic allocation mid-test (Simpson's paradox in ramp-ups), adding variants mid-flight, and how platforms enforce config immutability or version snapshots.”
The deep-research question this article will answer, sources cited in the finished piece.