Never edit a running experiment

01What mid-flight edits do to your data

A running experiment is a promise about what each label means: control meant this, variant B meant that, this population was eligible, this rule assigned them. The moment you edit a live variant, that promise breaks silently. Say you're running an A/B test of a two-step onboarding flow against a five-step control, and three days in someone swaps the variant's button copy from "Continue" to "Generate my plan" and drops a question. The dashboard still shows one line called "Variant," but that line now blends two different treatments: the original flow shown to the first three days of traffic, and a materially different flow shown afterward. If conversion moves, you cannot say whether it was the shorter flow, the new copy, the removed question, or a shift in the traffic mix that week. The result still produces a number. It no longer produces an answer to the question you set out to ask.

Weight changes during a ramp are the same failure wearing a statistics costume: Simpson's paradox. Ramping a variant from 5% to 50% of traffic doesn't just add more users, it changes who is being compared to whom, because the two allocation regimes usually cover different calendar periods with different traffic composition: weekday versus weekend visitors, a marketing push that landed midweek, a mobile app release. A variant can look like it's winning inside each individual ramp stage and still lose in aggregate once the stages are stitched together, because the mix of users differs at each traffic level. Randomization protects a comparison between groups running concurrently; it does nothing to make Monday's 5%-allocation population comparable to Saturday's 50%-allocation population.

Adding a variant mid-flight compounds the same problem into unequal exposure windows. A variant added on day 10 of a 14-day test has accumulated a fraction of the sample size and calendar coverage of the variants that launched on day one; it never saw the early-week traffic pattern, the weekend, or whatever campaign ran in week one. Comparing its conversion rate directly against variants that ran the full window doesn't control for time, it confounds treatment with when the treatment was live. None of this requires anyone to be careless. Optimizely warns explicitly against changing or testing an experiment once live visitors have been bucketed, because certain allocation changes reassign users and can produce a sample-ratio mismatch: a detectable sign that the observed traffic split no longer matches the configured one, and a standard red flag that the underlying data is no longer trustworthy for a decision.

02Peeking, the softer version of the same sin

Editing a variant's content is the loud version of this problem. Peeking at results and stopping the moment you cross a significance threshold is the quiet version, and it's more common because it feels like discipline rather than malpractice. The mechanism is the same one underneath: you're letting information you observed mid-experiment change what "the experiment" ends up meaning, without treating that change as a new experiment.

A significance test computed at a single fixed sample size has a known, bounded false-positive rate: typically the 5% you agreed to when you picked p < 0.05. Check that same test every day as data accumulates and stop the first time it crosses the threshold, and the effective false-positive rate is no longer 5%. Random noise alone will cross the 0.05 line at some point during a long enough monitoring window far more often than 5% of the time, because you've handed yourself dozens of independent chances to get a lucky-looking result and act on the first one. This is not a hypothetical edge case; it's the default behavior of "check the dashboard until it turns green."

The fix is not "never look." It's distinguishing between looking and reacting. Sequential testing methods (designed from the outset to support continuous monitoring, with the false-positive budget spent deliberately across looks rather than assumed away) are a legitimate way to check results early without inflating error rates. What's not legitimate is redefining the hypothesis, the metric, or the audience after seeing early results and then treating the full dataset as though it answered the question you asked at the end rather than the one you asked at the start. The research brief behind this piece calls that out precisely: the deeper problem with peeking isn't the act of looking, it's changing the question after observing a partial answer, then presenting all the data as evidence for the revised question.

03Versioning as the fix

This is why TraqLyte treats a campaign's configuration as mutable only until the moment it starts generating evidence, and immutable after. A campaign's factors, variants, cohorts, and cohort-variant mappings are freely editable while the campaign sits in its current draft version. The instant an Assignment row is created against that version (the first real user gets bucketed), the version locks. Every subsequent write to that campaign's factors, variants, or cohorts is refused: assert_mutable() checks whether any assignment exists under the campaign's current version and, if one does, raises VersionLockedError rather than letting the write through. This isn't a soft warning banner; it's a hard rejection at the one function every config-mutating code path in the system is required to call first, so there's no route in the codebase (admin UI or JSON API) that can bypass it.

Wanting to change something after that point is not a dead end, it's a different action: publish_version(). Publishing takes a full snapshot of the campaign's live factors, variants, and cohorts and writes it into an immutable CampaignVersion.full_config JSON blob, then increments campaign.current_version to open a fresh, empty draft. The old version's snapshot is never touched again: it's a permanent, queryable record of exactly what configuration was live when it was frozen. The new draft is exactly what it sounds like: editable again, with no assignments against it yet, until its first assignment locks it in turn.

The part that actually protects your data is what happens to assignments already on the books: they always keep pointing at the campaign_version they were created under, and they are never migrated forward to a newer version. A user assigned under version 3 stays recorded as having seen version 3's configuration forever, even after you've published versions 4 and 5. That's what makes "what did this person actually see" a question with one unambiguous answer instead of a moving target; that's the same guarantee the research behind this piece calls the difference between an editable current-state system and an append-only one: corrections happen by adding new facts, never by rewriting old ones. A typo fix, a weight change, a new variant: all of it is a new version, with its own clean set of assignments, not a quiet edit to the version that's already collecting evidence.

04What to look for in your platform

Every serious experimentation platform has arrived at some version of this rule, even where the product language differs. LaunchDarkly ends the current "iteration" the moment experiment-defining configuration changes and starts a new one, so each set of results maps to exactly one configuration. Optimizely documents that QA has to happen before live visitors are bucketed, and that changing allocation after launch can rebucket users and produce a sample-ratio mismatch. Amplitude has you create a new run rather than edit an existing web experiment, and separately warns that changing the bucketing salt can flip which variant a user lands in. Firebase keeps participants persistently assigned to their variant for the life of an experiment, but still warns that changing app behavior mid-run can affect results: persistence of the assignment label doesn't help if what's behind the label changes. The pattern across all of them: stopping a run, duplicating it, and publishing a corrected version are all fine. Silently rewriting the version that already produced evidence is not, on any of them.

When you're evaluating a platform (including this one), the honest question isn't "can I edit a running experiment," it's what happens when you try, and what the platform can show you afterward. A short checklist:

• Does a config-changing write against a live experiment get rejected outright, or does it silently succeed with just a warning toast?
• Is the rejection enforced at the API layer, or only hidden behind a disabled button in the admin UI? A rule that only the UI enforces isn't enforced: anything that can call the API directly walks straight past it.
• Can every past assignment and outcome be traced back to the exact configuration that was live when it happened, or only to whatever the config looks like today?
• Is there an append-only audit trail of publish/version events, or can history be quietly overwritten?
• Is "make a correction" a first-class, one-step action (duplicate the config, edit the draft, publish), or does it require working around the platform's own guardrails?

A platform that can't answer the third question with a straight yes isn't giving you experiments. It's giving you a live configuration with a chart bolted on, and charts drawn against a moving target are the fastest way to make a bad decision look statistically sound.