You ran the test, the new variant won by 12%, you shipped it, and three months later the conversion rate looks exactly as it did before. That is the normal outcome, and luck has little to do with it. Most “winning” tests were measured badly to begin with. Every one of the usual causes has a known guardrail.
There are seven common reasons, and they fall into three families.
Statistical traps: peeking and the winner’s curse
The first two problems come from how the test was read. Users had nothing to do with them.
1. You peeked and stopped at the first green number
The classic significance test assumes you fixed the sample size before you started and read the result once. Most testing dashboards show a live p-value, so people check daily and stop as soon as it dips under 0.05.
Every look gives random noise another chance to cross the line. In a well-known simulation, a widely cited guide showed that a test of a change with no real effect, checked repeatedly and stopped at the first “significant” reading, produced a false-positive rate of 26.1%. That is more than five times the 5% you think you are getting. The same article shows that with ten peeks, a result that looks significant at 1% is really only significant at about 5%.
So decide the sample size before launch with an A/B test calculator, write it down, and don’t call the test until you get there. If you really do need to stop early, for safety or speed, use a sequential method built for repeated looks. Our sample size tactic goes through the numbers.
2. The test was underpowered, so the winner’s curse picked the overshoot
Few teams understand this one, and it probably does the most damage. When a test lacks the traffic to detect the effect you hope for, a result can only reach significance by overshooting the true effect. The tests you declare winners are disproportionately the lucky ones.
A checkout converting at 3%, with 10,000 visitors in each arm, shows how it works. The standard error of the difference is about 0.24 percentage points. To be significant at 95%, the observed gap has to be at least about 0.47 points, a relative lift of roughly 16%.
Now suppose the change truly lifts conversion by 5% relative (0.15 points). Most of the time the test comes back flat. On the occasions it comes back significant, though, it reports a lift of 16% or more. You ship, celebrate a 16% win, and then live with the real 5%, or less.
A 2026 KDD paper found this outside the textbook. Its authors replicated eight A/B tests of four widely shared UI patterns (rounded buttons, page speed, a coupon-code field and a sticky call-to-action) with a median of 2.4 million users per experiment, and found the previously reported effects were “highly exaggerated”. Only two of the eight replications were significant in the expected direction. One was significant in the opposite direction. The authors put much of the inflation down to the earlier tests being underpowered.
Airbnb’s experimentation team has published a correction for the same bias, described in this summary of its method. Before adding winners’ lifts together, subtract a bias term from each, because on average the raw lifts overstate true impact.
The defence is to run tests at 80% power for the smallest effect you would act on, which is usually smaller than the effect you hope for. If your traffic can’t reach that, test bolder changes, or move to a higher-traffic step of the funnel. The A/B testing statistics lesson explains power in plain terms.
Time effects: novelty, regression and seasonality
The next three come from comparing the wrong moments in time.
3. Novelty wore off
Some changes win because they are new. A redesigned navigation or a bright new banner gets clicked out of curiosity, and the effect decays once returning users are used to it. Microsoft researchers define novelty as “the desire to use new technology that tends to diminish over time”, and its opposite, primacy, as engagement that grows as people adopt a change. Primacy explains why some changes that really are better look worse in week one.
Plot the daily treatment effect alongside the cumulative total. A lift that shrinks every day is a warning sign. Split results by new and returning users too: if only returning users show the lift, curiosity is a likely explanation. Changes to habitual surfaces (navigation, dashboards, pricing pages) need at least two full weeks.
4. Regression to the mean chose your test idea
Teams often test the page that had a terrible month, or the email whose open rate suddenly dropped. Extreme results tend to drift back towards average on their own, so a page tested right after a bad patch may recover during the test. In a proper randomised test that is harmless, because the control recovers too. The trap is the before/after comparison. “We changed the page in March and conversion went up” is not a test, and the recovery would probably have happened anyway.
The same logic bites after the test. Run ten ideas, ship the one with the biggest lift, and you have shipped the extreme of ten noisy numbers. Expect it to slide back towards the middle.
Never claim credit from a before/after chart. Compare against a concurrent control every time, and be most suspicious of the biggest result in a batch.
5. Seasonality and traffic mix moved under you
A test run during a sale, a festival period or a big paid campaign measures how that week’s visitors behave. Diwali shoppers, Black Friday bargain hunters and cold traffic from a new Meta campaign do not behave like your usual mix. A discount-led headline might win with deal seekers and lose with everyone else.
Day of the week changes things as well. B2B sites often see different intent on a Monday from a Saturday, and a test that runs Tuesday to Friday has never met your weekend users.
Run tests in whole weeks: at least one, preferably two. Log any campaigns or sales that overlap. When a test lands on an unusual period, re-run it in a normal one before shipping, or treat the result as specific to that audience.
Broken or cherry-picked tests
The last two come from faulty tests, or from picking the best of many numbers.
6. Sample ratio mismatch broke the test without telling you
You configured a 50/50 split and, with a large sample, the arms ended up 50.8/49.2. Something is wrong with assignment, redirects, bot filtering or logging. This is a sample ratio mismatch (SRM), and it happens more often than most teams assume. According to Microsoft Research, about 6% of A/B tests at Microsoft have an SRM, and about 10% of LinkedIn’s zoomed-in tests had the same bias.
The same article describes an MSN test where the new variant engaged users so strongly that a bot-detection filter removed them from the analysis. The test showed a loss. Once the filter was fixed, the variant turned out positive. An SRM can make a winner look like a loser, and the reverse.
On smaller sites the usual causes are:
- a redirect test where only the variant loads a second page, so slower or blocked visitors drop out of one arm
- a consent banner or ad blocker that stops the testing tag firing more often on one variant
- analysis filters (device, country, “users who saw the new element”) applied unevenly.
Before you look at any result, run a chi-square check on the user counts. Microsoft flags any test with a p-value below 0.0005. If you find an SRM, don’t trust the result until you know the cause.
7. You tested many things and reported the best one
Three variants, eight metrics and four segments make 96 comparisons. At a 5% false-positive rate, pure chance should hand you several “significant” results. Picking the one that came up and building a story round it is how most phantom wins are born.
Some tools correct for this. Optimizely, for example, applies false discovery rate control across metrics and variations, and treats the primary metric separately so it keeps the most statistical power. Many simpler tools apply no correction at all.
Choose one primary metric before launch. Everything else is either a guardrail (does it get worse?) or a lead for the next test, and neither is a reason to ship. If you must test many variants, apply a correction or put the traffic into fewer, bolder arms.
Pre-registration fixes most of this
Nearly every problem above comes from deciding things after seeing the data. Pre-registration means writing those decisions down before launch. It takes ten minutes and leaves far less room for wishful thinking. A short block in your test log is enough:
- Hypothesis, written as “Because we saw X, we believe changing Y for audience Z will move metric M.” The hypothesis design lesson covers this format.
- The primary metric, and the guardrail metrics that must not get worse.
- The minimum detectable effect you would act on, plus the sample size and duration needed to detect it at 80% power.
- The decision rule: what you will do if it wins, loses or comes back flat.
- Known risks, such as campaigns, sales or releases during the window.
When the test ends, compare the result with the plan, not with whatever would be most exciting. Refuse to launch a test that has no block.
A holdout shows whether the wins add up
Run every test properly and a year of “+4% here, +6% there” will still rarely add up to the sum on the slide. A holdout checks the total. You keep a small group of users away from every launch for a fixed period and compare them with everyone else. Statsig’s documentation describes a holdout as a way to see “what all your launches together did to a metric”, and recommends a low single-digit share such as 1% to 2%, kept for three to six months before release.
Reading a cumulative effect takes traffic, so holdouts suit products with a large logged-in base. Smaller teams can do a cruder version: once a quarter, compare the headline metric with the forecast that the shipped lifts implied. A large gap means your individual tests are overstating their effects.
| Why the winner faded | What it looks like | Guardrail |
|---|---|---|
| Peeking | Test stopped early on a big lift | Fixed sample size or sequential testing |
| Winner’s curse | Lift far above what you expected | 80% power for the effect you would act on |
| Novelty | Daily lift shrinks over time | Two full weeks; split new vs returning users |
| Regression to the mean | Test started after a bad month | Concurrent control; no before/after claims |
| Sample ratio mismatch | Arm sizes differ from the split | Chi-square check before reading results |
| Seasonality | Test ran during a sale or campaign | Whole weeks; re-run in a normal period |
| Multiple comparisons | Win on a secondary metric or segment | One primary metric, set in advance |
Re-run your biggest recent winner
Your shipped wins can be checked against the table above. Take the last five. For each, compare the planned sample size with the actual one, test the user counts for an SRM, and look at whether the daily effect was shrinking. Any that fail are unconfirmed, whatever the slide said.
Then work out the minimum detectable effect for your main pages with the A/B test calculator. Above 10%, button-colour tests are a waste of traffic and bigger changes are the only ones worth running. Any future lift above 20% should get a second look for bugs and an SRM before anyone shares it.
The most useful single step, though, is to run your biggest recent winner again. If it holds, you can trust it. If it doesn’t, you’ve learnt more than another win would have taught you.
Questions people ask
Why did my A/B test winner stop working after launch?
Usually because the measured lift was inflated in the first place. Stopping the test early, running it with too little traffic, or picking the best of many metrics all produce wins that are bigger than the true effect. Novelty effects and seasonality can also make a short-term lift disappear once the change is old news.
What is the winner's curse in A/B testing?
When a test has low statistical power, only results that happen to overshoot the true effect cross the significance line. So the tests you call winners are, on average, the ones that overestimated the lift. The fix is to run tests with enough power for the effect you care about, and to discount surprising lifts.
Is it OK to check A/B test results every day?
Looking is fine. Stopping the moment the result turns significant is not, because each look gives chance another opportunity to cross the line. Either fix the sample size in advance and decide once, or use a sequential testing method designed for repeated looks.
What is a holdout group in experimentation?
A small share of users, often 1–2%, who do not get any newly launched changes for a set period. Comparing them with everyone else shows what all your launches did together, which is a reality check on the sum of individual test wins.
Sources
- Guide: How Not To Run an A/B Test
- ACM KDD 2026: Trustworthy A/B Patterns and the Winner's Curse
- Microsoft Research: Diagnosing Sample Ratio Mismatch in A/B Testing
- arXiv: Novelty and Primacy, A Long-Term Estimator for Online Experiments
- Statsig Docs: Holdouts
- Optimizely Docs: False discovery rate control
- Towards Data Science: Airbnb's method for estimating experimentation impact
