Landing page A/B testing: plan the decision before the experiment

Choose a question the result can answer

A landing page A/B test compares versions shown to randomly assigned visitors and measures a defined outcome. It can help you judge a change, provided the assignment, measurement, and analysis are sound. Comparing this week’s redesigned page with last week’s page leaves other changes, such as traffic source or demand, mixed into the result.

Start with an observed problem and a proposed explanation. Perhaps customers repeatedly ask whether a bag fits their laptop. The test could move verified dimensions near the product choice. Write down what you expect to change and how you will measure it. GOV.UK’s testing guidance emphasizes defining the hypothesis and outcome before building the test.

Source: GOV.UK Data Community: A/B and multivariate testing

Foldwave: test where shoppers find sizing information

Foldwave shows bag categories, a carry-selection idea, and construction details. Imagine a real shop finding that visitors struggle to choose between two sizes. One version could keep its existing layout while another places the relevant capacity and fit information beside the choice. The product, price, and purchasing conditions would stay the same.

That test concerns information placement. Changing the headline, price, imagery, and checkout at the same time would answer a broader question about the whole package. It could not tell you which individual change caused the result. Choose the scope that matches the decision you need to make.

Define success and the problems you need to watch

Select a primary outcome that represents the intended business result. For a shop, that may be completed purchases per assigned visitor. For a service, it may be qualified inquiries, with qualification defined before the test. Keep supporting measures, such as clicks or form starts, available to explain behavior without silently promoting whichever measure happens to improve.

Also watch for harm elsewhere: errors, cancellations, unusable requests, or slower loading. Microsoft’s experiment-design guidance recommends measurable hypotheses, suitable success measures, and checks for regressions and data quality. Write these choices down so the team can judge the result against the original question.

Source: Microsoft Research: Patterns of trustworthy experimentation: pre-experiment stage

Decide whether you have enough traffic for the question

A small difference can require a large sample to distinguish from normal variation. Plan the sample using the baseline outcome rate, the smallest effect worth acting on, and the statistical method. Microsoft’s guidance includes statistical power in that planning. Include the time needed for delayed outcomes, such as a lead becoming qualified.

There is no reliable rule that every landing page needs the same number of visitors or the same run length. If the required sample would take too long, use customer conversations and task observation to find clear usability problems. Fix a broken form directly. You do not need an experiment to justify making a required action work.

Source: Microsoft Research: Patterns of trustworthy experimentation: pre-experiment stage

Check assignment and measurement before interpreting the lift

Verify that visitors receive the intended version, that repeat visits behave as planned, and that both versions record the outcome the same way. Keep test visits separate where your measurement setup permits it. Check the actual split against the planned allocation rather than assuming the testing tool delivered it correctly.

An unexpected imbalance can indicate a sample ratio mismatch. Microsoft describes this as a reason to investigate missing, duplicated, or unevenly observed data before trusting the result. A prettier chart does not repair a broken experiment. Pause interpretation, find the cause, and decide whether the test needs to restart.

Source: Microsoft Learn: Experiments best practices and recommendations

Use a stopping rule that matches the analysis

For a fixed-horizon test, plan the sample or end condition and avoid declaring victory just because an early result looks favorable. Repeatedly checking and stopping on a favorable result can increase false positives. Research on anytime-valid confidence sequences describes methods designed for continuous monitoring; those methods need an appropriate analysis, not simply more frequent dashboard visits.

Read the testing system’s statistical documentation and agree on the decision rule before launch. Stop immediately for a serious operational failure when necessary, but record that as an interrupted experiment. A test halted because checkout broke has not established which design customers prefer.

Source: arXiv: Anytime-valid confidence sequences in an enterprise A/B testing platform

Keep the result with the versions you tested

Record the two page versions, audience, dates, primary metric, uncertainty, data-quality findings, and decision. An inconclusive result can mean the test could not resolve the effect; it is not proof that the versions perform identically. Report the size and limits of what you observed.

Waveform supports editing and reviewing page versions. Random traffic assignment and statistical analysis require an experiment setup that supports them; this guide does not describe an included testing service. After choosing a version, verify the published page and keep watching the business outcome. The experiment answers the question for the conditions you tested, which may change over time.