A/B Test Sample Size Calculator

Sample size answers, “How much evidence do we need?” Duration answers, “How long will it take to collect that evidence?” You need both. A test that reaches its visitor target in two days can still be unrepresentative if weekends, campaign cycles, or paydays change who arrives.

Calculate the sample before you launch

Enter the current conversion rate, the smallest relative lift worth acting on (your minimum detectable effect, or MDE), expected eligible daily traffic, and confidence level. The estimate assumes a two-sided test, 80% power, and a 50/50 split.

A lower baseline rate or smaller MDE requires more visitors. That is not a calculator problem—it is the cost of separating a subtle signal from ordinary variation. If the result is impractical, test a bigger, evidence-backed change, choose a more frequent meaningful metric, or wait for more traffic. For a diagnosis after an inconclusive result, see why A/B tests fail to reach significance.

How long should an A/B test run?

Run until you meet the pre-planned sample and cover at least one representative business cycle. For many stores that means a minimum of seven days because weekday and weekend behavior differs. A business with a weekly promotion, monthly payday peak, or B2B buying pattern may need a longer cycle.

Seven days is a floor, not a magic finish line. If the calculator says 28 days, ending at day seven produces an underpowered result. If it says three days, keep running through the representative cycle before making a decision. Do not keep a test running indefinitely once the planned decision rule is met; seasonal shifts and changing campaigns can make a long test harder to interpret.

Use eligible traffic, not pageviews

Use the number of visitors who can actually enter the experiment and trigger the primary metric. Exclude bots, staff, visitors outside targeting, and traffic on pages where the variation cannot appear. A product-page test should not use sitewide sessions in its duration estimate.

Why peeking causes false winners

Checking data for technical QA is sensible. Declaring a winner every time the chart crosses a threshold is not. With ordinary fixed-horizon statistics, repeated looks inflate the chance of calling random noise a result. Set the primary metric, target sample, planned end date, and decision rule before traffic starts.

If your platform uses a sequential method designed for continuous monitoring, understand its stopping rules and follow them. Either way, do not change audience, allocation, metric, or variation after seeing early results unless you document the change and treat it as a new experiment.

A practical test-duration checklist

• Choose one primary metric and meaningful guardrails before launch.
• Set an MDE based on commercial value, not the lift you hope to see.
• Calculate visitors from eligible traffic and use a fixed split.
• Plan to reach the sample and a full business cycle—usually at least seven days.
• QA assignment, events, mobile behavior, and the control before launch.
• Review results at the planned decision point, including effect size and guardrails.

The statistical plan is also part of tool selection. A platform should preserve visitor assignment, make exposures and conversions inspectable, and explain how results are calculated. Read our VWO vs Optimizely comparison for the workflow questions to ask before committing to a platform.