Introduction
This Statistical Significance Calculator tells you if your A/B test result is real or just luck. Type in how many people saw each version and how many of them converted. The tool does the math for you right away.
It uses a two-proportion z-test, the standard way to compare two conversion rates. You get a clear yes or no answer, plus the p-value, z-score, uplift, and the smallest gap you would need to call a winner. Charts and a step-by-step breakdown show you how the numbers were found. If you want to dig into a single statistic on its own, try our p Value Calculator or Z Score Calculator.
You can pick your confidence level (80%, 90%, 95%, or 99%) and choose a one-tailed or two-tailed test. The calculator also runs a sample ratio mismatch (SRM) check to warn you if traffic did not split the way you planned, which often means a setup or tracking bug.
Use it to test landing pages, emails, ads, buttons, or headlines, and stop guessing which version really works better. Pair it with our Conversion Rate Calculator and CTR Calculator to track the metrics behind each test.
How to use our Statistical Significance Calculator
Enter the conversions and visitors for both test groups, pick your test settings, and the calculator shows your p-value, z-score, uplift, and whether your A/B test result is statistically significant.
Group A Conversions: Type how many people in your control group did what you wanted, like buying or signing up. This number cannot be bigger than your Group A visitors.
Group A Visitors: Type how many people saw your control version. The tool uses this to work out the Group A conversion rate for you.
Group B Conversions: Type how many people in your variation group converted. Again, keep it equal to or lower than the Group B visitors.
Group B Visitors: Type how many people saw your variation. The Group B conversion rate fills in on its own.
Confidence Level: Pick 80%, 90%, 95%, or 99%. This is how sure you want to be before you call a winner. Most A/B tests use 95%. To turn that level into a range around your result, use the Confidence Interval Calculator.
Test Type: Choose Two-Tailed to check for any difference between the groups, or One-Tailed to check only if one group is better. One-tailed tests reach significance faster. Our Critical Value Calculator shows the cutoff z-scores behind each option.
Expected Traffic Split: Enter the percent of traffic you planned to send to Group A. Group B's share fills in on its own, and the tool runs a sample ratio mismatch (SRM) check to spot split errors in your test setup.
Click Calculate to see your results, charts, and the step-by-step math. Use Reset to load the sample numbers again, or Clear Fields to start fresh.
What Is Statistical Significance?
Statistical significance tells you if a result is real or just luck. In an A/B test, you show two versions of a page. Group A is the control. Group B is the variation. One version may get more sign-ups or sales. But small gaps happen by chance all the time. A significance test checks if the gap is big enough to trust.
How the Test Works
This tool uses a two-proportion z-test. It compares the conversion rate of each group. The conversion rate is conversions divided by visitors. The test then looks at how far apart the two rates are, and how much data you have. More visitors means less noise and a clearer answer. The z-test leans on the bell curve, which you can explore in the Normal Distribution Calculator.
Key Terms
- P-value: The chance of seeing a gap this big if both versions were really the same. A small p-value means strong proof.
- Z-score: How many standard errors apart the two rates are. Bigger numbers mean a stronger signal. See how the spread is built with our Standard Error Calculator and Standard Deviation Calculator.
- Confidence level: How sure you want to be. At 95%, you accept a 5% chance of a false alarm.
- Relative uplift: How much better the winner is compared to the loser, as a percent. The Percent Change Calculator works the same way on any two numbers.
- Percentage points (pp): The plain gap between two rates. 51% minus 49% is 2 pp. Our Percent Difference Calculator helps keep points and percents straight.
One-Tailed or Two-Tailed
A two-tailed test asks, "Is there any difference?" It can catch a win or a loss. A one-tailed test only asks, "Is this version better?" One-tailed tests hit significance sooner, but they miss drops. Most teams should pick two-tailed.
Sample Ratio Mismatch (SRM)
If you planned a 50/50 split, each group should get close to half the traffic. When the real split is far off, something is broken. It could be a tracking bug, a redirect, or bad bot filtering. The SRM check uses a chi-square test to spot this, the same math you will find in our Chi Square Calculator. If it fails, fix your setup first. The test results cannot be trusted.
Tips for Better A/B Tests
- Pick your sample size and confidence level before you start. Our Sample Size Calculator and Margin of Error Calculator help you plan ahead.
- Let the test run for full weeks. Traffic changes by day.
- Do not stop early just because a result looks good. That causes false wins.
- "Not significant" does not mean "no difference." It may mean you need more visitors.
- A tiny win can be significant with huge traffic. Ask if it is worth the work by checking the Effect Size Calculator and running the numbers through a ROI Calculator.
Testing means instead of rates? Use the t Test Calculator for two groups, the ANOVA Calculator for three or more, and the Statistics Calculator for a quick summary of any data set.