Free tool
A/B Test Significance Calculator
Check whether your A/B test result is statistically significant. Enter visitors and conversions to get the p-value, confidence level and uplift instantly.
Variant A · control
Variant B · change
Confidence the difference is real
98.4%
B converts 22% better than A.
- A conversion rate
- 5%
- B conversion rate
- 6.1%
- p-value
- 0.0163
- z-score
- 2.402
Significant: B wins
At 95% confidence, the difference is unlikely to be random chance. If the test ran for full weeks and you didn't stop it early, it's reasonable to ship B.
How the significance test works
The calculator runs a two-sided two-proportion z-test. It compares the conversion rates of A and B against how much they would naturally wobble given your sample size.
- Conversion rates:
pA = conversionsA ÷ visitorsAandpB = conversionsB ÷ visitorsB. - Pooled rate:
p = (conversionsA + conversionsB) ÷ (visitorsA + visitorsB). - Standard error:
SE = √(p × (1 − p) × (1/visitorsA + 1/visitorsB)). - z-score:
z = (pB − pA) ÷ SE, and the p-value comes from the normal distribution.
How to read the result
Confidence is 1 − p-value. At 95% or above, the difference is unlikely to be random. Below that, you can't tell yet. That doesn't mean B is worse, only that you don't have enough evidence.
Example
A: 1,000 visitors, 100 signups (10%). B: 1,000 visitors, 130 signups (13%). The uplift is 30%, z ≈ 2.10 and the p-value ≈ 0.036, so you can be about 96% confident B is genuinely better.
Common mistakes
- Peeking. Checking every day and stopping when it turns green produces many false winners. Plan a sample size with the sample size calculator first.
- Stopping mid-week. Weekend visitors behave differently. Run whole weeks.
- Counting the wrong thing. Make sure each visitor is counted once and sees only one variant.
Frequently asked questions
What does statistically significant mean?+
That the difference between A and B is unlikely to be random chance. At 95% confidence, a difference this big would show up by chance less than 5% of the time if A and B were really the same.
Which test does this calculator use?+
A two-sided two-proportion z-test with a pooled standard error, the standard test for comparing two conversion rates.
Can I stop my test as soon as it hits 95%?+
No. Checking repeatedly and stopping at the first significant result inflates false positives a lot. Decide the sample size up front and evaluate once you reach it.
What's a p-value?+
The probability of seeing a difference at least this large if there were truly no difference. Lower is stronger evidence. 0.05 is the usual cut-off.
How many conversions do I need?+
There is no magic number, but with fewer than about 100 conversions per variant results swing a lot. Use the sample size calculator to plan.
More free tools
Numbers like these, for your own site.
Cool Analytics shows conversions, sources and every visitor's journey. No cookies, one line of code.