An AB Test Calculator is a statistical tool used in marketing, web design, and product development to compare two versions of a single variable to determine which performs better in achieving a specific goal.
An e-commerce website use an AB Test Calculator to determine whether changing the color of their “Buy Now” button from blue to green increases the number of purchases. They would show the blue button to one group of users (Group A) and the green button to another group (Group B), then use the calculator to determine if there’s a statistically significant difference in the conversion rates between the two groups.
AB Test Calculation Chart
| Metric | Version A | Version B |
|---|---|---|
| Sample Size | 10,000 | 10,000 |
| Conversions | 500 | 550 |
| Conversion Rate | 5.00% | 5.50% |
| Improvement | – | 10% |
| P-value | – | 0.0312 |
| Confidence Level | – | 96.88% |
AB Test Formula
One common formula used is the two-proportion z-test:
z = (p1 - p2) / sqrt((p (1-p) (1/n1 + 1/n2)))Where:
- p1 and p2 are the conversion rates of the two versions
- p is the pooled proportion
- n1 and n2 are the sample sizes
- p1 = 500/10000 = 0.05
- p2 = 550/10000 = 0.055
- p = (500 + 550) / (10000 + 10000) = 0.0525
- n1 = n2 = 10000
Plugging into the formula:
z = (0.055 - 0.05) / sqrt((0.0525 0.9475 (1/10000 + 1/10000)))
z ≈ 2.15This z-score can then be converted to a p-value or confidence level.
How to Calculate Sample Size for AB Test?
Calculating the sample size for an AB test involves considering several factors:
Baseline conversion rate
Minimum detectable effect
Statistical significance level (usually 95%)
Statistical power (usually 80%)
The formula is complex, so most people use online calculators or statistical software. However, here’s a simplified approach:
Sample Size per Variation ≈ 16 (p (1-p)) / (minimum detectable effect)^2Where p is the baseline conversion rate.
Let’s say your current conversion rate is 5%, and you want to detect a 20% relative improvement (i.e., an absolute improvement of 1 percentage point).
- Baseline rate (p) = 5% = 0.05
- Minimum detectable effect = 1% = 0.01
Sample Size ≈ 16 (0.05 0.95) / (0.01)^2 ≈ 7,600 per variationThis means you’d need about 7,600 visitors in each group, or 15,200 total, to have a good chance of detecting a 1 percentage point improvement in your conversion rate.
Related Statistics Tools