Calculating percentage confidence
For split testing, we need a mathematical way to determine the percentage confidence with which we can ascertain whether the means of two variants are different.
For example, let's assume we are sp
For split testing, we need a mathematical way to determine the percentage confidence with which we can ascertain whether the means of two variants are different.
For example, let's assume we are split testing variant A and variant B of a landing page. When a user hits the landing page, they are randomly assigned either variant A or variant B with a 50% probability.
Here's the question: let's say we have intermediate results as follows:
- Variant A: 18 signups
- Variant B: 32 signups
Resolution:
- Chi-squared test for overall detection of ANY difference between variants, z-score to test difference in means between individual variant vs. mean of all variants.
On March 7, 2025, Mike Morton initiated a Socra titled "Calculating percentage confidence," focusing on the statistical analysis of A/B testing for two landing page variants: A and B. The experiment involved split testing with a 50% probability assignment to each variant, aimed at determining if their means were significantly different.
The initial results showed that variant A garnered 18 signups, while variant B achieved 32 signups. To assess the differences, Mike employed a Chi-squared test to identify any overall discrepancies between the variants. Additionally, a z-score was calculated to evaluate the difference in means specifically between the individual variants and the overall mean.
Through this statistical analysis, Mike aimed to derive a percentage confidence level that would substantiate the effectiveness of variant B over variant A, providing clarity for future marketing strategies. The Socra remains incomplete, but it encapsulates a methodical approach to A/B testing and the importance of rigorous statistical methods in decision-making.By Mike Morton