Split and A/B Testing Confidence Calculator

Brought to you by Céondo, science on the web.

How to Use the A/B Test Calculator:

You need to provide at least the control data and the first treatment. The number of visitors treated is for example the number of unique visitors getting the given banner add and the conversions are the number of them clicking on it.

A treatment is statistically better than the control treatment if the confidence is above 95%.

^*: Confidence intervals of 25%, 15% and 5%. The confidence interval is also the percentage of change you expect from your treatments.

The Maths Behind

I created this calculator because all the calculators I found on the web did not include the maths behind them. So here you have, links to the maths:

• A/B and Split Testing by Céondo Ltd (Feed).
• Statistical Analysis and A/B Testing, it will provide you a very nice introduction to A/B testing and the formula to calculate the Z-Score of a treatment.
• Calculating the Cumulative Normal Distribution, to calculate the confidence based on the Z-Score.

PHP Code to do the Calculations

The code is extracted from the two previous links.

• cr: Calculation of the conversion rate.
• zscore: Calculation of the z-score.
• cumnormdist: Calculation of the cumulative normal distribution.
• ssize: Given a conversion rate, calculate the recommended sample size for different confidence intervals (0.25 worst, 0.15, 0.05 best) at a 95% confidenc.

<?php

$c = array(182, 35);
$tA = array(180, 45);
$tB = array(189, 28);
$tC = array(188, 61);

function cr($t) 
{ 
    return $t[1]/$t[0]; 
}

function zscore($c, $t) 
{
    $z = cr($t)-cr($c);
    $s = (cr($t)*(1-cr($t)))/$t[0] + (cr($c)*(1-cr($c)))/$c[0];
    return $z/sqrt($s);
}

function cumnormdist($x)
{
  $b1 =  0.319381530;
  $b2 = -0.356563782;
  $b3 =  1.781477937;
  $b4 = -1.821255978;
  $b5 =  1.330274429;
  $p  =  0.2316419;
  $c  =  0.39894228;

  if($x >= 0.0) {
      $t = 1.0 / ( 1.0 + $p * $x );
      return (1.0 - $c * exp( -$x * $x / 2.0 ) * $t *
      ( $t *( $t * ( $t * ( $t * $b5 + $b4 ) + $b3 ) + $b2 ) + $b1 ));
  }
  else {
      $t = 1.0 / ( 1.0 - $p * $x );
      return ( $c * exp( -$x * $x / 2.0 ) * $t *
      ( $t *( $t * ( $t * ( $t * $b5 + $b4 ) + $b3 ) + $b2 ) + $b1 ));
    }
}

function ssize($conv)
{
    $a = 3.84145882689; 
    $res = array();
    $bs = array(0.0625, 0.0225, 0.0025);
    foreach ($bs as $b) {
        $res[] = (int) ((1-$conv)*$a/($b*$conv));
    }
    return $res;
}


print zscore($c, $tA).' - '.cumnormdist(zscore($c, $tA))."\n";
print zscore($c, $tB).' - '.cumnormdist(zscore($c, $tB))."\n";
print zscore($c, $tC).' - '.cumnormdist(zscore($c, $tC))."\n";
print '1.645 - '.cumnormdist(1.645)."\n";