Split and A/B Testing Confidence Calculator

Treatment Visitors
Treated
Conversions Conversion
Rate
Z-Score Confidence Recommended
Sample Size*
Control
Treatment 1
Treatment 2
Treatment 3
 

Brought to you by Céondo, science on the web.

How to Use the A/B Test Calculator:

You need to provide at least the control data and the first treatment. The number of visitors treated is for example the number of unique visitors getting the given banner add and the conversions are the number of them clicking on it.

A treatment is statistically better than the control treatment if the confidence is above 95%.

*: Confidence intervals of 25%, 15% and 5%. The confidence interval is also the percentage of change you expect from your treatments.

The Maths Behind

I created this calculator because all the calculators I found on the web did not include the maths behind them. So here you have, links to the maths:

Logo

PHP Code to do the Calculations

The code is extracted from the two previous links.

<?php

$c = array(182, 35);
$tA = array(180, 45);
$tB = array(189, 28);
$tC = array(188, 61);

function cr($t) 
{ 
    return $t[1]/$t[0]; 
}

function zscore($c, $t) 
{
    $z = cr($t)-cr($c);
    $s = (cr($t)*(1-cr($t)))/$t[0] + (cr($c)*(1-cr($c)))/$c[0];
    return $z/sqrt($s);
}

function cumnormdist($x)
{
  $b1 =  0.319381530;
  $b2 = -0.356563782;
  $b3 =  1.781477937;
  $b4 = -1.821255978;
  $b5 =  1.330274429;
  $p  =  0.2316419;
  $c  =  0.39894228;

  if($x >= 0.0) {
      $t = 1.0 / ( 1.0 + $p * $x );
      return (1.0 - $c * exp( -$x * $x / 2.0 ) * $t *
      ( $t *( $t * ( $t * ( $t * $b5 + $b4 ) + $b3 ) + $b2 ) + $b1 ));
  }
  else {
      $t = 1.0 / ( 1.0 - $p * $x );
      return ( $c * exp( -$x * $x / 2.0 ) * $t *
      ( $t *( $t * ( $t * ( $t * $b5 + $b4 ) + $b3 ) + $b2 ) + $b1 ));
    }
}

function ssize($conv)
{
    $a = 3.84145882689; 
    $res = array();
    $bs = array(0.0625, 0.0225, 0.0025);
    foreach ($bs as $b) {
        $res[] = (int) ((1-$conv)*$a/($b*$conv));
    }
    return $res;
}


print zscore($c, $tA).' - '.cumnormdist(zscore($c, $tA))."\n";
print zscore($c, $tB).' - '.cumnormdist(zscore($c, $tB))."\n";
print zscore($c, $tC).' - '.cumnormdist(zscore($c, $tC))."\n";
print '1.645 - '.cumnormdist(1.645)."\n";