How Long Should I Run My A/B or Multivariate Test?

Another of those frequently asked Conversion Rate Optimization questions is ‘How Long Should I Run My Test For?’.

how-long-run-testThe simple answer is until you have produced a statistically significant result that proves there is or is not a difference between the control and the variation.

In truth though many people need to have an idea of the duration before they begin the test, often to provide to higher level colleagues who are trying to plan further tests or changes to a website.

Luckily there is a way that we can estimate how long you should run your test for to get a significant result, I would like to put the emphasis on estimate. This is not a fool-proof approach, however, it should prove to be acceptable 90% of the time.

I often use this approach when producing testing plans so that I can work out the best number of variations to test when I only have a limited time-frame, or to give stakeholders a clearer deadline as to when they are likely to get a result.

The calculations that I use can be complex for people who are not mathematically trained, as many developers tend to struggle with complex arithmetic, so with that in mind I have combined it all into a simple calculator so that you enter in a few details about your website and your test and it can provide you with the number of days that you should run your test for.

You can find the tool here: A/B Split and Multivariate Test Duration Calculator

You are required to enter the current conversion rate of the element that you are testing, the uplift % that you are looking to obtain, the number of variations that you are testing as well as details on how many visits the test will receive and the percentage of those visits that are included in the test.

The calculator then works out how many people are required through each variation to produce a statistically significant result for the test and bashes that together with your current traffic to provide an estimate as to the length of time to run your test.

As I noted before this should be acceptable for 90% of tests but more complex tests may need to have even more complex maths assigned to them.

It would be great to hear whether you find the test duration calculator helpful, or if you notice any problems with it. If you have any questions, concerns or compliments, feel free to leave them in the comments section below.