The one test that everyone should run
Aug 19, 2017There is one test that every single site should run regardless of what type of site it is. It will tell you a lot about your traffic and a lot about why you need to reach significance before calling a test.
I call it a variance test. A variance test is a test that measure the amount of noise and variance in your traffic. Its a brute force way of figuring out how many conversions it will take to reach significance and also how big of a lift you will need to produce in order to fill confident about it.
Setting up a variance test
The test is simple to set up and you can do it in about 5 mins. To set it up, create a new activity in Adobe Target or your favorite testing software. Then create 5-6 experiences that are exactly the same and have absolutely no changes to them. So rather than an A/B test, you have an A/A/A/A/A test. And that’s it. Go ahead and set it live.
This test will need to run just like any other test. It may take longer because there are 5 experiences. What you generally see with a variance test like this, is that one experience will tend to be “performing” better than the rest, even though its exactly the same as the others. At about 100 conversions per experience, you may see that one test is reporting a 20% lift or more. Which we know is impossible because all of the tests are exactly the same. This is the variance. In other words, its the noise of all of the other things that are happing that go into affecting whether or not a user makes a purchase and how that’s reported in a test.
Not the same widget
For example, I used to work in the travel/reservation industry. Think of all of the options that different users may face when making a reservation. One user might be looking at a 4 star $300 a night hotel, while another user is looking at a 2.5 star $100 a night room. And that just the start. Other things might include, length of stay, how many days until the trip, business or vacation, location, star rating, price per night, availability and a bunch of other factors. Each play into whether or not someone would book the room. In other words, we weren’t just selling the same widget to every single customer.
Convergence
The longer you run the test, the more the variance in lift between each experience shrinks. At 500 conversions per experience, the variance comes down and you may see the “winning” test only reporting a 10% lift. At 1,000 conversion each, the variance often comes down to around 7%. I typically run actual tests to at least 1,500 conversion per experience and hope to see at least a 5% lift. Sometimes you have to run them even longer to get confidence in the lift.
Because of variance, I find it hard to measure lift that is less than 4-5%. You could run a test for a very long time to try to get to confidence, but the amount of time seems to grow exponentially the smaller the lift you are trying to measure.
Use the variance test to determine how many conversions you need to reach before you can feel confident that you have reached significance. Also use it to determine the minimum lift that you feel confident about calling a winner. In my experience, its around 1,500 conversion per experience and a minimum of about 5%.