What to expect with your first A/B test
Aug 16, 2017After weeks of planning, designing and anticipation, you’re about to launch your first test. You’re excited. And you should be. You see great things ahead. Plus, you’ve spent months working up to this moment. Its not just the setup for this test. Its all the work that you have put into it over the past months to get to this point. You had to convince your boss, negotiate a contract for testing software, implement it on your website, design a test and navigate some serious office politics. And now its time to set it live and see what it can do.
But the truth is, there is a good chance that this test, the one you have put so much work into, is going to be a dud. That’s right. Its probably going to come back inconclusive, meaning it will probably end in a tie. But That’s OK. The reality is, 9 out of 10 tests are not winners and the odds of you hitting a home run your first time at bat are low, if not impossible.
But don’t get discouraged. There are great things ahead and you will find wins. Significant wins that will grow your business.
A key part of testing is setting expectations. You’ve probably heard great stories of people running a single test and finding a 40% lift. They make it sound so easy and make it sound like a 40% lift happens every day. And you’re hoping for the same. And maybe even your boss expects the same. Here are some guidelines for your first test.
Expect to fail
As I said before, most test come back inconclusive so be prepared, and prepare those above you, that part of testing is failing, or at least not always winning. It is important to set this expectation with the executives above you. You don’t want them go in thinking that your first test will be a win, because it probably won’t.
Set Expectations
Set the expectation that testing and optimization is a long term plan. Over time, the wins will roll in but it will take time. Set a long term plan with them that shows how many tests you think you can run in a quarter and how many wins that will get you. So if you have enough traffic to run 10 tests in a quarter, then you can expect 1 win per quarter.
Don’t shoot for the moon
Everyone wants a big win at first and the temptation is to shoot for the moon. But don’t. When you shoot for the moon, you will often spend weeks working on, designing and setting up a test only to have it end in a tie. This can eat up a lot of resources. You’ll feel discouraged and feel like you put a lot of time into something that didn’t work out. And the more resources that it uses, the more that people will feel like you’ve invested so much time it “Has to win”. In my experience, tests that get so built up that they “Have to Win”, usually don’t. The more time you put into it, the more you’ll feel discouraged. Instead, start with something that is both high potential for impact and also reasonably easy to implement.
First Ideas
Don’t test the first idea you have. I would recommend making a list of all the ideas that you have. Write them all down and then rank them by both potential for impact and difficulty to implement. Ideally, you find an idea that is both high impact and somewhat easy to implement. Potential for impact should be as close to a quantifiable number as possible. Something like X number of increased sales or signups.
Measurement
Make sure you know how you are measuring the test. I have seen many tests run, only to get a week in and realize that we weren’t measuring a key metric and we were missing key data. So the test has to be paused, fixed and re-started. Wasting a week of testing and time.
The most important thing with your first test is to get one going. Learn from the experience and plan to test over and over again until you find the wins. You may also want to consider running a Variance Test.