A/B testing is an ongoing process, involving tweaks large or small to see if version A, the control, or B, the modified variable, leads to a higher conversion rate. 

Running an A/B test is a lot like conducting a science experiment. A proper experiment follows the scientific method:

1. Identify a problem
2. Conduct research
3. Form a hypothesis
4. Perform experiment
5. Analyze the results
6. Conclusion

If you were to jump right into your experiment and start mixing chemicals together to “see what happens” without first identifying a problem that needs solving and doing background research, not only would you have a real mess on your hands, but you wouldn’t be able to make an accurate conclusion. 

Similarly, if you start A/B testing at random, you won’t be able to determine if your tests are finding the most effective solution, and you could end up making inaccurate conclusions. Not to mention, you’d be spending a lot of time running inefficient tests.

Just as a good science experiment should abide by the scientific method, so should your process of A/B testing. Let’s use a simple example to dive into what that looks like.

1. Identify a problem

Firstly, you’ll want to identify a problem area. This could happen in a number of ways. Tools to help you uncover issues include website analytics, user surveys, heatmaps, survey responses, and user testing results. 

For example, let’s say you are the Director of Marketing for the Really Cool T-shirt Company and you sell really cool t-shirts. You’ve had great success in local shops, and have recently launched a website that sells directly to the customer. Your site traffic is looking good, but you aren’t making many sales. 

Looking at a heatmap of your product page, you notice that after opening a product page, selecting t-shirt size and color, users are then moving the mouse erratically, eventually leaving the page. After checking the exit rate and user path, you may wonder if users are struggling to find your “add to cart” button.

2. Conduct research

Now that you’ve identified the potential problem, you’ll want to conduct research to help you formulate your hypothesis. 

In this case, let’s say you research best practices for calls-to-action, case studies on how other ecommerce companies solved a similar issue, and ideas for A/B tests to run on your “add to cart” button. 

Sometimes, the research is clear on what to do to solve the problem you’ve identified. 

Other times the research won’t be so clear, and that’s okay! The goal with A/B testing is to find out what works for your ecommerce site and your users, even if it goes against popular opinion.

3. Form a hypothesis

Armed with research, you can now form the hypothesis that will guide your testing. 

Let’s say your current “add to cart” button looks like the call-to-action below.

Add to Cart

Now, you’ll only want to test one of these changes at a time. For the sake of keeping this example simple, let’s say that from your research, you’ve learned a call-to-action should be large, should contrast the page background, and should appear clickable – all things your current CTA is not. 

Why? 

Because if you test multiple variables at once, you won’t be able to distinguish the precise impact of each variable. If you decide to make your “add to cart” button three times bigger, bright orange, and beveled, and your conversion rate doubles, how will you know which of the variables caused the increase? What if one of the variables had a negative effect and it’s being masked by the other two? 

By testing only one variable at a time, you can directly attribute the results to that very specific change. 

Bearing that in mind, your first hypothesis could be that by tripling the size of your CTA, you will see an improvement in your conversion rate.

4. Perform experiment A/B test!

To run your A/B test, you’ll need a testing tool. If you use Magento or Shopify, they both have their own testing extensions. Otherwise, you can use another platform like Google Optimize (free!), Optimizely, or Visual Web Optimizer. 

Most testing tools are intuitive and easy to use. Once you’ve selected your testing platform, you can set up your test. 

Similar in basic functions, each tool will allow you to create two versions of a webpage – version A, the control, and version B, the modified. The versions are then shown at random to a set percentage of users, for instance, 10% of users will see version A, and 10% will see version B. 

You’ll want to run the test for at least a week to account for any weekday versus weekend fluctuations. How long to let your test run depends to an extent on the popularity of your site and the amount of traffic it receives. 

The test will automatically turn off when it reaches statistical significance. If you receive a high volume of web traffic, you’ll reach statistically significant results much faster than a less frequently visited site.

5. Analyze the results

Now for the fun part: checking out your results! Which version led to a higher conversion rate, A or B? 

Were the results statistically significant? Statistical significance is the likelihood of a relationship between two variables is caused by something other than chance. 

This is especially important. Typically, you want to set a confidence level of 95%. This means you only accept the variation being testing over the control if you can be 95% certain the variation will outperform the control. In other words, there is only a 5% chance that version B outperformed version A due to random chance.  

The smaller the change you are testing, the higher you want your confidence level to be. This is because, with minor changes, random variance plays a bigger role than it does with more dramatic changes.

6. Conclusion

With the results in and analyzed, you can decide what action to take. 

Was your hypothesis correct? If so, you can go about implementing the new option, and then get started on your next A/B test!

In Summary

A/B testing is challenging. More often than not, your sales aren’t going to shoot through the roof only because you change the color of your “add to cart” button.

Focus on incremental, iterative improvements, not massive gains. It’s like going to the gym. Dramatic changes aren’t likely to happen overnight, and making a lot of huge changes at once involves a lot of risk. 

With a strategic plan and a bit of hard work, your results will build on each other, and you’ll see a great return. 

Looking for a partner to help with all your ecommerce site projects? We’ve helped brands like Royal Farms, Under Armour, and Wendy’s bring their visions to life – and we’d love to help you too. Drop us a line and we’ll get started.