A/B testing: What it is and how to do it

What is A/B testing?

Also known as split testing, A/B testing enables the comparison of two alternate versions (A and B) of a digital asset, such as an app, advert or webpage, to measure which one performs better. Having this data enables adjustments to be made to individual design elements or entire marketing campaigns to achieve better conversion rates and overall outcomes.

How can A/B testing influence User Experience (UX)?

When it comes to user experience (UX), A/B testing generally focuses on the functionality and usability of a website or product and can provide powerful data revealing which elements support a better overall user experience. Going through the process of testing and iterating on results can create an overall more engaging and effective experience for all users, with potential changes being the placement of a CTA (call to action), or the flow of a user journey. 

How does A/B testing influence design?

An A/B design test will be focused on the visual or graphic design of a website or product. This could include elements such as imagery, layout, font or colour palette. Ultimately the goal for an A/B design test is to determine which version is more visually engaging, or more likely to capture and maintain a user’s attention and is typically used to improve the aesthetics.

Marketing reasons for A/B testing

Some of the key reasons A/B testing could be used within a marketing function could be to:

• Hit higher conversion rates
• Increase website traffic
• Reduce bounce rate
• Decrease cart abandonment

How to conduct A/B testing

Follow these eight key steps to run your own A/B test:

• Pick one variable test

Identify a specific element that you want to test on your website or within your product. For example: a headline, product image or call to action. 

• Create a ‘control’ and a ‘challenger’

The control version is the original version of your asset and the challenger the new version, which includes the changes you want to test.

• Split your sample groups equally and randomly

Assign users to either the control or challenger groups and do so randomly, but try to make the groups of equal size. 

• Determine your sample size

Decide on the smallest number of users needed to achieve statistically significant results. An A/B testing calculator can help you figure out the optimal sample size. 

• Decide how significant your results need to be

Determine the level of confidence you need to declare a ‘winner’. This is often expressed as a p-value or confidence interval. 

• Use an A/B testing tool

Choose and use an A/B testing tool. Set up and run the test. Most tools will randomly assign users to the control or challenger group, then track and report on the results. 

• Test both variations simultaneously

Run both variations of your product or website at the same time and for the same period of time. This reduces time-based biases.

• Take action based on your results

Analyse the results of your test to identify the winner. If the challenger version outperforms the control, put the relevant changes in place. If not, try a different test. 

What features to look out for in a quality A/B testing tool

If you’re in the market to purchase an A/B testing tool, keep these features front of mind while you research.

• Easy to use: Whether you’re technically minded or not, you should be able to easily use the tool and it will ideally have a simple interface for creating experiments
• Real-time reporting: Know your experiment results right away so you can monitor the progress and make informed decisions
• Technical support: Having help with set up and general troubleshooting is important to get the outcomes you need from the tool
• Integration: Ensure that your solution can integrate with analytics tools to give you a comprehensive view of user behaviour and campaign performance
• Customisation: Look for a tool that provides flexibility in creating the experiment design, such as goal setting, and targeting specific user segments

Best A/B testing tools to try

There are many A/B testing tools available, each catering to different testing needs and with their own pricing models. We’ve rounded up five to get you started.

1 Google Optimise

Part of the Google marketing platform and natively integrated with Google Analytics, this solution is easy to use and customisable. Best of all, it’s free to use.

2 Crazy Egg

A great option for small businesses, Crazy Egg focuses on optimising websites and supports you in testing site pages with detailed reporting. There is a monthly charge, but you can trial it free for 30 days.

3 Optimizely

A popular tool, Optimizely offers a range of experimentation types including A/B testing, multivariate testing and personalisation. The interface is easy to use and integrates well with popular marketing tools. 

4 AB Tasty

Offering advanced targeting, user segmentation and a low code/no code testing approach, AB Tasty also has strong personalisation capabilities.

5 VWO

For enterprise-level use, VWO is an in-depth A/B testing and conversion optimisation platform offering a range of testing and heatmaps.