A Beginner’s Guide: How to do A/B Testing
The decision-making process in design has long been a prominent topic of discussion. Why do some designers make decisions that others do not, and why do specific layouts appear to perform better than others?
Google evaluated not only the two shades of blue but also the 39 hues of blue in between to select which to use for link text. But why did Google experiment with 41 different hues of blue, and how could a similar strategy benefit you or your organization? In this post, we’ll look at A/B testing, including what it is, why you should use it when to use it, how to run an A/B test, and its limits.
What is A/B testing?
A/B testing, also known as split testing, is a randomized experimentation process in which two or more versions of a variable (web page, page element, etc.) are shown to different segments of website visitors at the same time to determine which version has the greatest impact and drives business metrics.
Why should you do A/B test?
The goal of A/B testing is to provide a sample of customers (the experimental group) the variation version of the product and another sample of customers the existing version of the product (the control group). The difference in product performance between the experimental/treatment group and the control group is then tracked to identify the impact of this new version(s) of the product on the product’s performance. So, the purpose is to follow the statistic during the testing period to determine whether or not there is a difference in the product’s performance and, if so, what kind of difference there is.
“A/B testing can be beneficial, but it should not be at the price of other aspects of design.”
The goal of this test is to evaluate new product variations that will improve the performance of the present product and make it more effective and optimal, resulting in a favorable therapeutic impact.
Questions to ask before doing an A/B test
- What does a sample population look like, and who are the target product’s client segments?
- Can we use exploratory/historical data analysis (e.g., causal analysis) to uncover the solution to our business question?
Do we want to test a single variation of the target product or numerous variants? - Can we assure that the control and experimental groups are properly randomized and that both samples are unbiased and true representative of the true user population?
- Can we verify the consistency of the treatment vs. control effects throughout the test?
The fundamental A/B testing procedure
Where to Test
You must first have an existing website or app. With your website or app, you must choose an area to investigate and, eventually, aim to enhance. Choosing which area to focus on can be influenced by a variety of factors, including:
- Intuition: do you feel something may be improved and want statistics to back it up? Is there a section you’ve always disliked and want to try something different?
- Usability testing: has usability testing shown a problematic location or interaction? Have you tried out a novel solution and now wish to test it on a larger scale?
- Data: Do your analytics show that a specific page or screen is causing problems for your users? Are your users all leaving the same page?
What to Test
One of the most important features of A/B testing is that you only alter one variable at a time. At first sight, this appears to be a straightforward assignment, but it is quite easy to go too far and include too many variables.
For example, if you wanted to test a button, you might change the copy of a button as follows:
Alternatively, you may change the color:
However, combining both of these and testing a button with different content and a different color would significantly lower the value of the test.
You wouldn’t be able to tell why these two buttons performed differently if you tested them against each other: how much of the difference in performance was due to the text change and how much was due to the color change.
“It’s critical to restrict changes to one variable when conducting a useful A/B test.”
What you track will be determined by what you test. Before you begin A/B testing, make sure you know what you’re attempting to improve. In the instance of the button, you’d most likely count the number of individuals that clicked on it.
How to Run a Test
A/B testing is possible using a variety of apps. Among the most popular options are:
- A/B Tasty
- Visual Website Optimizer
- Google Analytics
- Optimizely
All of them (and more) provide the fundamental A/B testing method but differ in the extra capabilities they provide. Which one you select may be determined by the number of development talents you have, the amount of flexibility you desire, or simply by cost.
How extensive is the test?
So you’ve decided on the location of your test, the factors you’ll try to optimize, and the technical details. The final question to address before proceeding: how many users will you test with?
Google Analytics does not enable you to choose who sees the original version vs who sees the alternative or even how long the test will last. It is a helpful feature for beginners because it streamlines the whole procedure.
“A/B testing cannot tell you if you’re addressing the right problem.”
If you operate in a risk-averse firm, you might want to show the alternative to just 5–10% of your users, whilst others might divide the difference 50/50. The decision is ultimately determined by your goals and the amount and type of visitors your website or app receives.
A critical question to ask yourself when deciding how to split the test and how long it should run is: how big does the test need to be so that I can be certain the findings are accurate?
Your goal is to design a test with a large enough sample size that you can assert with greater than 90% certainty, “Their change produced that outcome.” How you split your test is, therefore, one aspect, but the length of time you need to conduct your test may be determined by the quantity of traffic your website or app receives.
Evaluate and make a decision.
Many people anticipate you to see results like these after all of the work you’ve put in to get people on board and set up the test:
But, more often than not, you’ll get something like this:
Even if the data reveals that you haven’t made any progress, you’re now in a better position than before since you can firmly assert what works and what doesn’t. If your assessment was successful, the following steps are entirely up to you. Ultimately it is up to you what you do with your newfound knowledge!
Limitations of A/B testing
Despite its rising popularity, A/B testing is not a panacea that can save every business but rather another weapon in your armory.
- A/B testing is an excellent technique for determining what works and what doesn’t. What it does not tell you is why. You’ll need to do qualitative user research for this. It is an important concept to grasp: data does not equal comprehension.
- While you could, in principle, compare a complete page design to an alternative and obtain statistics on its performance, you wouldn’t be able to determine what about that design was causing any difference in performance. Was it because of the design, the copy, or the links? The findings of such an experiment would be worthless unless combined with user research.
- A/B testing, because of its progressive nature, maybe a useful tool for continually improving your website or app. A/B testing cannot tell you if you’re addressing the right issue. You may be focusing your tests on the homepage and getting results, but another portion of the site may be the underlying issue.
Conclusion
“The purpose of A/B testing is to follow the key statistic during the test period and determine whether or not there is a change in the product’s performance and what sort of difference there is.”
When utilized correctly and for the proper reasons, A/B testing can be a fantastic tool. It can help your organization make modest changes and expand its success.
However, it is critical to recognize that A/B testing is only one tool in a much larger toolbox for any designer. So, while A/B testing may be quite beneficial, it should not be used at the expense of other aspects of design.
Thanks for reading!!! Do let me know if this was insightful or even slightly helpful, any feedback is welcome. Want to connect with me, find me here: LinkedIn | GitHub | Twitter | Medium
