UX Certification
Get hands-on practice in all the key areas of UX and prepare for the BCS Foundation Certificate.
Researchers sometimes ask participants which of two alternative designs they prefer. The data from these studies comprise opinions that have little predictive value. In contrast, multivariate A/B testing involves target users doing real tasks. The data from these studies comprise behavioural observations that predict real-world behaviour.
Photo by rawpixel on Unsplash
Designers learn early on that they should never produce one version of a design. Instead, they produce a small handful of designs, say 3-5, that allows them to explore alternatives. Either the client decides which one they prefer or they decide to put it to the vote by asking users.
Researchers who should know better sometimes refer to this as A/B testing. They have two designs, A and B, and then attempt to find out which one participants prefer.
This is not A/B testing. This is preference testing. The difference is not a semantic one. In our book, Think Like a UX Researcher, Philip Hodgson and I distinguish between three kinds of UX research evidence:
Preference testing provides weak evidence — evidence so weak it is worthless. In contrast, A/B testing provides strong research evidence.
A/B testing, as opposed to preference testing, is a form of multivariate testing. A/B testing takes place on a live web site. Half the visitors see one version of a page (Design A) and the other half see a slightly different version of the page (Design B). Development teams use A/B tests to test calls to action, product pricing and images on landing pages. The "winner" of an A/B test is the design that best drives the behaviour you want (such as purchases, donations or sign-ups to a newsletter).
One of my favourite examples of A/B testing comes from Google. When Google launched ads on Gmail, the development team wanted to optimise the ad hyperlink colour. They tested forty different shades of blue. They discovered that a slightly purpler shade of blue resulted in more clicks than a slightly greener shade of blue. The upshot from this testing was that by choosing the purpler shade of blue, Google made an extra $200m a year in ad revenue.
Imagine we asked participants which of those 40 shades of blue they preferred. Would people prefer the slightly purpler shade of blue that results in more clicks? Of course not. Expressed like this, it sounds ridiculous. So why does preference testing persist?
It persists because asking people which design they prefer appears to have face validity. Indeed, this kind of research method is popular in market research where marketers aim to discover which imagery, logo or brand identify people prefer.
But UX research is not market research. There are at least 4 reasons why preference testing has only a minor role, if any role at all, to play in UX research.
When researchers ask about preference, they generally show a participant two alternatives. They then ask them to pick the one they prefer. For example, "Do you prefer Design A or Design B?". Or "Would you prefer some feature (e.g. the navigation bar) presented on the left or right of a display?"
But that bears no resemblance to the way people make that judgment in reality. In the real world, people don't look at two web pages and make a decision on which one they prefer. Instead, people use a product to achieve a goal. Asking people to choose a design that they haven't used will lead to judgements weighted towards less important factors. Factors like a product's visual design, or whether it looks familiar, or an idiosyncratic preference (such as hating the colour orange). Instead, UX researchers should focus on the factors that lead to long-term usage and loyalty (such as "Can I do what I want to with this product?")
It's true that people know what they prefer. You can tell me if you prefer Coke or Pepsi, Apple or Microsoft, or your friend Amy or your friend Janice. But what you don't know is why you feel that way. Preference testing asks people to introspect on the reasons for their choices so that the development team can choose a design direction. Questions like, "Why do you prefer design A over Design B?" or "Why do you prefer the navigation on the left of the screen?" are common. It turns out people aren't good at answering this kind of question. People don't know why, or they don't care enough to answer, or they may not want to tell you. When asked for an opinion, most people will form one on the spot. Such opinions aren't carefully considered or deeply held. It's not that UX researchers don't care what people like: it's just risky making important design decisions based on fickle opinions.
Preference testing asks people to imagine a future where these designs exist and then predict which of the two they will use. But research shows that people are poor at predicting a wide range of future behaviours. This includes the time it takes to complete tasks, the likelihood that they will have long and happy relationships, how much they will save, how they will perform in exams, how much they will give to charity, and how likely they are to adopt healthy behaviours. The best predictor of your users' future behaviour is their past behaviour, not their current opinions. Rather than ask people what they prefer, we should instead be discovering which they perform best with.
Your research question is the purpose of your research. It guides the design of your study. Whether Design A is better than Design B is an example of a valid research question.
But a research question is not a question you can ask participants in your study. As an example, your research question may be, "Does cycling affect employee productivity?" but you can't ask participants to answer this question and expect to get meaningful answers. Instead, you may ask a participant, "How many times a week do you cycle to work?" or "What stops you cycling more often?" and compare those with productivity measures.
In fact, the way you answer your research question may not entail asking participants any questions at all. You may simply ask participants to do tasks as you observe (usability testing). In "The Mom Test", Rob Fitzpatrick captures this well when he writes:
"Trying to learn from customer conversations is like excavating a delicate archaeological site. The truth is down there somewhere, but it's fragile. While each blow with your shovel gets you closer to the truth, you're liable to smash it into a million little pieces if you use too blunt an instrument."
The difficulty with A/B Testing is that you can't use it early in design. What if we have two prototypes of an early design concept and want to compare them? This is presumably why people are drawn to preference testing instead.
But this is exactly why usability testing was invented. Now, rather than ask people which they prefer, we ask people to use a prototype and discover the usability issues that need to be fixed.
A usability issue is anything that:
Usability testing provides strong research data because it is based on behaviour — what people do rather than what they say they do. That's the next best thing we have to a time machine because we get to project the test participant forward to a future when your concept has become a real product and we get to see how it will perform.
-oOo-
Dr. David Travis (@userfocus) has been carrying out ethnographic field research and running product usability tests since 1989. He has published three books on user experience including Think Like a UX Researcher. If you like his articles, you might enjoy his free online user experience course.
Gain hands-on practice in all the key areas of UX while you prepare for the BCS Foundation Certificate in User Experience. More details
This article is tagged usability testing.
Our most recent videos
Our most recent articles
Let us help you create great customer experiences.
We run regular training courses in usability and UX.
Join our community of UX professionals who get their user experience training from Userfocus. See our curriculum.
copyright © Userfocus 2021.
Get hands-on practice in all the key areas of UX and prepare for the BCS Foundation Certificate.
We can tailor our user research and design courses to address the specific issues facing your development team.
Users don't always know what they want and their opinions can be unreliable — so we help you get behind your users' behaviour.