Numbers for numbers' sake?

When I was about 10 I used to sit by the side of the road with a small notebook and pencil and write down the license plate numbers of passing cars. I can still remember some of them: SVY673 and XDN210 spring readily to mind, and MDT20 was a particular favourite. Usually as I was writing down one number, ten more would whizz by unnoticed, but I didn't care. It was a good way of passing time and it felt important. It was also perfectly pointless.

I didn't realise it at the time, but the same approach to data collection exists in the adult world, too. Soon after a company discovers the value of usability, it wants to start measuring it. Over time, UX teams gather a sizeable body of metrics and begin to feel important — but these measurements are rarely, if ever, used for anything related to the business and they seldom seem to surface in design meetings or drive development decisions.

Has the exercise of writing down usability numbers become perfectly pointless too?

Let's step back for a moment and look at why most organisations measure stuff.

C-level managers use numbers because they have predictive value. For example, I may notice that when the number of people entering my store increases (at Christmas, say), so does the number of sales I make. So if I can increase the number of shoppers at other times of the year — for example, by advertising — I would expect that to lead to an increase in sales and profit. "Footfall" becomes a coincident metric that I can use to predict profit.

Most companies have no shortage of metrics relating to the success and failure of their products. They have sales figures, service incident rates, customer loyalty indicators, product return rates, customer support call-centre volumes, ‘customer instruct' rates and so on.

These are useful metrics, but they all suffer from the same problem: they are so-called ‘lagging' indicators. You can obtain these metrics only after you have launched the product — and sometimes long after the launch date. This is fine if the figures are strong, but if your product is failing, this is too late to find out about it. All you can focus on then is costly damage limitation via retroactive fixes, additional call-centre agents, and product replacements or, in some cases, a product recall.

What businesses need is a ‘leading' indicator: a metric that can predict product success or failure before the product has been released. That's where usability measurements come in.

Step 1: Measure usability

The international usability standard, ISO 9241, contains a definition of usability that we can use to operationalise and measure usability. I've written about this elsewhere, but briefly we need to measure the effectiveness and efficiency of a system (both can be measured objectively) and include a measure of user satisfaction (a subjective measure).

You need to measure these three components of usability for each red route, and then the results can be combined across tasks to give an overall measure of effectiveness, efficiency and satisfaction. Finally, you can aggregate the three measures into a single metric for usability.

When arriving at your single usability metric, you may want to weight one of your usability measures more highly than the others. For example, for a museum kiosk, effectiveness might be the most important of the three measures. For an intranet, efficiency might be the most important measure. And for a game on an iPhone you might want to make satisfaction the most important measure. The point is that although you need to collect all three measures to get a fully rounded picture of usability, it's OK to prioritise one of the measures over the others in coming up with your single usability metric.

Step 2: Correlate your UX metric with business metrics

In this step, you need to work out the predictive value of your metric. There's a slow way of doing this and a fast way.

The slow way is to gradually build up a database of usability metrics for your products, wait for them to be released and then afterwards examine the business metrics associated with the product, like return rate and calls to customer support. Depending on your product, this might take months at best and could even take years.

The quicker way is to run some usability tests of your products that are already in the market place. I'd suggest picking three products, one with (say) a high volume of calls to customer support; one with a ‘typical' volume of calls; and one with a lower-than-average volume of calls. Run a usability test of each product with about 20 participants and calculate the single usability metric for each one. You now have three data points you can use to predict call volume from a usability metric.

Step 3: Start predicting product success and failures

As your body of data increases you will be able to correlate usability metrics with business metrics and start to make predictions about the likely success of a product. You will be able to step up in a business meeting and say things like this: "Our new product has a usability metric of 53%. If we launch it now we can expect to see a customer satisfaction rate of only 40% and a return rate of 25%. In addition, the customer support agents are likely to see a 20% increase in call volume. Are we sure we want to risk this?"

User experience is about more than numbers

Paul Brodeur has written, "Statistics are human beings with the tears wiped off". The same could be said for usability measurements. I'd be the first to confess that these numbers capture only one aspect of the user experience — but it's an aspect that we can use to provide real business value and to ensure that user experience has a voice at the business table. It's not only about numbers, but numbers are certainly part of what we do. And if we're going to collect numbers, we owe it to ourselves to do something useful with them, and not simply write down the license plates of passing cars.

Sign up for our newsletter to hear about future articles where we'll show you how to use usability metrics to diagnose problems with your product or interface.

Acknowledgements

Thanks to Beth Maddix, Lynne Tan and David Travis for their comments on this article.

About the author

Philip Hodgson

Dr. Philip Hodgson (@bpusability on Twitter) holds a B.Sc., M.A., and Ph.D. in Experimental Psychology. He has over twenty years of experience as a researcher, consultant, and trainer in usability, user experience, human factors and experimental psychology. His work has influenced product and system design in the consumer, telecoms, manufacturing, packaging, public safety, web and medical domains for the North American, European, and Asian markets.



Web Usability: An Introduction to User Experience

Jan 12-13, London: A 2-day immersion seminar that will show you how to boost sales and conversion rates, increase usage and improve customer satisfaction.More details

Download the best of Userfocus. For free.

100s of pages of practical advice on user experience, in handy portable form. 'Bright Ideas' eBooks.

UX newsletter

Every month, we share an in-depth article on user experience with over 10,000 newsletter readers. Want in? Sign up now and download a free guide to usability test moderation.

Related articles & resources

This article is tagged iso9241, strategy, metrics, usabilitytesting.

Search for articles by keyword


Our services

Let us help you create great customer experiences.

Upcoming courses

We run regular training courses in usability and user experience.

Get free email updates

Join the thousands of other people who get their monthly fix of user experience insights from Userfocus and get a free guide to usability test moderation.