Measuring the usability of everyday products

A while back, our friends at Neo Insight developed a slide show titled "A Day in the Life of Usability". It's a simple but engaging self-running presentation that demonstrates just how pervasive usability is in our daily routine. From alarm clocks and heating controllers, through to voice mail and vending machines, the slides demonstrate that in modern cultures we're struggling with technology virtually every hour of every day.

This means that when you're not interacting with the computer on your desk, you're probably interacting with another computer. Buying a train ticket? Try the computer in this kiosk. Preparing a snack? Try the computer in this microwave oven. Watching TV? Try the computer in this set-top box. These things may not have a mouse attached, but in many other respects they are as much of a computer as the one you're using at the moment.

Explore this further. Add up all the time you've wasted trying to get a new gadget to achieve the simple goal for which you bought it and you may find that computers have helped you waste literally weeks of your life.

ISO 20282: "Ease of Operation of Every Day Products"

So I'm pleased to say that we now have an international standard on measuring the usability of these devices. Titled "Ease of Operation of Every Day Products", ISO 20282 has four parts:

• Part 1: Design requirements for context of use and user characteristics.
• Part 2: Test method for walk-up-and-use products.
• Part 3: Test method for consumer products.
• Part 4: Test method for the installation of consumer products.

ISO 20282 applies ISO 9241-11 to the user interfaces of everyday products. Part 1 describes a design process, and parts 2-4 describe a test method. Let's look at both of these in turn.

Part 1 standardises the design process

Part 1, "Design requirements for context of use and user characteristics", provides a set of sensible design guidance for anyone who is developing consumer technology. It outlines a five-step process that the design team should follow:

1. Identify the main goal of your product. For example, for a ticket machine at a railway station the main goal would be, "buy a ticket". For a digital camera, it will be "take a photo". You can think of these main goals as red routes.
2. Identify which user characteristics and which elements of the context of use could affect the ease of operation of your product. For a ticket machine, this list might include the age and physical abilities of the user, as well as the location of the machine and any ambient noise. If you're unsure which characteristics matter, the standard contains a checklist of important characteristics and a useful annex that describes typical performance levels.
3. Establish the impact of each of these characteristics on the ease of operation of your product. For example, if we expect wheelchair users to buy tickets from the machine, this will have an impact on how high we can place the controls.
4. Ensure that the product design takes account of these characteristics. Again, using a wheelchair user as an example, we can consult anthropometric tables to calculate a suitable arm reach for 95% of wheelchair users.
5. Review the final design to ensure it complies with the characteristics. This is simply a sanity check to make sure that you close the loop between design specification and design implementation and leads neatly into the test methods that comprise parts 2-4 of the standard.

The other parts standardise the test method

The remaining three parts of ISO 20282 (parts 2-4) propose test methods for measuring the usability of every day products. The three test methods are essentially the same and will be familiar to anyone who has observed a usability test. You define the key goal of the product (such as using a telephone to make a phone call or using a television set to watch a television programme) and then observe participants as they try to complete this goal. The different parts of the standard add flesh to this skeleton test method according to whether this is a walk up and use product (part 2), a consumer product (part 3) or an installation of a consumer product (part 4).

Unlike Part 1, these parts aren't international standards: in the jargon of ISO, part 2 is a "Technical Specification" ("TS" for short) and parts 3 and 4 are "Publicly Available Specifications" ("PAS" for short). This means the documents do not currently fulfil the requirements for a standard: instead, they reflect the consensus within an ISO committee. In the UK, BSI have issued each of these parts as a "Draft for Development", essentially a consultation document. The idea behind this is to give organisations three years to try out the test methods and report back to BSI on their experience. At the end of the three years, BSI will collect together any comments on the test methods and work out how to revise them to create proper standards. Alternatively, BSI may decide to reject the test methods or to extend the consultation period for another three years. (If you use the test methods and want to provide feedback, comments can be sent to iso_tc159@din.de).

From the perspective of usability, what makes each of these test methods interesting is that they propose ways for measuring usability that may be different to those you're familiar with.

To measure 'everyday product' usability, you need to collect real data

The first difference is with the data collected. Rather than use a "thinking aloud" usability protocol, where people are asked to carry out a task and the administrator identifies usability problems, these parts of ISO 20282 recommend that you collect real performance measures.

People familiar with the definition of usability given elsewhere by ISO might therefore expect measures of effectiveness, efficiency and satisfaction. But for every day products, the test methods in ISO 20282 define the critical performance measure as "effectiveness" — can users complete the main goal of the product? (The test method defines this more completely as, "the percentage of users who achieve the main goal(s) of use of a product accurately and completely".)

Measures of efficiency and satisfaction are optional, although the test method acknowledges efficiency may be important if you're likely to have a queue of people lining up to use the machine (such as with our ticket machine on a busy station concourse).

To measure 'everyday product' usability, you need to recruit large samples

The second difference from most usability tests is that the test methods in ISO 20282 emphasise large sample sizes: at least 50 participants. Given that many "thinking aloud" usability tests involve as few as 5 participants, 50 participants may seem overkill. The larger sample is partly to ensure you have truly representative users and partly to provide results with sufficient predictive accuracy. The larger the sample size, the greater confidence you can have in your results.

If you're anticipating this to mean weeks of data collection, remember that for most consumer products there is only one key goal: "the most frequent and/or important user goal that the product is intended to support" as it says in the standard. This means that each participant will be asked to carry out just one or two tasks with the product, so the participant session time should be much shorter than with "thinking aloud" testing. My estimate is that each participant could be briefed, tested and sent on his or her way in 20 minutes. Assuming you can locate your test in a location where you will get a good flow of participants — such as a railway station if it's our prototype ticket machine — you could complete the data collection comfortably in 3 days.

This is a sea change in usability

If you're interested in web usability, you may be tempted to dismiss ISO 20282 as irrelevant to you. But there's a bigger issue here.

The field of usability has matured a lot over the past 5 years. For many of our clients, usability testing used to be a novel approach to assessing usability. Now, usability testing is an established part of our clients' design processes. "Thinking aloud" techniques are now used at the correct time — early on in the lifecycle, to find usability problems with early prototypes — not at the end of design, when it's too late to fix problems. Increasingly, clients are demanding more objective measures of usability once the design is completed. ISO 20282 reflects this sea change in usability testing and will help ensure that we have robust and reliable test methods for all interactive designs, whether these are products, software or web sites.

Acknowledgements

Thanks to Nigel Bevan for commenting on an earlier draft of this article.

About the author

Dr. David Travis (@userfocus on Twitter) holds a BSc and a PhD in Psychology and he is a Chartered Psychologist. He has worked in the fields of human factors, usability and user experience since 1989 and has published two books on usability. David helps both large firms and start ups connect with their customers and bring business ideas to market. If you like his articles, you'll love his online user experience training course.