Discount Usability: Time to Push Back the Pendulum?

If you stepped into a usability test in the early days of the discipline, it would have looked very much like a formal experiment in a psychology lab. Subjects were asked to carry out tests and the data were analysed using statistical methods. All that was missing was the white lab coat. Unsurprisingly, these tests were expensive to carry out and complicated to run, leading to the development of a breakaway discipline: discount usability.

Discount usability

Discount usability introduced three key techniques that aimed to simplify methods of data collection:

• Thinking-aloud usability tests;
• Low-fidelity prototypes;
• Heuristic evaluation.

These techniques work best as part of an iterative design cycle where usability problems are found and fixed and then the next "throwaway" prototype is again quickly tested with a small number of participants. This approach truly revolutionised the field. It would now be difficult to find a usability practitioner who did not use most of these techniques during an assignment.

In fact, the pendulum has now swung so far that many people are dismissive of traditional, lab-based testing.

Limitations of discount usability

But discount methods fall short in one important area. They cannot answer the question, "How usable is this system?" For example, imagine you are in the business of developing a new mobile phone. Your marketing department have identified a key product attribute: the ease with which the user can enter and send a text message. Your new phone needs to beat the competition. Which method will you choose?

Discount methods will help you spot usability problems during the early design phase, but they will not be able to show how you stack up against competitor products. This type of usability evaluation — testing against usability metrics — requires a more traditional, lab-based approach. For example, when it comes to measuring the ease with which the user can enter and send a text message, we might want to put the new phone and its competitors in a head-to-head test. We could then collect robust measures such as the time it takes to complete the task, the number of errors made (such as typing or menu navigation errors) as well as some subjective ratings from participants.

Usability metrics

Usability metrics are precise, quality measures used to evaluate the system. Their purpose is to produce a system that is neither under- nor over-engineered. To understand why usability metrics are useful, remember the joke about the guys out camping disturbed by a bear. As one of them puts on his trainers, the other says: "What are you doing, you’ll never outrun a bear!" The first one says: "I don’t need to outrun the bear. I just need to outrun you". Usability metrics help you outrun the competition.

We recommend collecting usability metrics in the three Es: effectiveness, efficiency and emotion.

• Effectiveness
  The accuracy and completeness with which users achieve specified goals.
• Efficiency
  The accuracy and completeness of goals achieved in relation to resources.
• Satisfaction
  Freedom from discomfort, and positive attitudes towards the use of the system.

Effectiveness measures of usability

Effectiveness refers to the accuracy and completeness with which users can achieve their goals. Typical measures include:

• Number of power tasks performed;
• Percentage of relevant functions used;
• Percentage of tasks completed successfully on first attempt;
• Number of persistent errors;
• Number of errors per unit of time;
• Per cent of users able to successfully complete the task;
• Number of errors made performing specific tasks;
• Number of requests for assistance accomplishing task;
• Objective measure of quality of output;
• Objective measure of quantity of output;
• Per cent of users who can carry out key tasks without reading the manual.

Efficiency measures of usability

Efficiency refers to the amount of effort users need to put in to achieve their goals. Typical measures include:

• Time to execute a particular set of instructions;
• Time taken on first attempt;
• Time to perform a particular task;
• Time to perform a particular task after a specified period of time away from the product;
• Time to perform task compared to an expert;
• Time to learn to criterion;
• Time to achieve expert performance;
• Number of key presses taken to achieve task;
• Time spent on correcting errors;
• Number of icons remembered after task completion;
• Time to install a product;
• Per cent of time spent using the manual;
• Time spent relearning functions.

Emotional measures of usability

Emotion refers to how users feel about the system. Typical measures include:

• Ratio of positive to negative adjectives used to describe the product;
• Per cent of customers that rate the product as "more satisfying" than a previous product;
• Rate of voluntary use.
• Per cent of customers who feel "in control" of the product;
• Customer rating on a 7-point scale anchored with "makes me more/less productive";
• Per cent of customers who would recommend it to a friend after two hours’ use;
• Per cent of customers that rate the product as "easier to use" than a key competitor.

Push back the pendulum

If you are developing a new product or a new piece of software, be sure to continue to use discount methods to eradicate usability problems. But when you need to answer the question, "How usable is this system", push back the pendulum and collect usability metrics.