How to prioritise usability problems

Drinking from a fire hydrant

Running a usability test has been compared with taking a drink from a fire hydrant: you get swamped with data in the form of usability issues that need to be organised, prioritised and (hopefully) fixed. Although it's tempting to use your own judgement in determining severity, this causes a difficulty when a developer challenges your decision: "How did you make that issue critical? I think it's more of a medium problem".

Having a standard process for defining severity means that you can be consistent in the way you assign severity and means that you provide the transparency needed for people to check your work.

In fact, by asking just 3 questions, we can classify any usability problem.

Does the problem occur on a red route?

Red routes — frequent or critical tasks — are the most important tasks that the system needs to support, by definition. For example, if the "on-off" button on your newly designed gadget is hard to operate, all of your users will be affected. Because problems on red routes affect more users, they are more severe.

Is the problem difficult for users to overcome?

Some usability problems are show-stoppers: users just can't proceed. For example, if an important control is hidden in a dialogue box or behind a right click, the functionality may as well not exist for some users. Other usability problems are easy to workaround. Hard to solve problems are more severe because they have a bigger impact on completion rate.

Is the problem persistent?

Persistent problems — problems that keep cropping up — are more severe because they have a bigger impact on time on task and on customer satisfaction. An example of a persistent problem is a web site that doesn't have underlined hyperlinks. This means users can find the links only by "minesweeping" over the page. This problem is persistent, because even when users know the solution to the problem they still have to experience it. Note that "persistent" means that the problem occurs repeatedly, throughout the interface — users come across the problem on multiple screens or pages.

We can put these three questions in a process diagram and use it to define four severity levels.

Download this decision tree as a pdf

How should you interpret the severity levels?

Critical

This usability problem will make some customers unwilling or unable to complete a common task. Fix urgently.

Serious

This usability problem will significantly slow down some customers when completing a common task and may cause customers to find a workaround. Fix as soon as possible.

Medium

This usability problem will make some customers feel frustrated or irritated but will not affect task completion. Fix during the next "business as usual" update.

Low

This is a quality problem, for example a cosmetic issue or a spelling error. Note: Although this is a minor issue in isolation, too many "lows" will negatively affect credibility and may damage your brand.

Moving beyond intuition

If you currently prioritise usability problems using 'gut feel' or intuition, you run the risk of being exposed as a fraud by a developer or manager who asks you to justify your priority ranking. Instead, deliver more robust findings by using this decision tree.

Love it? Hate it? Join the discussion

How do you prioritise usability issues? Do you think it's important to use a reproducible technique like this?