Discuss 4 forgotten principles of usability testing

Great article David - and I too share your experience of 'home grown' usability tests being far from optimal - and sometimes misleading - particularly when they make the cardinal sin of asking opinion.

Whilst I agree partly with the sentiment in Steve Krug's 'rocket surgery made easy' - an hours worth of reading a book does not make an effective usability researcher (thank god or I would not value what I or you do for a living). Even by a non-specialist can get useful insight though if they follow your tips above (and get some training) and i am glad that indeed testing is now being seen as 'mainstream' - its always been a bizarrely ignored no brainer in my opinion.

As with any tool or method the value only comes if you know how to use it and use it properly - in the usability arena eyetracking is another misused and abused technique that in the wrong hands delivers worthless or misleading results - I see these very often regrettably...

Cheers

Jon

So what exactly do you mean by 'many usability tests are worthless'?

Do you mean:

A) they could have learned more than they did? or

B) acting on the findings from this test positively decreased the usability of this product? or

C) the things that they found out from this test were irrelevant, and therefore simply wasted the clients money?

I think that your message will be interpreted as B or C, whereas in fact I suspect that the true answer is A. Which is to say, I personally have *never* observed a usability test that fell into categories B or C. Whereas nearly every test I've ever observed (including my own) fell into category A.

I would say that the thing that distinguishes a really good test from an adequate test is that we learn quite a bit more from the good test. But that's different from saying that we learned nothing, or were actually flat wrong, in our findings from a less good test.

What I now believe is that you'll be quoted all over the internet by people who say: look, even this respected researcher thinks that this technique is a waste of time. Don't bother with it: just go with the 'designer is genius' approach and launch without any testing.

The issue is making usability a bigger deal than what it really is, and worrying too much to not to do it. (what a paradox).

The whole goal of usability testing is to see if what you build/building would deliver the user goals and uncover issues of the product sooner rather than later. Thus this article is great in reminding us that it's better to look for users who fit the behavior profile rather than perfecting the 'demographic' recruitment process. Sometimes even showing your work-in-progress to the members in another team can uncover a lot of issues up front.

Having said that, I'm not discounting professional usability testing... it's just that we shouldn't be too bogged down by the methodology talk and not do the testing.

Having just finished the first day of a series of Usability tests for a new client website, I wanted to echo your point "Focus on what people do, not what they say".

Despite all planning and effort to the contrary, it can be a really difficult task convincing a new client with no experience with this sort of testing that behaviour is more important than a test candidate's stated beliefs.

I suppose for the the client, the end result is bound to be a change to the website, so any comments that communicate what those changes could look like should be listened to right? Today I could see the client's mind drifting off as they were mentally weighing up the validity of the "I think it would look better in blue..." comments being blurted by the testee!

So the problem must be, how do we get testers to talk about what they want to do, rather than what they think something should look like?

This article challenged me to ask the question “What is good usability testing?”

I enjoyed reading about your thoughts on finding methods to make usability testing more informative and many of the points you make are both valid, as well as appealing, but following some of these guidelines could prevent vital information from being discovered.

First, it is my belief that research and the analysis are not synonymous. As a researcher, I depend heavily on intelligent interpretation of the data. The firm I work for, User Insight, is so adamant about this that we utilize a strategist on every project. The strategist’s sole mission is to watch all of the research objectively and report patterns appropriately. The analysis acts as a “check and balance” and empowers the researcher to explore deeper.

Second, usability cannot be examined in isolation, as it is only part of the user experience. It is crucial that moderators explore not simply IF users can do it, but all of the questions surrounding it, such as “Do they want to do it?” and “How much would you pay for this service?” As you noted, questions like these are not designed to inform the future price point of the product. However, these questions reveal the user’s perceived value for the product. Learning that a user can do something is only part of the story; they might not know what they accomplished or they might not have the desire to ever do it. Some studies reveal findings outside of usability, which will inherently make an impact on the usability of the product. I have seen this many times.

I do agree with your point that user testing needs to continually strive to create a natural user experience. As pointed out earlier, untimely questions can create a flawed experience for a participant during a task-based study. However, I think it is both appropriate and valuable to have a user examine a screen after the task has been completed. I have seen first-hand how valuable this can be. Encouraging a participant to take a deeper look at a screen after completion of a task can reveal issues with terminology, layout and content which could all go unnoticed if not instigated by the researcher.

Thanks again for writing and sharing this thought-provoking article!

Hi David

I am sympathetic with the analysis in your article. However, I sense the problem much bigger than the quality of the usability testing activity. For me it's the move from UX as User Research, to UX as a design and development function.

The impression I get is of straw man UCD. i.e. product development life-cycle is made to look like user research can be conducted; small low-fi to hi-fi releases of software. Although the actual "usability testing" is validation from the project team; validation by committee rather than through user involvement.

When research is actually completed it done using guerilla techniques, which emphasise any feedback is better than no feedback. On UX Exchange, Steve Krug's "Don't make me think" was regularly recommended as one of the better resources for usability testing and the whole premise of the book is anything is better than nothing!

In addition, the popularity of just in time (JIT) development processes popularised by 37Signals which advocates getting products out there as early as possible and catching the bugs in the wild. Eroding the validity of a slightly slower paced, more thoughtful approach to designed and development embodied in UCD.

In short, there is a toxic mix of rebranding UX and JIT development process, that leaves little room for people to learn about, plan, run and interoperate a successful usability testing session.

As always, a very readable and practical Userfocus article.

In response to Caroline's comment (number 2), I definitely have seen (b) - the product actually being made worse - as a result of user tests. This has primarily been due to two factors:

- unrealistic situations and tasks
- massive (unintentional) bias from the the moderator, influencing the outcome

In small-sample tests (i.e. most real world tests in most organisations) the experimental flaws are so severe that the results and consequent proposals for change are actively at odds with what would benefit users performing real tasks.

The answer to this is not to say "usability tests are useless", or "you must be an experimental psychologist to carry out user tests", but to educate and inform people - exactly as David is doing - about the largest pitfalls and how to avoid them. People are not *trying* to do bad research - they just need more help (as we all do).

Thank you for this article. Can you please elaborate on the notion of disregarding demographic data? Would you test adult users on the website intended for children? Or am I taking you too literally?

Also a semi-related question, can you help me understand the rationale for why discussions and comments are on a separate page than the original article? I'm not criticizing the design decision, I am simply hoping to better understand the pros an cons of this approach.

I really enjoyed this article. So many times have I seen design by committees create their own usability requirements based on their own bias opinions and whims. Thanks for this as it was almost a way of venting frustration.

Hi David,
A good article and very thought-provoking. Please keep writing articles like this!
Greg