How to sleep better at night

A week or so before a usability test, I often have a least one bad night’s sleep where I wake up worrying that I’ve forgotten something. Do the participants have a map to the venue? Have we devised the correct tasks to address the test objectives? Will the tech support guy really be there when I phone up to tell him that the prototype won’t load?

I remember one usability test I ran many years ago where the client and three members of the design team had arrived from New York to view the test. We were using a rented lab in a seedy part of London, above a kebab shop. (It’s a long story). We started the first day at noon, to give our visitors time to beat their jet lag.

Everything was set, but the first participant was a no-show. “This kind of thing happens,” I said, “but don’t worry, we’ve scheduled some extra testing slots to cover for this”. It got to 1pm and our next participant failed to arrive. Awkwardness turned to embarrassment. Just as I was about to call the recruiter, she called me. “Your participants can’t find your building,” she told me. “You’ve given me the address of a kebab shop”.

I’d forgotten to tell the recruiter that people needed to use a side door to get into the lab. We’d arrived before the kebab shop had opened. But at lunchtime, the bright lights, the smells and the activity from the kebab shop effectively made the side door disappear from view. People literally couldn’t see it.

As a direct result of this experience (and a few others), I developed the usability test plan toolkit. This is a detailed document that describes everything from how you get to the test venue to the exact words the test moderator will use. This document is the result of all of the mistakes I’ve made in over 20 years of running usability tests (and believe me, I’ve made lots of mistakes).

I send a copy to everyone who has anything to do with the test. It’s my comfort blanket in the days building up to a test and it helps me sleep better at night.

But sadly, there’s a problem with my comfort blanket.

The lean usability test plan

Some people in start-ups who have used my toolkit have mentioned that it’s a bit documentation-heavy for their needs. They appreciate that usability tests need planning, but because they are doing virtually everything single-handedly, they don’t need all of the detail I use in my test plans (where there are often multiple stakeholders).

As more companies dip their toe in the water of user experience, this UX team of one is becoming increasingly common. The typical person in a UX team of one is responsible for all stages of user centred design on a project.

At first, I dismissed these comments. “Harrumph,” I harrumphed. “These people are lightweights who shouldn’t set foot in a usability lab. If you can’t spend the time writing a test plan, you shouldn’t run a usability test”.

But over the weeks, the issue gnawed at me. Usability testing isn’t just for experimental psychologists. What can I do to make it easier for people on agile teams? Does a test plan need to be so detailed for every situation? If not, what can be removed? What is the essence of a test plan? And how short is short?

“How about a single page?” I wondered.

The Usability Test Plan Dashboard

So, here’s the result of my deliberations: my attempt at reducing a multi-page test plan down to a single page:

Click the image for a larger view.

Download the Usability Test Plan Dashboard as a pdf

I’ve noticed in the past that single page dashboards like this can be extremely powerful. Managers are quite happy to look over a single page, while they may never open a longer document. Developers post it next to the kanban board and consider attending the test. People in the design team can easily take action on what’s written down.

Let’s go through the various sections of the usability test plan dashboard. These sections are partly based on the the usability test plan toolkit with modifications suggested by several usability test practitioners who reviewed early drafts.

Product under test

In this section, you provide a high level description of the product under test and describe its key business and user experience goals. Examples for a web site might include, “Reduce support calls” or “Sell products online”. The purpose of this section is to demonstrate that you understand the business goals of the product and that the usability test is linked to these. If there’s anything unusual about the context of use (such as a location-aware mobile app), you should mention that too.

Business case

Here you’ll briefly describe why you need to do this test. It doesn’t require a complete cost-benefit analysis, but you should briefly review the expected benefits (and perhaps the potential costs of not testing).

Test objectives

Your objectives must be specific: “We want to see if the product is easy to use” is too general. To formulate test objectives, try thinking about:

• Any aspects of the site/application that are of concern.
• Tasks that you think might be difficult.
• Groups of users you are worried about.
• Feedback from users: emails, phone calls, requests for help.
• Concerns of your client/management.
• Problems raised by designers/developers.
• Untested beliefs about users and usage held by the design team.

Participants

Describe the key characteristics that you will use to screen participants. Examples might include, “Spend between £50-£200 per month online”, “Regularly downloads and installs software from the Internet”, “Owns a DSLR camera”. You don’t need to be very specific here as you will do that in the screener itself. The purpose of this section is so that everyone agrees on the kind of participants you will be testing.

Test tasks

Test tasks are the beating heart of a usability test. People often get hung up on testing a certain number of participants, but did you know that you’ll find more usability problems in your test by including a variety of tasks than you will by including more participants? Your tasks should clearly relate to the test objectives you’ve set.

Responsibilities

This is where you name the individuals who are responsible for making the test run smoothly. Who will recruit participants? Who will moderate the test? Who will provide technical support if the prototype fails to load?

Location and dates

In this section, you’ll provide the location of the test and let people know the date of the test. You should also describe when, where and how the results will be disseminated to the design team.

Procedure

This section shows the play-by-play activities in the test itself. This purpose of this section is to give people an appreciation of what will happen during a participant session.

What the dashboard is not

I’m sure I don’t need to say this to such an intelligent readership, but this is a test plan dashboard — it’s not a test plan. It bears the same relationship to a test plan as the Business Model Canvas bears to a business plan. It covers the highlights but misses some of the important detail.

For example, even if you’re a one-person business developing a smartphone app and you’re making all of the decisions about your usability test — even if you’re testing with friends and family — there is still some detail missing. You’ll still need:

• A recruitment screener.
• A statement of informed consent / video consent form.
• A discussion guide (at least in outline form).
• A post-test questionnaire / word choice form.
• Screen shots for annotation or a datalogging system.

In other words, it’s a dashboard: it’s not a complete replacement for a usability test plan. (This is where I include a link to the usability test plan toolkit and encourage you to buy it).

Try it out

You can download a template of the usability test plan dashboard. It’s in PDF format with editable fields, so you can type straight into it. Alternatively, just drag it onto a slide in PowerPoint or Keynote and add your own text on top. (Because it’s in PDF format, you can resize it on your slide without losing quality). Of course, you can also re-draw it if you prefer. It’s issued under a Creative Commons Attribution Share Alike 3.0 Unported License.

Here's an article on the GDS User Research Blog that shows how I used the 1-page test plan with an agile team in HMRC.

Interested in this topic?

Find out more on our 3-day, user experience immersion seminar.