One of the early decisions you need to make when planning a usability test is to decide where to run it: will the participant take part in the test from their own location (such as their home or place of work) or will the participant need to travel (either to your office or to a rented facility)?
You then need to decide where you’ll be: will you be travelling too, or will you run the test from your base?
This leads to four possible location scenarios, as shown in the table.
|Participant at base||Participant away|
|You at base||Remote usability test||Corporate lab-based test|
|You away||Contextual usability test||Rented facility|
This provides a useful framework for deciding which kind of usability to run.
So what are the pros and cons of each kind of usability test?
The contextual usability test
With a contextual usability test, you go to the participant’s location, such as their office or home, and run the usability test at the participant’s desk or kitchen table.
Testing in the participant’s location has a number of benefits:
- Participants may be more willing to take part because they won’t need to travel (think stay-at-home parents or disabled users).
- You may also find that participants are less anxious about the testing situation, so the behaviour you observe is often more realistic.
- If they are at their own base, participants will also be more familiar with their own browser, operating system and computer compared to a test machine that you ask them to use. This familiarity is especially important with disabled participants who may need access to specialised assistive technology which is nearly always heavily customised.
- Another benefit of these kinds of test is that you get a first-hand view of the participant’s context — the way they actually live and work, and the systems they use day-to-day — and this nearly always challenges assumptions and gives you useful insights you can apply to later designs.
But testing in the participant’s environment also has a number of disadvantages:
- Although the participant won’t have to travel, you will. This can add hours to your testing time and those hours rapidly turn into days if you’re testing several participants.
- You’ll also find that distractions and interruptions — like telephone calls in the office or demanding children at home — are common during the test.
- You also need to factor in the disruption you cause when running a test: if you’re running the test in an open plan office, your test may disturb other people nearby who are trying to work.
- Related to this is the difficulty of accommodating any observers: it is very difficult to have more than one member of the design team attend and their role has to be carefully defined (especially if they are just there to observe rather than actually help run the test).
- Recording test sessions is also a challenge: if you set up a videocamera in an office without permission you get about 5 minutes before a man in a security uniform appears at your elbow. And if the IT department have locked down the participant’s computer to prevent software installation, you can say goodbye to software-based recording (such as Morae or Silverback).
The remote usability test
With a remote usability test, the participant again stays in their home or office but this time you stay at your base as well. This kind of test comes in two flavours: human-moderated and computer-moderated remote usability testing.
The difference between human-moderated and computer-moderated tests is that human-moderated tests need an administrator to guide the participant through the test (usually by telephone or via a service like Skype). This means you see what the participant is doing as they use the system (by screensharing) and you can hold a conversation with them. With a computer-moderated test, the computer steps the participant through the various tasks, not dissimilar to an online survey, and you’re not able to observe participants or speak with them.
Both these types of test have a number of advantages.
- People can take part from anywhere in the world which means you can have a diverse participant sample.
- Because the time commitment from participants is less onerous (they don’t need to travel), participants don’t need as much notice to take part so you can often organise these tests at short notice (it’s not unusual to complete data collection for a computer-moderated test in 24hrs).
- The tests themselves are also substantially cheaper because more participants can be tested in the same amount of time (mainly because you spend less time on the pleasantries necessary when you meet someone face-to-face).
- As an administrator, you can be more flexible in your time slots: you can run tests in your pyjamas or (with computer-moderated remote tests) even while you’re asleep.
- Computer-moderated remote tests also offer the possibility of previously unheard-of sample sizes for a usability test: it’s easy to get hundreds of participants to take part.
- With human-moderated remote tests, it’s also simpler to get cover for no-shows: “floaters” (people who agree to be available throughout the day in case a scheduled participant doesn’t attend) just need to agree to be on the end of a telephone over the duration of your human-moderated remote test.
- Another advantage for people new to usability testing is that human-moderated remote tests are easier to moderate: participants are less likely to sit in silence on the telephone compared to when they are next to you in a lab. This means you don't need to work as hard to get participants to think aloud.
- Some people also claim that participants are more honest in their opinions when tested remotely.
But there are some disadvantages.
- The main disadvantage is that remote testing is only feasible with software products.
- You’ll also find that participants experience distractions and interruptions with both kinds of remote test. With computer-moderated tests in particular, you have no idea what participants are doing during the test as you’re not there: they might be checking Facebook, email, even dancing to Psy’s latest video on YouTube.
- A particular problem with human-moderated tests is that you need to set up screen sharing with the participant. Although several programs exist that support screen sharing, these always need either a software install or a Java applet install. Participants are usually happy to do the installation but browsers like Safari and Chrome are becoming increasingly hostile to Java applets and participants may require considerable hand holding to complete the process.
The corporate lab-based test
With a corporate lab-based test, the participant comes to your lab to take part in the test.
- The main advantage with this kind of test is that you have complete control over the environment. This makes it easier to plan and set up the test (especially if you’re testing something physical like a printer or a washing machine) and it also makes it easier for you to troubleshoot any problems that arise (for example, if you need to do some last minute printing of revised test scenarios).
- A usability lab will also have room for observers to watch the test in progress (either through a one-way mirror or over screen sharing software projected on the wall of a nearby room): the impact that this can have on changing people’s view of users can’t be overstated.
- A corporate usability lab also makes a strong statement about the organisation’s commitment to usability: in my experience, a usability lab quickly becomes a scheduled stop on any tour made by a senior manager.
Ten years ago, most usability tests were like this but these tests don’t have the same dominance today. One reason is that many people who want to run usability tests don’t have the space or the money to maintain a lab. (But remember that a small conference room can often double-up as a usability lab: in fact, I once ran a usability test in a cupboard).
Nevertheless, this type of test also comes with its own set of problems:
- It’s difficult not to bias the participant’s behaviour when they have arrived at your head office and are using your product on your territory. This may cause participants to be less critical of what you show them, and blame themselves for any poor performance.
- You also need to allocate time in the schedule for participant lateness, time to welcome the participant, time to make the participant a coffee, and time to help them relax. This can create 15-30 minutes of dead time per participant.
The rented facility test
A test in a rented facility runs very much like a test in a corporate lab, but it comes with some additional advantages:
- You’ll have a least one worker bee whose job is to help you with the test, which reduces the amount of time you need to spend crawling under desks and attaching wires.
- The observation area is typically a bit more lavish with food and refreshments as standard.
- Because the facility will not have your company’s name stuck to the side of the building, participants may be willing to be more critical of any system that you show them.
- Another benefit is that most research facilities also offer a participant recruiting service. Testing in your own lab requires you to find a recruiter or set up your own participant pool.
The main disadvantage is of course money. Price varies depending on the facility itself, but costs in the range £800 - £1200 per day are common. Another disadvantage (if you’ve not used the facility before) is that you’ll need to spend time familiarising yourself with the set up.
So which do I choose?
Like all usability tests, you need to decide on your objectives.
- If the aim is to expose the design team to real user behaviour, then not much will beat a test in a corporate lab or a rented facility. Although observers can also watch a remote, human-moderated test in progress there’s something about having a live user in an adjacent room that focuses people’s minds.
- If the aim is to find usability issues so they can be fixed, a remote, human-moderated test offers great value and is quick to set up.
- If the aim is to measure benchmark values of performance so that you can measure and track progress over time or against competitors, you’ll find the huge sample offered by a remote, computer-moderated test is the most effective way to achieve this goal.
About the author
Dr. David Travis (@userfocus on Twitter) is a User Experience Strategist. He has worked in the fields of human factors, usability and user experience since 1989 and has published two books on usability. David helps both large firms and start ups connect with their customers and bring business ideas to market. If you like his articles, why not join the thousands of other people taking his free online user experience course?
Foundation Certificate in UX
Gain hands-on practice in all the key areas of UX while you prepare for the BCS Foundation Certificate in User Experience. More details
Every month, we share an in-depth article on user experience with over 10,000 newsletter readers. Want in? Sign up now and download a free guide to usability test moderation.
User Experience Articles
Our most recent articles
Our most commented articles
Search for articles by keyword
- 7 articles tagged accessibility
- 4 articles tagged axure
- 5 articles tagged benefits
- 16 articles tagged careers
- 8 articles tagged case study
- 1 article tagged css
- 8 articles tagged discount usability
- 2 articles tagged ecommerce
- 17 articles tagged ethnography
- 14 articles tagged expert review
- 2 articles tagged fitts law
- 4 articles tagged focus groups
- 1 article tagged forms
- 6 articles tagged guidelines
- 11 articles tagged heuristic evaluation
- 7 articles tagged ia
- 14 articles tagged iso 9241
- 11 articles tagged iterative design
- 2 articles tagged legal
- 11 articles tagged metrics
- 3 articles tagged mobile
- 8 articles tagged moderating
- 3 articles tagged morae
- 2 articles tagged navigation
- 9 articles tagged personas
- 15 articles tagged prototyping
- 7 articles tagged questionnaires
- 1 article tagged quotations
- 4 articles tagged roi
- 17 articles tagged selling usability
- 12 articles tagged standards
- 47 articles tagged strategy
- 2 articles tagged style guide
- 4 articles tagged survey design
- 5 articles tagged task scenarios
- 2 articles tagged templates
- 21 articles tagged tools
- 56 articles tagged usability testing
- 3 articles tagged user manual