When a good participant is hard to find

Assume you've been asked to run a usability test. How would you solve these three participant recruitment problems that I've come across in the past year?

  • The developers of a mobile application aimed at barristers want to run a usability test. How can they recruit busy, high-powered people to test their application?
  • The designers of a new flat-bed seat for first class airline travellers want to test out the seat controls with its target audience. How can they recruit and incentivise very wealthy people to take part in the evaluation?
  • A technology company develops software for the security service that helps find and track down terrorists. How can they recruit spies to test out their software?

You can probably come up with ideas for recruiting some of these hard-to-find participants. For example, the developers of the mobile application aimed at barristers could simply pay barristers' fees. And maybe we could invite the first-class passengers to a corporate entertainment event and take each one aside for 30 minutes for feedback on the seat design. But you'll need deep pockets to carry out those ideas. For example, it could mean paying barristers an incentive of '476.

Use surrogate participants

Here's another solution. What if we used surrogate participants: people that will never actually use the product, but who share important and relevant characteristics with the target audience?

A usability test with people who aren't real users of your product may seem absurd: usability test participants should be real end users. This is because surrogates may experience problems that won't affect real users (false positives). And genuine users may encounter difficulties that, for one reason or another, don't bother surrogates (false negatives).

But what if the choice is between using surrogate participants or no usability test? Are some surrogates better than others?

Distinguish super subs from blooper subs

When deciding on surrogates, the first question to ask is, "What domain knowledge do people need to use this product?" For example, with our airline seat, if the aim is to test out the usability of the controls we may not need to test with wealthy individuals at all. Although standard class passengers may not have had the experience of travelling first class on an Airbus, they have pretty much the same cognitive skills and physical abilities as first class passengers. Using surrogates to test the usability of every day products like this is relatively straightforward.

In contrast, if we test our mobile application for barristers with people who know nothing about the law, we'll get lots of false positive usability issues around legal terminology. This is because barristers have domain knowledge about the law that isn't found in the general population. Using surrogates for this kind of usability test is more problematic.

So how can we ensure our surrogate participants are as close as possible to real users? You want people who share similar traits with your personas. Here are some specific ideas.

Don't use the person at the next desk

The easiest surrogate participant to find is the person at the next desk. But using internal employees in your usability test is very risky. These people will know the product and may have even designed it. They may appreciate the design constraints and trade-offs you made and so have lower expectations of the product. In my experience, that makes internal employees almost always unsuitable as surrogates for genuine participants.

New recruits

In contrast, new employees in your organisation will be relatively unaware of the product's history. These people may know about the work domain but may have little, if any, experience with your own solutions. This makes them a potentially good choice as a surrogate and helps you gain a ‘fresh’ perspective.

The main risk you need to manage: new recruits may be unwilling to be too critical of the product in case they are perceived as criticising their employer. You can manage this risk by briefing them properly beforehand so they understand you want a no-holds-barred approach. They may also be more-than-usually anxious that they are being tested. So tell them to treat it as an opportunity to showcase their analytical skills.

Recently retired professionals

If you can't get to your customers because they are too busy, for example if they are barristers, surgeons or company directors, then look for people who used to do the job. Retired people are often looking for opportunities to keep active and frequently miss work.

The main risk you need to manage: you may find some false positive usability issues related to the known problems of aging, such as issues with visual acuity — although in practice we find that most early-retired people are as sharp as their younger colleagues. And if your retired surrogates have been out of the job market for more than a few years, their assumptions about the work context may be wrong.

People who work alongside genuine users

Another good surrogate for busy professional people is their co-workers. For example, could you use a pediatric nurse in place of a consultant to test out some of your system's functionality?

The main risk you need to manage: your surrogates will not have the same level of domain knowledge.

Students and their teachers

Like co-workers, students and teachers in the field will be familiar with the common concepts and the terminology used in the profession. For example, a trainee barrister may serve as a good surrogate for your fully qualified barrister and a medical student may serve as a good substitute for a doctor. Consider setting up a relationship with a nearby university to tap into their academic resources as this will give you access to teachers as well as students.

The main risk you need to manage: your surrogates may have the domain knowledge but they will lack the experience to know about the acceptable short-cuts and compromises that take place in practice.

Training course delegates

Does your company train people in how to use the products or software you design? If so, find out if you can piggy-back on these courses and run a brief usability test. Explain to delegates that they are getting the opportunity to design the next version of the product. This is a great way to get access to even the most difficult to find participants after all, I'm sure that even spies like James Bond went on a training course once.

The main risk you need to manage: people at training events are novices, by definition. They may have the right level of domain knowledge but they may not have genuine users' previous experience with your application. Also, because you won't have a lot of time, you'll need to plan your sessions very carefully and you'll also need to provide a tempting incentive to delegates who’d probably rather be chatting with colleagues over coffee and biscuits.

General advice on using surrogates

Before you start using surrogates, consider these three points carefully.

First, using surrogates only makes sense for certain kinds of user research. Surrogates are a bad choice for field visits because the context of use is such a critical component in that research. So when using surrogates, you need to choose your methodology carefully. Usability testing is an example of where it can work.

Second, don't use this as an excuse to avoid asking real users to evaluate your product. For most products, it's relatively easy to find and recruit genuine users, so there's no need to use surrogates. But even if your users are hard to get to, you should still make an effort to build a database of participants that you can call upon later — genuine users are the gold standard. Recruiting them may not be as hard as you think: they just need an opportunity to volunteer. Here are some ideas:

  • Include a sign up form on your web site calling for participants.
  • Find discussion boards and blogs read by your users and place an invitation.
  • Write an article about your product for a trade magazine and include a request for people to test it.
  • Ask for help from the societies and professional organisations to which your participants belong.
  • Make your study portable and run your test in a side booth at a convention or trade show.

If you get just one participant a month signing up via these routes, this still gives you something to build upon because you can then ask these participants to refer other people to you.

Third, in any usability test where you use surrogates, try to include at least one person who is the genuine article. Then compare his or her results with the surrogate users. If the findings seem very different, start weighing the risks of design changes.

Acknowledgements

Thanks to David Hamill and Catherine Kearns for improving this article.

About the author

David Travis

Dr. David Travis (@userfocus on Twitter) is a User Experience Strategist. He has worked in the fields of human factors, usability and user experience since 1989 and has published two books on usability. David helps both large firms and start ups connect with their customers and bring business ideas to market. If you like his articles, why not join the thousands of other people taking his free online user experience course?



Foundation Certificate in UX

Gain hands-on practice in all the key areas of UX while you prepare for the BCS Foundation Certificate in User Experience. More details

Download the best of Userfocus. For free.

100s of pages of practical advice on user experience, in handy portable form. 'Bright Ideas' eBooks.

If you liked this, try…

These 8 tips for screening participants will show you how to recruit articulate, representative users for your research, quickly filter out the people you don't want and help you avoid the dreaded "no show". Writing the perfect participant screener.

Related articles & resources

This article is tagged discount usability, usability testing.


Our services

Let us help you create great customer experiences.

Training courses

Join our community of UX professionals who get their user experience training from Userfocus. See our curriculum.

David Travis Dr. David Travis (@userfocus) has been carrying out ethnographic field research and running product usability tests since 1989. He has published three books on user experience including Think Like a UX Researcher.

Get help with…

If you liked this, try…

Get our newsletter (And a free guide to usability test moderation)
No thanks