Discuss Usability testing with hard-to-find participants

I've discovered that, often, the target participant we think we need doesn't really use the system. In a recent example, we evaluated a reporting tool for surgeons, but in reality the surgeons will never use it. Their time is too expensive to be used actually writing reports. The nurse or the lab technician will be the actual user. Ditto another case for banking software aimed at 'high rollers'. In reality it is their accountants or traders or personal assistants that play around with the tools. So it is important to double check that it really is the barrister or the surgeon or the airline pilot that you need and not someone lower in the chain -- not as a surrogate but as the actual user. This is important because, in the example I mentioned, if we had tested with the actual surgeons we may have paid a lot of money to get misleading data.

On a related note, we have conducted tests of medical systems with the hard to get high level experts (such as Endocrinologists and Cardiologists) as well as with their nurse assistants, and their lab technicians. Testing with these experts can sometimes present it's own challenges. Very often they tend to sit back and pontificate -- in fact they are often surprised to discover they have to do anything with the system, thinking that we just wanted to listen to their gems of knowledge. Often, when they make an error, they can be too immediately dismissive of the system. In other words, they tend to distance themselves from this low-level of feedback, and want to simply tell you where you are going wrong. It is still possible to collect useful data but if the development team are watching they can be too swayed by the experts' opinions and may give them priority over the behavioural errors of, say, the technician (who is the real user).

It's really useful to have an article on this - it's a real-world problem that seems to be largely ignored by people who write in this area: so thanks for the recognition.

I work in an environment where the people we want to recruit are practising accountants. So, like barristers, they are generally used to charging by the hour: and every hour they spend with you is an hour they're not charging out.

We've found a few techniques to be effective
- building up a database of potential participants by developing a "panel", with its own charter that hopes to address some of the reasons why accountants are reluctant to engage
- running "open days" where attendees receive a lot of benefit (access to expertise, information about the future, industry analysis . . . and usability sessions). Essentially, making it worth their while to give us some of their time.
- obtaining contacts from training visits: often people who go out to deliver training have a great perspective on people who would be willing to help, and we have a high success rate contacting those people after training has taken place. And in our case, these people often aren't novices: they can be expert in other pieces of software or closely associated domains.

I agree with the article about using surrogates being potentially risky. There are generic things that you can test with a much wider range of people, but for anything domain specific, I feel it's really important to get to the real users. They can often highlight not only issues with the performance of the task, but with the deifintion of the task itself - and this is something that surrogates will rarely be able to do.

Just wanted to say thanks for an excellent article.

Recently I came up against the 'hard to get' target participant problem: in my case it was people in very particular circumstances (e.g. terminally ill or about to experience foreclosure on their home).

We haven't had to actually source test participants yet but when the time comes, thanks to your comprehensive examination of the different options - including some I hadn't thought of - I have some ideas about what I might do.

Thanks again for taking the time to think this through and put those thoughts into an article.