Further comments on this article are now closed.
I proposed the word list to a product team who were suggesting a questionnaire for users after a session with a new product, and was absolutely delighted with the results. Even if - as in our case - the users choose not to pick many words, it acted both as a great tool to investigate why they chose the words they did and as a persuasive piece of evidence for developers and managers who were not present.
I can see this working as a way to structure a post-experience interview but it must be a pain to count up any large number of results for analysis. I wonder if I could make it into an online survey.. maybe Google Docs would do it...
We have been using the word choice for a long time now together with the three elements of usability testing of our product (effectiveness, efficiency and satisfaction). We have realized that it's very important because when we have strange and/or conflicting results from the usability measures the word choice often offers a very good explanation to why. It is not only the words that have been chosen that are interesting but also the words that haven't been chosen. Of course this is easier with a smaller list of words (we have about 18 words with equal amount of negatives as positives).
This is a very useful article with implications that go well beyond usability testing in the traditional sense.
I work with professional services firms to help them measure client satisfaction and loyalty. My starting point is nearly always a variant of the Net Promoter Score (NPS) recommended by Fred Reichheld. (Likelihood to Recommend and Likelihood to continue Using followed by open-ended probes around what it would take to get to a perfect score.)
I could easily adapt my approach to include the exercise described in this post.
By presenting a tailored list of words, which would include both negative and positive adjectives and brand attributes, I could quickly focus in on areas that the respondent felt were most relevant to his or her experience with the service.
The list could be presented via an online questionnaire, the link for which I'd forward during or shortly prior to the telephone interview. At the appropriate time, we'd go through the list (which would be randomized) and the respondent could quickly tick off the words of interest. On the next page, only the selected words would appear and s/he could choose the five for further discussion.
The survey site would capture the data - and I'd be able to follow along on paper as the respondent make the choices.
The technique could be adapted well to focus groups also.
Thanks again for describing the Microsoft approach and for preparing the Excel tool.
The Customer Satisfaction Questionnaire Blog
I want to address the common misconception that it is necessary to "balance" usability questionnaires -- to have an equal number of positively and negatively aligned items. For research investigating evidence of the influence of response styles (such as the agreement or acquiescence tendency) on this questionnaire, including all references, see:
Lewis, J. R. (2002). Psychometric evaluation of the PSSUQ using data from five years of usability studies. International Journal of Human-Computer Interaction, 14, 463-488. (drjim.0catch.com)
Even though my research was on the PSSUQ, the discussion/findings apply generally to questionnaire-based usability satisfaction measuresl.
From p. 467-469:
"It is a common practice in questionnaire development to vary the tone of items so, typically, one half of the items elicit agreement and the other half elicit disagreement. The purpose of this is to control potential measurement bias due to a respondentís response style. An alternative approach is to align the items consistently.
Our rationale in consistently aligning the items was to make it as easy as possible for participants to complete the questionnaire. Furthermore, the use of negatively worded items can produce a number of undesirable effects (Ibrahim, 2001), including 'problems with internal consistency, factor structures, and other statistics when negatively worded items are used either alone or together with directly worded stems' (Barnette, 2000, p. 363). The setting in which balancing the tone of the items is likely to be of greatest value is when participants do not have a high degree of motivation for providing reasonable and honest responses (e.g., in some clinical and educational settings).
'Thus, first and foremost, the survey or questionnaire designer must determine if using negatively worded items or other alternatives are needed in the context of the research or evaluation setting. Unless there is some pervasive and unambiguous reason for not doing so, it is probably best that all items be positively or directly worded and not mixed with negatively worded items.' (Barnette, 2000, p. 363)
Obtaining reasonable and honest responses is rarely a problem in most usability evaluation settings. ...
Thanks for your considered comments. I've read your article, but it's not clear to me if your assertion is that the "acquiescence bias" doesn't exist, or that it just doesn't apply to questionnaire-based usability satisfaction measures. Could you elaborate?
Sorry, it took me a while to get back here. To elaborate, I am not suggesting that acquiescence bias never exists, but that it almost always doesn't matter in the development of questionnaires intended to measure satisfaction with usability that are administered during or immediately following a usability test, for the reasons laid out in my paper in the International Journal of Human-Computer Interaction (evidence from the literature review that mixture of positive and negative items leads to artifacts in construct validity; demonstration of how response style artifacts cancel out when comparing products).
Furthermore, Jeff Sauro (www.measuringusability.com) and I recently completed some research showing that this "cure" of mixing item alignment is worse than the "disease" of response style bias, at least in the context of a commonly-used questionnaire, the System Usability Scale (SUS). In the upcoming CHI 2011 conference in Vancouver (May 7-12), we'll be presenting a paper entitled, "When Designing Usability Questionnaires, Does It Hurt to Be Positive?" If you're planning to attend, drop by -- I think you'll find it interesting. I'll post the abstract in a separate reply.
As promised in my previous post, here is the abstract for the paper Jeff Sauro and I will be presenting at CHI 2011:
"When designing questionnaires there is a tradition of including items with both positive and negative wording to minimize acquiescence and extreme response biases. Two disadvantages of this approach are respondents accidentally agreeing with negative items (mistakes) and researchers forgetting to reverse the scales (miscoding). The original System Usability Scale (SUS) and an all positively worded version were administered in two experiments (n=161 and n=213) across eleven websites. There was no evidence for differences in the response biases between the different versions. A review of 27 SUS datasets found 3 (11%) were miscoded by researchers and 21 out of 158 questionnaires (13%) contained mistakes from users. We found no evidence that the purported advantages of including negative and positive items in usability questionnaires outweigh the disadvantages of mistakes and miscoding. It is recommended that researchers using the standard SUS verify the proper coding of scores and include procedural steps to ensure error-free completion of the SUS by users. Researchers can use the all positive version with confidence because respondents are less likely to make mistakes when responding, researchers are less likely to make errors in coding, and the scores will be similar to the standard SUS."
I really love the idea of using a word list and word cloud as a way to gain feedback, and have been using it for the last couple of days in some user testing.
What we've found is that users are choosing words that don't reflect their experience at all! Users that have clearly struggled with using a system are choosing words like straightforward and easy to use, despite requiring assistance to complete most tasks.
Is there anyway to word the request, or change the word list, that would encourage users to be more reflective of reality when choosing words?
You clearly need to get to the bottom of why people are choosing words that don't appear to reflect their behaviour. What do people say in the post-test interview? The interview is more important than the actual words chosen (despite the seductiveness of the word cloud) so you should be asking, "I noticed that 'straightforward' was one of the words you chose. Can you tell me what led you to pick that word?" When I do this, I sometimes find that a participant is using the word differently to what I expect, so don't be surprised to hear, "I chose straightforward because the product is much too basic for my needs". It's a case of you say to-mah-to, I say to-may-to.
Web Usability: An Introduction to UX
Sept 23-24, London: Get hands-on practice in all the key areas of usability, from identifying your customers through to usability testing your web site. More details
Download the best of Userfocus. For free.
100s of pages of practical advice on user experience, in handy portable form. 'Bright Ideas' eBooks.
Every month, we share an in-depth article on user experience with over 8,000 newsletter readers. Want in? Sign up now and download a free guide to usability test moderation.
User Experience Articles
Our most popular articles
Our most commented articles
Our most recent articles
- Feb 4: Cheap and free under-the-radar alternatives to field visits
- Jun 3: What Gordon Ramsay can teach us about giving feedback to design teams
- May 6: My place or yours? How to decide where to run your next usability test
- Apr 8: The usability error you don't know you're making
- Mar 4: Adapting your usability testing practise for mobile