Discuss Measuring satisfaction: Beyond the usability questionnaire

I proposed the word list to a product team who were suggesting a questionnaire for users after a session with a new product, and was absolutely delighted with the results. Even if - as in our case - the users choose not to pick many words, it acted both as a great tool to investigate why they chose the words they did and as a persuasive piece of evidence for developers and managers who were not present.

I can see this working as a way to structure a post-experience interview but it must be a pain to count up any large number of results for analysis. I wonder if I could make it into an online survey.. maybe Google Docs would do it...

We have been using the word choice for a long time now together with the three elements of usability testing of our product (effectiveness, efficiency and satisfaction). We have realized that it's very important because when we have strange and/or conflicting results from the usability measures the word choice often offers a very good explanation to why. It is not only the words that have been chosen that are interesting but also the words that haven't been chosen. Of course this is easier with a smaller list of words (we have about 18 words with equal amount of negatives as positives).

This is a very useful article with implications that go well beyond usability testing in the traditional sense.

I work with professional services firms to help them measure client satisfaction and loyalty. My starting point is nearly always a variant of the Net Promoter Score (NPS) recommended by Fred Reichheld. (Likelihood to Recommend and Likelihood to continue Using followed by open-ended probes around what it would take to get to a perfect score.)

I could easily adapt my approach to include the exercise described in this post.

By presenting a tailored list of words, which would include both negative and positive adjectives and brand attributes, I could quickly focus in on areas that the respondent felt were most relevant to his or her experience with the service.

The list could be presented via an online questionnaire, the link for which I'd forward during or shortly prior to the telephone interview. At the appropriate time, we'd go through the list (which would be randomized) and the respondent could quickly tick off the words of interest. On the next page, only the selected words would appear and s/he could choose the five for further discussion.

The survey site would capture the data - and I'd be able to follow along on paper as the respondent make the choices.

The technique could be adapted well to focus groups also.

Thanks again for describing the Microsoft approach and for preparing the Excel tool.

Andy Perkins
The Customer Satisfaction Questionnaire Blog

I want to address the common misconception that it is necessary to "balance" usability questionnaires -- to have an equal number of positively and negatively aligned items. For research investigating evidence of the influence of response styles (such as the agreement or acquiescence tendency) on this questionnaire, including all references, see:

Lewis, J. R. (2002). Psychometric evaluation of the PSSUQ using data from five years of usability studies. International Journal of Human-Computer Interaction, 14, 463-488. (drjim.0catch.com)

Even though my research was on the PSSUQ, the discussion/findings apply generally to questionnaire-based usability satisfaction measuresl.

From p. 467-469:

"It is a common practice in questionnaire development to vary the tone of items so, typically, one half of the items elicit agreement and the other half elicit disagreement. The purpose of this is to control potential measurement bias due to a respondent’s response style. An alternative approach is to align the items consistently.

Our rationale in consistently aligning the items was to make it as easy as possible for participants to complete the questionnaire. Furthermore, the use of negatively worded items can produce a number of undesirable effects (Ibrahim, 2001), including 'problems with internal consistency, factor structures, and other statistics when negatively worded items are used either alone or together with directly worded stems' (Barnette, 2000, p. 363). The setting in which balancing the tone of the items is likely to be of greatest value is when participants do not have a high degree of motivation for providing reasonable and honest responses (e.g., in some clinical and educational settings).

'Thus, first and foremost, the survey or questionnaire designer must determine if using negatively worded items or other alternatives are needed in the context of the research or evaluation setting. Unless there is some pervasive and unambiguous reason for not doing so, it is probably best that all items be positively or directly worded and not mixed with negatively worded items.' (Barnette, 2000, p. 363)

Obtaining reasonable and honest responses is rarely a problem in most usability evaluation settings. ...

I really love the idea of using a word list and word cloud as a way to gain feedback, and have been using it for the last couple of days in some user testing.

What we've found is that users are choosing words that don't reflect their experience at all! Users that have clearly struggled with using a system are choosing words like straightforward and easy to use, despite requiring assistance to complete most tasks.

Is there anyway to word the request, or change the word list, that would encourage users to be more reflective of reality when choosing words?