Usability Test Reporting
You know when a profession is mature, because the services and products offered by practitioners share a fair amount of consistency. So for example if I commission two different architects to carry out a house survey, their reports should be pretty similar. One may be cheaper than another, and one may be better able to describe the problems with the roof in terms I will understand, but the problems they find should be consistent.
Usability and variability
Embarrassingly, we have known for a while now that this doesn’t apply to usability testing. The well-publicised work of Rolf Molich shows us that when different usability groups are asked to carry out a web site evaluation, they find lots of usability issues. The problem is that each group finds only a sub-set of all the usability problems. Just one group (of nine) in Molich’s study found more than 25% of the problems. (More detail can be found at Molich’s web site).
Given that all these people would describe themselves as "usability professionals", it’s hard to blame the findings on different skill sets or competencies. A more likely contributor is the fact that the different groups carried out usability testing in a variety of different ways.
So it’s interesting that, during the period of Molich’s work, the US National Institute of Standards and Technology (NIST) initiated an effort to "Define and validate a Common Industry Format for reporting usability tests and their results". The overall aim of the project was to increase the visibility of software usability.
The Common Industry Format (or ‘CIF’ to its friends) isn’t a visual template that helps make usability reports look the same, nor does it tell you how to run a test. However, the framework of the report defines a consistent method of carrying out usability tests. For example, you can only write a compliant report if you take objective usability measures of effectiveness, efficiency and satisfaction (these definitions come from the international usability standard, ISO 9241-11). The report also requires information such as the design of the test (including information about independent variables), data scoring procedures (including operational definitions of usability measures) and details of the statistical analysis used. Following this type of guidance will help ensure consistency and contrasts with the more common approach, where usability tests aren’t "designed", they just happen. The CIF became an ANSI standard in December 2001 (ANSI/NCITS 354-2001) and became an international standard in 2006 (ISO/IEC 25062:2006 "Common Industry Format (CIF) for usability test reports").
Formative and Summative tests
The CIF makes a distinction between "formative" and "summative" usability tests. Formative tests are carried out:
- During the development of a product;
- To mould or improve the product;
- Virtually anywhere (you don’t need a lab);
- With the test administrator and the participant co-present.
The outputs from a formative test may include:
- Participant comments in the form of a "thinking aloud" narrative (for example, attitudes, sources of confusion, reasons for actions);
- Photographs and highlights videos;
- Usability problems and suggested fixes.
In contrast, summative tests are carried out:
- At the end of a development stage;
- To measure or validate the usability of a product;
- To answer the question: "How usable is this product";
- To compare against competitor products or usability metrics;
- To generate data to support marketing claims about usability;
- In a usability lab;
- With the participant working alone.
The outputs from a summative test may include:
- Statistical measures of usability (for example, success rate, average time to complete a task, number of assists);
- Reports or white papers.
The CIF applies to summative usability tests. The table below compares the advantages and disadvantages of the two methods.
|Formative or diagnostic test||
|Summative or measurement test||
Achieving consistent tests
If your design process is human centred, and you aim to follow the process usability standard, ISO 13407, then you are probably carrying out summative usability tests already. The CIF will provide you with the consistent framework you need to report your results.
The usability tests in Molich's study used formative methods, to which the CIF doesn't strictly apply. So I hope Molich can be persuaded to repeat his study with summative methods and CIF-compliant reporting. This may show that the usability profession is more mature than we think.
About the author
Dr. David Travis (@userfocus on Twitter) is a User Experience Strategist. He has worked in the fields of human factors, usability and user experience since 1989 and has published two books on usability. David helps both large firms and start ups connect with their customers and bring business ideas to market. If you like his articles, why not join the thousands of other people taking his free online user experience course?
Foundation Certificate in UX
Gain hands-on practice in all the key areas of UX while you prepare for the BCS Foundation Certificate in User Experience. More details
Every month, we share an in-depth article on user experience with over 10,000 newsletter readers. Want in? Sign up now and download a free guide to usability test moderation.
User Experience Articles
Our most recent articles
Search for articles by keyword
- 7 articles tagged accessibility
- 4 articles tagged axure
- 5 articles tagged benefits
- 16 articles tagged careers
- 8 articles tagged case study
- 1 article tagged css
- 8 articles tagged discount usability
- 2 articles tagged ecommerce
- 19 articles tagged ethnography
- 14 articles tagged expert review
- 2 articles tagged fitts law
- 5 articles tagged focus groups
- 1 article tagged forms
- 7 articles tagged guidelines
- 11 articles tagged heuristic evaluation
- 7 articles tagged ia
- 14 articles tagged iso 9241
- 11 articles tagged iterative design
- 3 articles tagged layout
- 2 articles tagged legal
- 11 articles tagged metrics
- 3 articles tagged mobile
- 8 articles tagged moderating
- 3 articles tagged morae
- 2 articles tagged navigation
- 9 articles tagged personas
- 15 articles tagged prototyping
- 7 articles tagged questionnaires
- 1 article tagged quotations
- 4 articles tagged roi
- 17 articles tagged selling usability
- 12 articles tagged standards
- 47 articles tagged strategy
- 2 articles tagged style guide
- 5 articles tagged survey design
- 6 articles tagged task scenarios
- 2 articles tagged templates
- 21 articles tagged tools
- 58 articles tagged usability testing
- 3 articles tagged user manual