Usability Test Data

Some definitions

Let's take a closer look at what is meant by quantitative and qualitative data. Definitions of ‘quantitative’ and ‘qualitative’ are uncontroversial and can be found in any standard statistics text book. Witte (1989), for example, presents the distinction concisely, defining quantitative data as follows:

When, among a set of observations, any single observation is a number that represents an amount or a count, then the data are quantitative.

So the body weights reported by a group of students; or a collection of people's IQ scores, or a list of task times in seconds, or Likert scale category responses, or magnitude rating scale responses, are quantitative data. Counts are also quantitative, so data showing size of family, or how many computers you have at home, for example, are quantitative.

Witte defines qualitative data as follows:

When, among a set of observations, any single observation is a word, or a sentence, or a description, or a code that represents a category then the data are qualitative.

So "yes-no" responses, people's ethnic backgrounds, or religions, or attitudes towards the death penalty, or descriptions of events, speculations and stories, are all examples of qualitative data. Note that numerical codes can be assigned to represent qualitative responses (for example, "yes" could be assigned 1 and "no" could be assigned 2). However, these numbers do not transform qualitative data into quantitative data.

Note the emphasis on "a single observation". The fail-safe way to distinguish between quantitative and qualitative data is to focus on the status of a single observation or datum, rather than on an entire set of observations or data. When viewed as a whole, qualitative data can often bear a striking resemblance to quantitative data. 57 "yes" responses vs. 43 "no" responses looks like quantitative data. But it is not. Although these numbers are important (and essential for some statistical procedures) they do not transform the underlying qualitative data into quantitative data.

The case of rating scales

Rating scales present an interesting case because they are used to capture subjective opinions with numbers. The ensuing data are often considered to be qualitative. However, rating scales are not designed to capture opinions per se, but rather they are designed to capture estimations of magnitude. Rating scales do not produce qualitative data. Data from Likert scales and continuous (e.g. 1-10) rating scales are quantitative. These scales assume equal intervals between points. Furthermore they represent an ordering, from less of something to more of something — where that ‘something’ may be "ease of use" or "satisfaction" or some other construct that can be represented in an incremental manner. In short, rating scale data approximate interval data and so lend themselves to analysis by a range of statistical techniques including ANOVAs. Qualitative data do not have these properties, and cannot be ordered along a continuum, or compared in terms of magnitude (although qualitative data can still be analysed statistically).

While quantitative studies are concerned with precise measurements, qualitative studies are concerned with verbal descriptions of people’s experiences, perceptions, opinions, feelings and knowledge. Whereas a quantitative method typically requires some precise measuring instrument, the qualitative method itself IS the measuring instrument. Qualitative data is less about attempting to prove something than it is about attempting to understand something. Quantitative and qualitative data can be, and often are, collected in the same study. If we want to know how much people weigh, we use a weighing machine and record numbers. But if we want to know what their weight means to them we need to ask people questions, hear stories, and understand experiences. (See Patton, 2002), for a comprehensive exposition of qualitative data collection and analysis methods).

Subjective Data and Objective Data

A related distinction (and a frequent source of confusion, especially when used in the context of qualitative and quantitative data) is that of subjective and objective data. The "rule" to note is that subjective data result from an individual's personal opinion or judgement and not from some external measure. Objective data on the other hand are "external to the mind" and concern facts and the precise measurement of things or concepts that actually exist.

For example, when I respond to the survey question "Do you own a computer?" my answer "Yes" represents qualitative data, but my response is not subjective. That I own a computer is an indisputable fact that is not open to subjectivity. So my response here is both qualitative and objective. If I am asked to give my general opinion about the price of computers, then my response "I think they are too expensive" will be both qualitative and subjective. If I am asked to report the chip speed of my computer and I reply "2.0 GHz" then my response is both quantitative and objective. If I respond to the question "How easy is your computer to use on a scale of one through ten?", my answer "seven" is quantitative, but it has resulted from my subjective opinion, so it is both quantitative and subjective.

Examples of qualitative and quantitative data

	Quantitative	Qualitative
Objective	"The chip speed of my computer is 2 GHz"	"Yes, I own a computer"
Subjective	"On a scale of 1-10, my computer scores 7 in terms of its ease of use"	"I think computers are too expensive"

Confusion often arises when people vaguely assume that "qualitative" is synonymous with "subjective", and that "quantitative" is synonymous with "objective". As you can see in the above examples, this is not the case. Both quantitative and qualitative data can be subjective or objective.

Usability smoke and mirrors

We could put this all down to troublesome semantics and dismiss the matter as being purely academic, but the reality is that clarity of thought and understanding in this area can be critically important. Misunderstanding and — worse — misuse of these terms can signal a poor grasp of one’s own usability data, and may reduce the impact of the results on product design decisions. It can result in the wrong analyses, or in no analysis at all, being conducted on numerical data. For example, it is not uncommon for usability practitioners to collect subjective rating scale data, and then fail to apply the appropriate inferential statistical analyses. (This is often because they have mistakenly assumed they are handling qualitative data and they assume that these data cannot be subjected to rigorous analyses). It is also not uncommon for usability practitioners to collect nominal frequency counts and then to make claims and recommendations based solely on unanalysed mean values.

Handling usability data in this casual way can reduce the value of a usability study, leaving an expensively staged test production with a smoke and mirrors ending. Such outcomes are a waste of company money, they cause product managers to make the wrong decisions, and they can lead to costly design and manufacturing blunders. They also reduce people's confidence in what usability can deliver.

The discipline of usability is concerned with prediction. Usability practitioners make predictions about how people will use a web site or product; make predictions about interaction elements that may be problematic; predict the consequences of not fixing usability problems; and, on the basis of carefully designed competitive usability tests, make predictions about which choice of design a sponsor might wisely pursue. Predictions need to go beyond the behaviour and opinions of a test sample. In this respect we care about the opinions and behaviours of our test sample only insofar as they are representative of the target market of interest. But we can have a known degree of confidence in the predictive value of our data only if we have applied appropriate analyses. So failing to conduct statistical analyses on both quantitative and qualitative data collected during a summative usability test is a difficult wicket to defend. Such a stance on data analysis could be justified only if we cared not to generalise our results beyond the specific sample tested. This would be a very rare event, and in this case we would not actually be testing a sample but rather the entire population of target users.

Quantitative data are not better or worse, or more or less valuable, than qualitative data. But objective, fact-based data do have greater predictive value than subjective data. Where possible usability professionals should strive to design studies that collect objective, fact-based data.

References

Patton, M. Q. (2002) Qualitative Research and Evaluation Methods (3rd Edition). Sage Publications.

Witte, R. S. (1989) Statistics (3rd Edition). Holt, Rinehart & Winston, Inc.