We began by defining the critical tasks for the web site. For example, a critical task for the Orange web site is to select and buy a handset and a talk plan. We then measured user performance by asking real customers to carry out the tasks under real-world conditions. From these tests, we collected separate measures of effectiveness, efficiency and satisfaction (the three components of usability, as defined by the International Standards Organisation, ISO 9241-11:1998).
Using a proprietary algorithm, we then computed a customer experience benchmark and displayed it on a 1-page dashboard. The benchmark could be tracked across time, across releases and across competitor web sites. In order to help embed the benchmarking process in the company, we trained Orange's internal Human Factors team in the method and developed a monitoring tool, written in Microsoft Excel. The tool guided Orange's internal team through the steps of setting up a usability test, collecting the data from participants, and creating reports.
User experience metrics help provide an unambiguous specification of what usability means, taking into account the key success factors for the web site. This keeps the development team focussed on what's important to customers. At a practical level, collecting user experience metrics also helps streamline the development process and improve quality.