I often tell people that the great thing about taking a user centred approach to design is that people don’t change much over time, whereas technology changes rapidly. So focusing on users is a great way to future proof your designs. And the same thing occurred to me as I looked over the quiz: the kinds of question I was asking 13 years ago to measure an organisation’s usability maturity are still just as relevant today.

The original set of questions were focused on e-commerce sites specifically, so I’ve now re-phrased some of them so they refer to any product, service and any application area.

You can use the results of this test as a “before” and “after” measure of your organisation’s usability maturity, or you can use it to identify particular areas for improvement. You can also use it as a checklist after you complete each phase of the design to check that you’ve addressed the key issues.

I’ve split the questions into broad groups that cover business analysis, user research, design and deployment. I’ve deliberately avoided tying it to any specific development methodology, so you can ask these questions whether your approach is waterfall or agile. The only assumption about your development process is that there’s an opportunity for iteration.

Treat these as YES/NO questions, and use them to generate team discussion about where there might be room for improvement in your design and development process.

Business analysis

This step provides the business context for the design activity. You begin by identifying the stakeholders for your new development. This includes all those people who have an interest in the success or failure of the product or service, such as management, technical support and regulatory bodies in the industry. Next you identify your user experience vision: your view of what using the product or service will be like 5 years or so in the future. This provides a design target that you can use to ensure you are progressing towards your design goals. The final part of this step is to segment the market, so you can identify the key user group that will become the focus of your design solution.

Questions to ask:

• Have all the project stakeholders been identified (not just the end users)?
• Have the motivations of each stakeholder been made explicit?
• Has existing research been collated and reviewed (such as call centre analysis, web analytics and interviews with front line, customer facing people)?
• Does the project have a user experience vision that clearly communicates the project aims to the design and development team?
• Is it clear how the product or service will meet the business objectives?
• Are the objectives of the product or service clear and unambiguous and are they consistent with the organisation’s overall business goals?
• Is the product or service meeting a real user need?
• Have competitor systems been identified?
• Has user research been carried out to identify the most appropriate market or beachhead segment that will be the prime focus of the first release?
• Are the target users clearly identifiable (i.e. the product or service avoids trying to be “something for everybody')?

User research

In this step, you aim to build a rich description of users, the environment in which they use the product or service and the critical tasks they want to carry out with it. You begin by building user profiles: a set of personas that describe the goals and behaviours of the key user groups. Next, you create environment profiles: descriptions of the social, technical and physical environment within which the product or service will be used. Finally, you identify red routes: a list of the critical tasks that users need to carry out easily and successfully for the system to be a success.

Questions to ask:

• Have potential users been observed in context and have their needs been identified?
• Is there a description of the range of intended users?
• Is there a description of the environments (physical, socio-cultural and technical) within which users will use the product or service?
• Is there a description of the different tasks that users will want to carry out?
• Has this list of user tasks been prioritised and have the critical and frequent tasks been identified?
• Has this prioritised list of tasks been used to deliver the design in stages, with the most important functionality delivered first?
• Was information about users, their tasks and their environments collected using a range of techniques (e.g. interviews, observation, surveys)?
• Do all members of the design team observe users regularly?
• Has the context of use information been summarised in an engaging and practical form (e.g. as personas and user story maps)?
• Has the context of use information been used to drive the design process?

Design

This step is an iterative process. You start the process by developing key performance indicators: quantitative measures, based on key user and business requirements, that the management team use to determine if the design is ready to launch. You then move on to develop the information architecture: the high-level, conceptual model, showing how all of the functions and features will hang together. Next, you create prototypes, starting with paper sketches and then moving onto more interactive prototypes. The final part of this iterative loop is to evaluate usability by asking potential users to carry out realistic tasks.

Questions to ask:

• Have key performance indicators been set and has the design been evaluated against these metrics?
• Are these criteria based on clear user input and do they cover the areas of user performance (effectiveness and efficiency) as well as user satisfaction?
• Has the conceptual model been identified?
• Has a content inventory been produced, and has this been sorted, ordered and categorised to match users’ mental model of the information?
• Is the navigational framework and the terminology used in the design based on this user research?
• Have various prototypes been developed and evaluated and have the results of these evaluations been used to improve the designs?
• Has the design been tested for usability with representative users?
• Has the design been changed to address the findings of these evaluations?
• Have usability defects been prioritised based on their impact on users and have the defects been tracked to completion?
• Has the design been documented in a living prototype or a Style Guide?

Deployment

To paraphrase Winston Churchill: this is not the end. It is not even the beginning of the end. This is the end of the beginning. In this step you find out how your product or service was actually used in practice and use these insights to drive the next release of the product or service, or to design a future product.

Questions to ask:

• Is feedback from users (e.g. help desk reports, post-release surveys and field visits) collected regularly and frequently to ensure that the product or service continues to meet business and user needs?
• Is this information used to identify changes in the user base, their environments and their tasks?
• Is the specification of the context of use (user, environment and task profiles) updated regularly to reflect any major changes?
• Is this information used to identify the key areas of the product or service to maintain and enhance?
• Are measures of conversion rate, fulfilment and customer retention tracked regularly and frequently?
• Is there verification that the business requirements were met?
• Is there verification that the stakeholder requirements were met?
• Is there verification that the user needs were met?
• Is there verification that the key performance indicators were met?
• Did the design team hold a post-implementation meeting to discuss the effectiveness of the design processes and identify areas of improvement?