[Philosophy] [Types of Assurance] [Types of Resource] [Rationale]

Six approaches to the assurance of usability have been described by Daly-Jones et al. (1997), Bevan et al (1998),  Earthy (1998a,b) and EUSC (1998).  Bevan (2001b) describes the international standards that relate to each approach.  From a practical point of view each approach can have a role to play in providing assurance of the usability of a product or system.  As such, the approaches may be seen as components of a managed assurance scheme. For any business a combination of components will be required to give the required degree of assurance that product quality targets as regards usability are valid and achievable.  The table below summarises each component, their purpose and makes comments on practical application.

Components of a managed assurance scheme

Component	What	Purpose(s)	Comment
Product Attributes	Assessing whether a product conforms to ergonomic guidelines e.g. many parts of ISO 9241	Used at end of a development phase to verify successful completion Used at end of project as part of certification May be diagnostic, i.e. provide advice on re-design of the product Conformance to the principles of ISO 9241 meets the software ergonomics requirements of 90/270, the VDU Directive	Retrospective analysis Specific to a product or version of a product Difficult to relate to context of use Does not assess consequence of defects
User Performance and Satisfaction	Measuring the usability of a product e.g. ISO 9241 Part 11	Used at end of a development cycle for validation Used at end of project as part of acceptance Can be used to assess suitability of a product to be acquired May be combined with diagnostic feedback, i.e. provide advice on re-design of the product	Retrospective analysis Specific to a version of a product Only valid for the context used for measurement Does not diagnose cause of defects Thorough assessment may be costly
Process Certification	Assessing whether a user-centred development process was used e.g. Bevan et al (1998)	Increased confidence for purchaser Feedback for developer Interpretation of ISO 13407 for generic products Project process definition Applies to a single product or a product line Depends on developer having an identifiable process	Retrospective analysis Specific to a product or product line
Organisational Human-Centredness	Assessing the maturity of human-centredness of an organisation e.g. Human Centredness Scale (Earthy, 1998b)	Initial assessment of client Awareness-raising for Maturity and Usability Model for management/attitude change Extra scale for software/system process assessments Source of practices for software/system assessments	Not precise about how to bring about improvements
Technical Competence	Accrediting the ability of an organisation to act as a provider of usability services e.g. Bevan et al  (1998), MErgS, EurErgs	Accreditation of usability consultants Subcontractor qualification Design of training Organisational development Staff development Professional qualification	Sensitive to staffing Limited applicability Does not address product development
Process Capability	Assessing the capability of an organisation to perform user-centred activities e.g. ISO TR 18529	Assessment of a baseline of user-centred processes Assessment for accreditation of user-centred processes Elaboration of ISO 13407 Education in what needs to be done in user-centred design Model for process improvement Industry reference model	Organisational commitment required Novel extension to  process assessment

 DECIDING THE BEST COMBINATION OF COMPONENTS TO ASSURE USABILITY

          This section reviews general approaches to the assessment of the six components with respect to usability.  The question for the Human Factors practitioner is, which combination gives the best management of project risk? 

          Traditional assurance of product attributes takes a product and tests it against an agreed standard (e.g. Microsoft Windows™ look and feel).  Evaluations based on fixed performance or feature attributes become progressively less useful with product complexity.  This is especially limiting with information technology (IT) systems and quality attributes as complex as usability.  Product inspection can find some possible causes of usability problems, but cannot assess their consequences. 

          Assurance of user performance and satisfaction (e.g. "a specified group of secretaries shall be able to complete an expenses form with no errors in under an hour") forms a necessary element in usability assurance.  However, the specification of valid performance metrics and test conditions may not be possible until well after a project has started.  Whilst forming part of product assurance, this element contributes to risk management only late in the project and cannot be used for selecting a supplier or assessing project risk at the early stages. 

          Process Certification can only be carried out retrospectively.  For this reason it is of most value (a) to potential purchasers and users of a certified product and (b) as evidence that a contract requirement to apply a process standard has been met. Because process-oriented product certification schemes test a more stable set of criteria than product assessments they are more often applied to product lines rather than single types or versions.  In an attempt to facilitate a use of process assessment for product lines, Davies and Brady (2000) have proposed the concept of 'project capabilities' for use in one-off and small batch complex products or new product lines.  The project capability concept addresses the activities of bid preparation and project execution, and is in many ways more akin to process capability (see below) than process certification.

          All the more traditional components of assurance (i.e. Product Attributes, User Performance and Process Certification) evolved when the project drivers of cost and timeliness were much less demanding. Concurrent engineering (where project phases are to a greater or lesser extent overlapped), which evolved to meet more stringent cost and time requirements, creates a new problem in that projects need to take account of changes in user requirements, context of use and base technology during design and development.  The value of traditional assurance schemes is reduced and those responsible for assurance need to find ways of predicting product quality.  These are discussed below.

          Generic approaches to maturity based on Crosby (1978), such as Organisational Human-Centredness (Ehrlich & Rohn, 1994; Flanaghan, 1996; Earthy, 1998b), identify potential risks but are not particularly precise about specific solutions. Technical Competence in the relevant discipline is of course relevant (eg. professional qualifications such as MErgS - Member of the Ergonomics Society - and EurErgs - European Ergonomist), but in all but the smallest projects factors such as team dynamics, and project context, will have a more significant impact on product attributes than individual contributions alone. 

          Process Capability (Humphrey, 1989) is unique in offering (a) a prediction of the ability of a business to deliver a product that meets a required level of performance, (b) an indication of the factors that hinder this ability, and (c) the means of addressing such shortcomings.  These strengths have led to the widespread adoption of process capability as an element in the assurance of timely and effective software delivery. This paper describes the development and application of process capability in the assurance of usability.

                ISO TR 15504 Software process assessment defines a process as: a set of interrelated resources and activities that transform inputs into outputs. Processes are defined at the level of what is done to develop and operate a system (Annex 3 contains examples of human-centred processes). Processes are specified through methods, techniques, work instructions, etc. A process has a purpose and fulfils a business requirement. A disciplined evaluation of an organisation’s processes against a reference model of good practice is called process assessment.  Processes are assessed against a “capability scale”, an ordinal scale that defines the extent to which the process is implemented and controlled.  Process assessments generally focus on identifying improvement priorities (i.e. a formative evaluation), but can be summative against a required capability level. Action taken to change an organisation’s processes so that they meet the organisation’s business needs and achieve its business goals more effectively is called process improvement.  Process improvement is an iterative activity that starts with an assessment of current capability.  The cycle of assessment and improvement can be tailored to business need in a structured manner.

Lloyd’s Register of Shipping and Process Contracting Limited © 2003