1 1 Lecture 7 Requirements Validation – Part 1 Enterprise Business Requirements 32569 Autumn 2017 2 Part 1- Lecture Outline Ø Software Quality Assurance Ø Verification and Validation Ø What is requirements validation? Ø Analysis versus Validation Ø Advantages and limitations of validation Ø Different levels of formality in validation Ø Formal Inspection Process Ø Formal requirements reviews and inspections Ø Assignment 2 Discussion2 3 Software Quality Assurance Ø Requirements validation and inspections is an important part of the Software Quality Assurance (SQA) activities. Ø SQA form a set of management activities to ensure that software products meet some acceptable standards of quality. Ø SQA activities are applied throughout the software development process 4 Software Quality Assurance Ø Goals of SQA activities: • To improve software quality by appropriately monitoring both the software and the development process that produces it. • To ensure compliance with the established standards and procedures for the software and the software process. • To ensure that any problems in the product, the process, or the standards are brought to management’s attention, so these problems can be addressed and fixed. Ø Software engineers address quality by: • applying rigorous technical methods • conducting formal technical reviews • performing well planned software testing3 5 Verification and Validation 6 Application versus Machine Domain Adapted from Jackson, 1995, p1704 7 V & V criteria for Requirements Ø Verification Criteria: • The Program running on a particular Computer satisfies the Specification • The Specification, given the Domain properties satisfies the Requirements Ø Validation criteria • Did we deliver (and understand) all the important Requirements? • Did we discover (and understand) all the relevant Domain properties? Source: Adapted from Jackson, 1995, p170-171 8 V and V example Ø Requirement R: • Reverse Thrust shall only be enabled when the aircraft is moving on the runway Ø Domain Property D: • Wheel pulses on if and only if wheels turning • Wheels turning if and only if moving on runway Ø Specification S: • Reverse thrust enabled if and only if wheel pulses on Ø Verification • Does the flight software P, running on the aircraft computer C, correctly implements S? • Does S, in context of assumptions D, satisfy R? Ø Validation • Are our assumptions D, about the domain correct? Did we miss anything? • Are the requirements R, what is really needed? Did we miss anything?5 9 What is requirements validation? Ø The objective of requirements validation is to check the set of requirements which have been defined and specified to discover the following potential problems with these requirements: • Lack of conformity to quality standards • Poorly worded and ambiguous requirements • Conflicting requirements • Infeasible requirements • Incomplete requirements 10 Analysis versus Validation activities Ø To fix problems of requirements specification, we need to repeat the earlier phases or activities of RE such as requirements elicitation, analysis and negotiation. Ø Although analysis and validation activities have much in common (e.g. finding omissions in and conflicts between the system requirements), we can make a distinction between them.6 11 Analysis versus Validation activities Ø Requirements analysis is concerned with “raw” requirements as elicited from stakeholders. These requirements are usually incomplete and are expressed in an informal and unstructured way. We may use several notations (e.g. UC, DFD, ERD, STD) to describe and analyse these requirements. Ø Requirements validation is concerned with checking a final draft of a requirements document that should include all the system requirements and where identified inconsistencies and incompleteness may have been removed. The document and the requirements in it should follow some defined quality standards and should be acceptable by the project sponsors (and other stakeholders). 12 Requirements problems that may be discovered during validation: Ø Lack of conformance to quality standards Ø Poorly worded requirements which are ambiguous and open to more than one interpretation Ø Errors in models of the system or the problems that needs to be solved by using the system Ø Requirements conflicts which were not detected during the analysis process Ø Requirements that are not feasible to implement. Ø Requirements that are in conflict with the domain properties discovered during domain analysis and elicitation7 13 Validation Inputs and outputs Requirements Validation List of Problems/Defects Agreed Actions Requirements Specification Document Organisational/Domain Knowledge Organisational/quality standards 14 Limitations of validation Ø Often there is nothing against which the requirements can be validated. Hence, we cannot prove that a specification is absolutely correct with respect to user requirements. Ø The validation process can only increase our confidence that the specification describes a system which when built will meet the real needs of the intended users. Ø Requirements validation could be a prolonged process and there is always a natural tendency to rush the validation so that system development can begin.8 15 Advantages of validation Ø Discovering and fixing requirements problems early can avoid a lot of expensive rework of the system design and implementation. Ø Studies have shown that errors in delivered software systems which are a consequence of requirements errors may cost up to 100 times as much as programming errors. 16 The requirements review process Plan Review Distribute Documents Prepare for Review Hold Review Meeting Follow-up Action Revise Documents Source: Page 90 and 91 of Kotonya and Sommerville9 17 Pre-review Checking Check Document Check Document completeness Check Document Against standards Run Automatic Document checker Requirements Document Problem Report Source: Page 93 of Kotonya and sommerville 18 Compliance with standards Ø Before distributing the requirements document for general review, one should carry out a pre-review check to ensure that the document structure and the defined requirements meet the standards that has been adopted (such as IEEE or ISO standards or organisational level standards) Ø An analyst or an engineer who knows standards well but has not been involved in the specification writing should typically do this check.10 19 Formal Requirements Inspection Ø Assign a group of people to systematically inspect the requirements specification, then meet to discuss problems identified with the requirements, and agree on how these problems should be fixed. Ø Formal inspections have proved to be a cost effective method of identifying defects in requirements related documents 20 Choice of Inspection team members Ø Members of the inspection team should ideally be multi-disciplinary and must be selected from people with different backgrounds such as end-users, customer representatives, domain experts, software engineers responsible for system design and implementation, and of course one or more requirements engineers/business analysts.11 21 Validation Checklist Ø Checklists assist in focusing the attention of those who are responsible for validation on the critical attributes of the requirements document. Ø Checklists add structure and rigour to the validation process such that validators will not forget to check some aspect of requirements. Ø Checklists are useful in training people in how to do requirements validation. 22 Validation checklist contents Ø Questions in a validation checklist may be based on some or all of the following general issues: • Are requirements complete? • Are requirements consistent? • Are requirements understandable? • Are requirements ambiguous? • Is the requirements document organised & structured? • Are requirements traceable? • Is information unnecessarily repeated in the document? • Are the requirements document as a whole and do individual requirements conform to a defined standard?12 23 Example of checklist questions Checklist question Quality attributes Is each requirement uniquely identified? Traceability, conformance to standards Are specialised terms defined in the glossary Understandability Does a requirement stand on its own or do you have to examine others to understand what it means? Understandability, completeness Do individual requirements use the same term in different ways? Ambiguity Is the same service requested in different requirements? Consistency, redundancy 24 Example Source: Kotonya & Sommerville, page 9913 25 Problems & Recommendations Source: Kotonya & Sommerville, page 99 26 Example of a checklist Requirem ents complete consistent Easy to understan d Unambiguous structured traceable Conform to standard R1 R2 R3 R414 27 Examples of Follow-up Actions Ø Requirements clarification by either rewording some sections or rephrasing the entire text Ø Find the missing information Ø Resolve requirements conflict Ø Review unrealistic Requirements with the possibility of removing them. Ø Postpone requirements to the next release if not currently feasible 28 Details and Size of a checklist Ø Checklists are normally expressed in a fairly general way. Ø They should be expressed such that non system experts should be able to understand them. Ø As a general rule Checklists should not be too long, normally no more than ten items but it depends on the complexity of the system and the size of the document too.15 29 Limitation of Checklists Ø If the checklist becomes too general and hence too vague, it is impossible to answer checklist questions in any useful way. Ø Unlike program inspection checklists where fine-grained checklists are concerned with very specific defects, they are not very practical for requirements inspection because of the differences between the requirements for different types of systems. 30 Reviewer’s checklists Ø You can find some sample checklists for reviewing software requirements specifications and use case documents as well as other tools and techniques at: http://www.processimpact.com/pr_goodies.shtml16 31 Formal review Ø Formal reviews follow a well-defined process. Ø It produces a report that identifies the material that has been reviewed, the reviewers, and the judgement of the reviewers as to whether the product is of acceptable standard. Ø The members of a formal review team share the responsibility for the quality of the review. Ø The most widely known type of formal review is called an “inspection” 32 Different levels of “formality” Ø Informal: • Meeting over coffee with a colleague • Regular project team meetings Ø Formal: • Scheduled meeting • Prepared participants • Defined agenda • Specific format • Documented output17 33 Different levels of “formality” Ø Informal peer reviews • Desk Checks • Pass around • Walkthrough Ø Semi-formal Reviews • Model validation Ø Prototyping Ø Management progress reviews Ø Formal Inspections Ø Testing informal formal 34 The formal inspection Process Ø Formal inspection is recognised as software industry’s best practice. It is a process management tool used to collect defect data to analyze and improve the quality of the development process. Ø It plays a major role in training junior staff and transferring expertise. Ø The inspection process was first developed and introduced by Michael Fagan in1970s at IBM mainly for inspecting program code. Since then others have extended his method for design and other software work products.18 35 Roles in Formal Inspection -1 Ø Author: The author of an SRS is usually the analyst who has elicited the requirements and wrote the specification. The author takes a passive role during inspection Ø Moderator (or leader): plans the inspection with the author, coordinates the activities, and facilitates the inspection meeting by distributing the material to the participants several days before the meeting. 36 Roles in Formal Inspection - 2 Ø Reader – One inspector is assigned the role of reader who during the inspection meeting paraphrases the SRS one requirement at a time. The other participants then point out potential problems. Ø Recorder (or scribe) – uses the standard inspection forms provided to document issues raised and the defects found during the inspection meeting19 37 Entry Criteria for Inspection Ø Entry Criteria are pre-requisites that set some clear expectations for the authors to follow when preparing for an inspection: • The document conforms to the standard template • The document has been spell checked • The author has visually examined the document for any layout errors. • Any predecessor or reference documents that the inspectors require to examine the documents are available. • Line numbers are printed on the document to facilitate referring to specific locations during the inspection meeting • All open issues are marked as TBD. • The moderator did not find more than three major defects in a ten minute examination of a representative sample of the document. General Inspection Guidelines Ø Before the review stage § Schedule formal reviews/inspections into the project plan § Provide training for reviewers if needed § Ensure all involved are prepared in advance Ø During the review • Review the product and not its creator § Keep comments constructive, professional and task focused • Follow the agenda closely § Group leader must prevent reviewers drifting • Limit extraneous debate and rebuttal § Record issues for later discussion/resolution • Identify issues and problems but don’t try to resolve them • Take written notes Ø After the review • Assess the quality of the review process followed 3820 Review Meeting ØDo not start until all members are present ØGroup Leader states: “We are here to review product A for purpose B” Ø Group Leaders introduces the reviewers and explains the recording procedure Ø Group Leader briefly checks that • Everyone has received the reviewing material • Everyone has come prepared ØGroup Leader explains the method for review 39 40 Different Methods for Inspection Ø Checklist • uses a checklist of questions/issues • review structured by issue on the list Ø Walkthrough • one person presents the product step-by-step • review is structured by the product Ø Round Robin • each reviewer in turn gets to raise an issue • review is structured by the review team Ø Speed Review • each reviewer gets 3 minutes to review a chunk, then passes to the next person • good for assessing comprehensibility21 41 Exit Criteria for Inspection Ø Your inspection process must have defined exit criteria that must be satisfied before the leader announces that the inspection is complete, e.g.: • All issues raised during the inspection have been addressed • Any changes made in the document and related work products were made correctly • The revised document has been spell-checked • All TBDs have been resolved, or each TBD’s resolution process, target date, and owner has been documented. • The document has been checked into the project’s configuration management system 42 Defect Checklists Ø In order to assist the inspection team for typical errors, it is common to develop a checklist for each type of requirements documents. Such checklists bring to the attention of the inspection team the most frequently cited problems.22 43 Example Checklists Ø Wiegers' Checklist for Inspecting Requirements Specifications http://www.processimpact.com/pr_goodies.shtml Ø The NASA Formal Inspections Guidebook and Standard are available from the NASA Software Technology Assurance Center at http://satc.gsfc.nasa.gov/fi/fipage.html Ø Don Firesmith's Checklist for Requirements Specifications in the OPEN process framework web site: http://www.opfro.org/index.html?Components/WorkProducts/RequirementsSet/SystemR equirementsSpecification/SystemRequirementsSpecificationInspectionChecklist.html~Co ntents Ø Also these will be available on UTS online: • NASA JPL's Checklist for Requirements • NASA JPL's Checklist for Functional Requirements ASSIGNMENT 2 DISCUSSION 4423 45 Reference Ø Sommerville and Sawyer, “Requirements Engineering, A good practice guide”, Wiley, chapter 8. Ø Kotonya and Sommerville, “Requirements Engineering, processes and techniques”, Wiley, chapter 4. Ø K Wiegers and J Beatty, “Software Requirements” 3rd edition, Microsoft publishing. 2014