Lade Inhalt...

Risk management in software quality assurance

Risk-Based Testing

Hausarbeit 2010 22 Seiten

BWL - Unternehmensführung, Management, Organisation


Table of Contents

Executive summary

List of Abbreviations

List of Figures

List of Tables

1 Introduction

2 Methodology

3 Definitions
3.1 What is risk?
3.2 Risks & Software
3.3 What is software quality?
3.4 What is software testing?

4 Risk-Based Testing
4.1 Risk identification & classification
4.2 Risk assessment
4.2.1 Quantifying amount of damage (according to REDMILL 2004, pp. 6 - 8) .
4.2.2 Quantifying probability of damage
4.2.3 Using metrics (description according to AMLAND 2000)
4.3 Allocating, selecting and monitoring risk (PINKSTER et al. 2006, pp. 29 - 31)

5 Risk and Requirement-Based Testing

6 Conclusion



ITM Checklist

Executive summary

This presented assignment is part of the MBA studies at the University of Applied Science in Düsseldorf (FOM Hochschule für Oekonomie & Management, Düsseldorf) and covers the subject “Risk Management”.

In general, risk can be expressed as product of amount of damage and probability of damage. Due to the fact that software controls more and more aspects of life in modern industrialised societies, software failures inherit risks for businesses, human health or even human life. Software testing is a structured approach to minimise product risks of software systems. When the problem arises that, due to a given budget and timeframe, it is not possible to cover all parts of the software through testing, Risk-Based Testing is a possibility to test the most critical parts of the software first or more intensive. When using this method, both amount of damage and probability of damage must be quantified. Quantifying the amount of damage must happen by considering the different viewpoints of the software system’s stakeholders, while quantifying the probability of damage can only happen indirectly, for example through quality indicators like the complexity of the software itself, the quality of the documentation etc. When having derived quantitative values both for the amount of damage and the probability of damage, the priority of the test cases can be determined by using a risk matrix. Furthermore, these values can also be used for metrics.

An extension of Risk-Based Testing is Risk and Requirement-Based Testing, where product risks are linked to the requirements against which the software is tested in order to gain an overview if the lists of requirements and risks defined for the software are complete.

List of Abbreviations

illustration not visible in this excerpt

List of Figures

Fig. 1: An example for a risk matrix 8

List of Tables

Table 1: Examples of project and product risks categorised by business and ICT (Taken from PINKSTER et al. 2006, p. 18)

Table 2: Example of linking product risks to quality attributes for an automated teller machine (ATM) (Taken from PINKSTER et al. 2006, p. 24)

Table 3: Linking product risks, priorities and requirements, shown with the example of an ATM (Table taken from PINKSTER et al. 2006, p. 35, shortened)

1 Introduction

Today, software controls more and more tools and machines which are part of the daily life in modern industrialised societies. Therefore, risks exist which are directly linked to the non-functioning of software. Due to this, a sufficient quality of software systems must be ensured through software quality assurance & testing. When in software development projects time and budget do not allow to cover the whole software functionality through software testing, Risk-Based Testing as a method targeted to make sure that the most critical functions are covered may be applied. During this elaboration, the Risk-Based Testing approach will be presented and discussed.

2 Methodology

The research methods used for this elaboration were literature reviews of books, scientific papers and further internet sources.

3 Definitions

3.1 What is risk?

A risk is a problem “…which could occur in the future and has unwanted consequences” (SPILLNER et al. 2008, p. 225, translated). Risk can be expressed as product of amount of damage and probability of damage (SPILLNER & LINZ 2005, p. 184).

3.2 Risks & Software

A failure of software system can inherit heavy risks for businesses. One popular example occurred in 2009, when a software error paralysed the whole mobile phone network of Germany’s largest mobile telecommunication provider, T-mobile (,1518,620388,00.html - 27/4/2010). When safety-relevant software fails, for example the software of an aeroplane, even human life can be endangered. NEUMANN (1995, pp. 12) provides an impressive list how faults in information technology systems caused severe accidents and losses of human life. Therefore, software must be of sufficient quality in order to prevent such events.

According to PINKSTER et al. (2006, p. 17), all risks being relevant for software test projects can be divided into two categories, which are project risks and product risks. PINKSTER et al. define project risks as risks which are related to the management of the testing process. They make a further distinction between business-related project risks which are related to time and budget and ICT-related project risks, for example an instable test environment. According to these authors, software testers are mainly focussed on the second risk category, which can also be divided into “business” and “ICT”-related product risks. In order to give a better understanding of these risk categories, Table 1 shows some examples of risks belonging to each category.

Table 1: Examples of project and product risks categorised by business and ICT (Taken from PINKSTER et al. 2006, p. 18)

illustration not visible in this excerpt

3.3 What is software quality?

According to DIN 55350, quality is “the totality of features and characteristics of a product or activity which are related to its ability to fulfil given requirements” (DIN 55350, cited in RÄTZMANN 2004, p. 31, translated)

When applying this definition in software, software quality is the “totality of features and characteristic values of a software system which are related to its ability to fulfil defined or assumed requirements.” (SPILLNER & LINZ 2005, p. 251, translated). Following these definitions, it can be stated that all characteristics and features are part of the software quality and not only characteristics and features which were specified (RÄTZMANN 2004, p. 32).

Furthermore, according to ISO/IEC 9126-1, software quality is determined by the following quality attributes (cited in BALZERT 2009, pp. 468 - 471, translated):

- “Functionality: Ability of the software to provide functions in order to fulfil the specified requirements under defined conditions.”
- “Reliability: Ability of the software to maintain a specified level of operation when being used under defined conditions.”
- “Usability: Ability of the software of being understood and operated by the user. Ability of the software being learned by the user and being “attractive” when used under defined conditions.”
- “Efficiency: Ability of the software to maintain an appropriate level of operation under defined conditions regarding the used resources.”
- “Maintainability: Ability of the software to be changed. Changes can be corrections, improvements and adaptations to a changed environment, changed requirements and changed functional specifications.”
- “Transferability: ability of the software to be transferred from one environment to another environment. Another environment can be another organisational environment, another hardware environment or software environment.” The quality of a software product can be determined through the practice of software testing.

3.4 What is software testing?

Software testing is an important mean of minimising risks, because software testing gives information about existing problems and if the solution of a detected problem was successful or not. Therefore, it reduces uncertainty and helps to estimate risk (SPILLNER & LINZ 2005, pp. 185 - 186), as long as these risks belong to the category “product risks”.

According to SPILLNER & LINZ (2005, p. 8 - 9), software testing must be strictly discerned from debugging, which is done by the software developer in order to remove defects, while testing means that the test object is executed in a systematic and random way in order to detect error results and to determine the quality of a software product. Testing also means that the actual behaviour is compared with the specified behaviour. Part of the test process is not only the execution of the program, but also the planning of the test process and the interpretation of the test results.

4 Risk-Based Testing

Exhaustive software testing is often not possible in a finite timeframe and with reasonable costs. Planning the software testing process must therefore be selective (REDMILL 2004, p.3). In a risk-based testing approach, those tests are applied first which evaluate critical parts of the software. Through a risk-based test prioritisation, critical parts of the software are tested more intensively and earlier than other parts of the software inheriting a lower risk (SPILLNER & LINZ 2005, p. 186). The core question therefore is to find out which parts of the software are critical and must be given special attention within the test process.

According to PINKSTER et al. (2006), a structured risks management process in the context of software testing consists of the following steps:

- Risk identification
- Risk classification and assessment
- Risk allocation and selection
- Risk monitoring

Differing from the structure given by PINKSTER et al., the topics “risk identification” and “risk classification” should be discussed together during the next chapter.

4.1 Risk identification & classification

BACH (1999a) recommends two methods of risk identification (BACH uses the term “risk analysis”): An inside-out analysis and an outside-in analysis: The inside out analysis starts from the product and its components. BACH recommends to ask the following three questions for each component:

- “Vulnerabilities: What weaknesses or possible failures are there in this component?”
- “Threats: What inputs or situations could there be that might exploit a vulnerability and trigger a failure in this component?”
- “Victims: Who or what would be impacted by potential failures and how bad would that be?”

This way of analysing tries to answer the question which risks are connected with the different parts of the software product. It can go into technical details and requires therefore technical insight. Therefore, it should be done in cooperation with the software developer.

The outside-in analysis as presented by BACH goes the opposite way: A list of potential risks is created in advance and afterwards connected with the different parts of the product. For example, a risk list can be oriented on quality criteria (see also chapter 3.3.) in the sense that the question is asked what would happen of the requirements based on the quality criteria were not met. There can also be a generic risk list, for example:

- Which parts of the software are new?

- Which parts of the software are very complex?

The outside-in analysis also leads to the next step, which is risk classification. Similar to BACH’S second approach, PINKSTER et al. (2006, pp. 23 - 24) recommend to classify product risks which were already identified within the first step according to the quality attributes of ISO 9126 (see chapter 3.3). The authors make that clearer by applying this to software which is controlling an automated teller machine (see Table 2):

Table 2: Example of linking product risks to quality attributes for an automated teller machine (ATM) (Taken from PINKSTER et al. 2006, p. 24)

illustration not visible in this excerpt

Grouping possible risks around quality attributes can be a help for test stakeholders being part of the risk identification process to make sure that the list of risks is complete. Afterwards, a first arrangement of tests can be developed which is addressing each quality attribute (PINKSTER et al. 2006, pp. 24 - 25).

4.2 Risk assessment

After identifying risks, they must be assessed in order to define the test priorities. Risks can be assessed in a qualitative way, for example with the “MoSCoW” principle, which means that each product risk is assigned to the following classes: Must test, Should test, Could test, Wont test (PINKSTER et al. 2006, p. 25). With the help of these classes, the priority within the test process can be established relative to each other. By using the above mentioned formula amount of damage × probability of damage, risks can also be assessed in a quantitative way. Dealing with a mathematic formula means that both factors have to be given quantitative values in order to achieve an outcome which can be used to set the priorities for the test process. In most cases, quantification will be done with indicators and scales (such as “1” for low, “2” for medium, “3” for high) and not with monetary values. During this chapter, it will be discussed what must be considered in order to quantify both the amount of damage and probability of damage.

4.2.1 Quantifying amount of damage (according to REDMILL 2004, pp. 6 - 8)

Due to the fact that each system has a number of stakeholders, the consequence or the amount of damage of a failure is likely to be different for each stakeholder. For example, a bank may run a bank account system which displays the failure that customers are denied access to their accounts: The costs of such a failure, in other words the amount of damage, may be low from the perspective of bank operators and maintenance personnel. But if such a failure is published in press, it can lead to a loss of reputation, which may lead to diminished goodwill, a decline of market share and marketing costs in order to regain the company’s former market position. From the business perspective, such a failure inherits a huge amount of damage. It is also possible to estimate the possible amount of damage from the perspective of software developers: A certain number of failures may lead to the loss of opportunity to win another project within the same organisation. When estimating the consequences of a software failure, it must therefore be defined in advance from which perspective they are considered from.

Furthermore, the information regarding the amount of damage to which a failure may lead to must be gained from appropriate sources. Appropriate sources are for example the people who have defined the objectives of the systems, for example senior managers or system strategists.



ISBN (eBook)
ISBN (Buch)
438 KB
Institution / Hochschule
FOM Essen, Hochschule für Oekonomie & Management gemeinnützige GmbH, Hochschulleitung Essen früher Fachhochschule
risk risk-based testing




Titel: Risk management in software quality assurance