Risk Based Testing 102
Author: Unnati Bajpai
Introduction: Risk-based testing (RBT) is a methodology that can be described as “prioritization of various requirements on the basis of the risk they carry and testing them according to the derived priority.”
In this approach, testing is prioritized in terms of the probability that some feature of the program will fail and the estimated cost of failure. The greater the probability of an expensive failure, the more important it is to test that feature as early as possible and as carefully as possible.
What is Risk and how it impacts software projects: Risk, in general term, can be defined as an uncertain damage or loss that is probable to occur in future. To be more specific “Risk is probability or threat of a damage, injury, liability, loss, or other negative occurrence, caused by external or internal vulnerabilities and that may be neutralized through pre-mediated action.”
Now let us understand “risk” in terms of software development. More or less, the Software Development Process has three major criteria or limiting factors, which are considered for defining success of a project:
- Budget
- Time
- Requirements
A successful software project must satisfy all user requirements within estimated time and budget. Any uncertainty or possibility of loss may result in non conformance of any of these key factors, leading to overtime / over-budget or poor quality project.Software risks, which impact above 3 key factors, can be broadly categorized as:
- Requirement Risks: Lack of requirement analysis, requirements that are not in-line with customers’ needs, poorly defined requirements, ambiguous requirements, inadequate requirements, invalid or impossible requirements etc.
- Technical Risks: Complexity of architecture, technology change, configuration change, inadequate technical support/ knowledge, lack of domain knowledge etc.
- Schedule Risks: Insufficient budget, unrealistic time line, inadequate skilled resources, improper resource planning etc.
Software projects are always full of these risks and uncertainties. The risk factors affect projects in different ways. Schedules often slip when a risk realizes and these slips can have serious consequences such as increased project cost, changed scope, and compromised quality. The impact of these risks is especially severe for companies focused on time-to-market. In throat cutting competition, missing a market window by short period of time can destroy market share and hurt profitability. For medium to large sized companies, it could have negative long-term strategic consequences. For startup - small sized companies; one missed opportunity may turn to closing their doors. It could also impact the internal operations of a company by lowering moral and productivity.
Why RBT is needed: As explained in the above section, we can term risk as an unwanted event that has unfavorable or negative consequences. In order to negate them or make them less severe, there occurs a need to identify various dimensions of software project risk and to build and test a model guided by theory that relates the risk dimensions to project performance. This gave rise to the need of Risk Based Testing.
Potential reason for adopting RBT can be the aggregated impact of limited resources – time, budget and human. On one hand, the project’s calendar doesn’t allow sufficient time for a thorough testing of all functions and on the other hand budget limits the number of skilled human / software resources. In such a scenario, test coverage of all minute part of application would not be possible. To overcome or balance this situation, focus should be put on those areas that represent the largest risk, if a fault occurred.
Building Blocks: Before we start discussing Risk Based Testing process, we must understand few basic concepts that altogether create a base for RBT.
1. The Pareto Principle:
The Pareto principle, (also known as the 80-20 rule), states that, “for many events, roughly 80% of the effects come from 20% of the causes. “ This is a generic principle that can be applied to almost all aspects of life (definitely the ratio can vary). For business - "80% of sales come from 20% of your clients.” For economics - “80% of the wealth in a country belongs to 20% of the population.”
1.1 Pareto principle in relation with Software Testing :
“In software, roughly 80% of the defects come from 20% of the LOC (Lines of code).
Or
“Usually 20% of the total features of an application will allow user to do 80% of his work”
This principle allows testers to concentrate their testing effort in this specific area (20%) where there are chances to get most of the deviations (defects). This is the best guideline to achieve respectable testing coverage within limited time, budget and resources.
Figure.1
The above chart describes the relation between Risk and Testing effort: where most of the problem has been discovered within first 20% of testing effort. To make it real, test scenarios, those cover the highest risks, should be given preferential treatment. The testers should first test what's most important and if time permits, less important features can be tested.
2. Quality Attributes: Software quality attributes can be considered as the benchmarks that describe application’s intended behavior within the configuration / environment for which it was built. The quality attributes provide the measurement of the fitness and suitability of a product. Below, I have mentioned a brief introduction of various quality attributes:
2.1 Functionality - A set of attributes that bear on the existence of a set of functions and their specified properties. The functions are those that satisfy stated or implied needs such as “Suitability”, “Accuracy”, “Interoperability”, “Compliance”, “Security”.
2.2 Reliability - A set of attributes that bear on the capability of software to maintain its level of performance under stated conditions for a stated period of time – “Maturity”, “Recoverability “, “Fault Tolerance “.
2.3 Usability - A set of attributes that bear on the effort needed for use, and on the individual assessment of such use, by a stated or implied set of users :- “Learn-ability”, “Understandability” , “Operability”.
2.4 Efficiency - A set of attributes that bear on the relationship between the level of performance of the software and the amount of resources used, under stated conditions: “Time Behavior” and “Resource Behavior”.
2.5 Maintainability - A set of attributes that bear on the effort needed to make specified modifications: “Stability “, “Analyzability”, “Changeability”, “Testability”.
2.6 Portability - A set of attributes that bear on the ability of software to be transferred from one environment to another: “Install-ability”, “Conformance”, “Replace-ability”, “Adaptability “
3. Prioritizing Requirements: Priority can be defined as “The state of being superior or ahead of something else”. When we talk about requirements prioritization, then we are setting the importance of a requirement in comparison with another requirement. We do it for determining the list of which candidate requirements of software product is superior to others.
For small projects, the stakeholders may agree on requirement priorities informally but larger contentious projects need a more structured approach, which removes some of the emotion, politics, and guesswork from prioritization. There are several prioritization techniques such as Relative Priority, Systematic Priority, Cost – Value approach, Quality Function Deployment, Binary Search Tree, MoSCoW method and many others. Explaining each of them is out of scope for this course, however I would like to put some light on MoSCoW technique( since this will be used in later sections of the course).
3.1 The MoSCoW Technique:
In this technique, all the stakeholders of the application (Client, Project manager, Business Analyst etc.) reach a common understanding for the importance (priority) they place on each requirement.
The MoSCoW stands for:
- M - MUST have this / Must Test this.
- S - SHOULD have this / Should test this.
- C - COULD have this / Could Test this
- W - WON'T have this time / Won’t test this time.
Here, requirements are categorized on above four criteria. A Must have requirement is one, which has highest priority and that must be included in the delivery. A Should have Requirement is one, which is also critically needed for the application but can be implemented in later releases. Could have requirements are good to have features, which are less critical. Won’t have requirements are least critical, and are considered inappropriate for this delivery. These may be considered in later releases if time and budget permits.
4. Quality Criteria: Quality criteria are an agreement between client and the vendor which states - what kind of defects in which numbers will be accepted by the client. To make a release to be possible, the application must satisfy the agreed upon quality criteria. For example: if quality criteria of a project is set to (Critical - 0, Major - 0, Minor- 10 and Cosmetic – 50) than it means that, the “to be delivered project” must not have any Critical OR Major priority issues, however it may have maximum 10 Minor and Maximum 50 Cosmetic issues. The Quality criteria can be as ambitious as it can be considered to be a Zero defect delivery.
The Risk Based Test Methodology:
Since we have already discussed the building blocks for RBT, now we have to co-relate them to gain understanding of the RBT methodology. In a scenario where we have huge business requirements, short timelines, limited resources then we have to make sure that our testing must cover the most essential functions. Establishing each chunk of functionalities relative importance lets us sequence our construction of business and test scenarios to provide the greatest quality at the lowest cost.
The RBT process can be described by these major steps:
- Step1 - Define all requirements in terms of Risk associated to them
- Step2 - Based on risk assessment, prioritize the requirements
- Step3 - Plan and define tests according to requirement prioritization
- Step4 - Execute test according to prioritization and acceptance criteria.
Here we will see how the concepts, discussed in earlier chapters, ease the process of defining test strategy for a project having tight test schedule and budget with lots of requirements to be tested. The first two steps are common for development stream and test stream, however Step 3 and 4 are specifically dedicated to testing.
- Step 1 – Various risks associated with the applications are identified by the stakeholders of the project. The stake holders here should be a mix of business and technical team – that means, involving people from various departments and with various backgrounds: for example, the client, customers, business experts, technical experts, project manager, project Leader, users, developers and infrastructure representative. The risks can be identified in a number of ways whichever is considered suitable to the stakeholders:
o By using past experience
o Checklists of recurrent risks
o By holding workshop
The “impact analysis” for each identified risk is done in order to understand the consequences.
- Step 2 – Once all the possible risks and their impacts are analyzed, then project manager has to get the requirements prioritized. To achieve this- combination of Pareto Principle and various prioritization techniques (discussed above) are used by the stakeholders. Priority of the requirements should be agreed upon and should be updated in functional requirement document; the same should also be conveyed to the development and the test teams.
- Step 3 – Once we have the Requirements with priority tagged with them, test activities should be started keeping the priority of the requirements in mind. Business scenarios and test scenarios for the “must have” requirements must be derived first and then for “Should”, “Could” and for “Won’t have”, if time permits. Here while the test scenarios are being written, we must set priority to each test scenario. This priority can be set with the common understanding of tester and test lead/ manager. Each test case should have its priority – “Must test”, “Should test”, Could test”, “Won’t Test”- applying Pareto principle (here I have used MoSCoW priority just for example).
Wondering, how to have a “Could test” priority test case for a “Must have” requirement? Here pops the “quality attributes” & “quality criteria” in to picture. We have to consider which Quality attributes are in scope of testing and their respective weightage with respect to Quality Criteria set for the release. Test cases for the quality attributes with high weightage and having minimum limit in quality criteria should be given high priority and vise vice.
Figure – 2 - Requirement and Test Specification Diagram.
For Example:
o Case 1 - if Functional quality attribute has high weightage and quality criteria quotes Critical functionality – 0 defects, then this means that the application should be tested for functionality having no critical functional issues. Hence test cases representing this criterion should have “Must test” priority. A functional test case which covers the navigation or User Interface requirement will have “Could test” priority.
o Case 2 - If “Usability” quality attribute has high weightage and quality criteria quotes 0 Cosmetic – defects (Most unlikely case), then this means that the application should be tested for usability having no usability/cosmetic/ GUI issues. Hence test cases representing this criterion should have “Must test” priority.
- Step 4: If any of the identified risk realizes by this time (When test execution is scheduled), then there is quite good chances of schedule slippage from development side. In this case, since final deadline to customer can’t be changed, obviously the “Test execution Schedule” would be shrunken. In such scenario, Test manager again has to apply the Pareto principle and has to finalize the scope of testing in the reduced timeline that will ensure least risk and highest quality. Test scenarios/ Cases that have “Must test” priority would be executed first to ensure basic Quality criteria are met. Then if time permits, test cases with “Should test” priority would be executed and so on. Concept here is to test the most important features first, ensuring the bare minimum quality requirements, and then if time and budget permits then widening the scope of testing to lesser important requirements.
Test Techniques Helpful for carrying RBT:
Test techniques that have specific purpose of capturing issues from most error prone areas are the best suited for RBT:
- Boundary Value analysis
- Equivalence partitioning
- Decision tables
- Path Flow testing
- Some amount of exploratory / Experience based testing
- Others, based on type of testing need to be performed
Risk Based Test Reporting:
Risk based Test reporting provides all the Stakeholders to understand the Project’s current status in a better and more understandable way. Test manger produces a Test report that includes Rick coverage matrix, based on risks identified (discussed earlier). The main objective of risk based test reporting is to be able to display the test coverage of risks, i.e - which of the identified risks have been tested so far and which of them have not yet been.
(Figure from: www.cs.tut.fi/tapahtumat/testaus04/schaefer.pdf)
These reports basically help in decision making for the release. If the release is made today, then what are the identified risks which might realize, since they are not yet addressed by the test team? At the same time – how safer are we, i.e – how many risks are covered by testing?
Advantages of RBT:
The Most important advantages of this approach are as follows:
- Linking the product risks to the requirements identifies gaps.
- With limited time, money and qualified resources, testing concentrates on the most important matters first, thus delivering the most optimal test.
- The focus lies on risks to the business instead of just the functionality of the information system.
- During testing, communication (Test Reporting) takes place in a language (risks) that all stakeholders understand, hence involvement of testing is greater in comparison to mere list of issues communicated.
- It offers a negotiating instrument to client and test manager alike when available means are limited.
References:
http://testmythoughts.blogspot.com/2007/11/risk-driven-testing-risk-based-testing.html
http://testausosy.ttlry.fi/webfm_send/120/
www.en.wikipedia.org/wiki/Pareto_principle
www.findarticles.com/p/articles/mi_qa3713/is.../ai_n9398985/
