Risk Based Testing 103
Author: Unnati Bajpai
Introduction: This course presents the risk based test methodology and tips on techniques and reporting.
The Risk Based Test Methodology: Since we have already discussed the building blocks for RBT, now we have to co-relate them to gain understanding of the RBT methodology. In a scenario where we have huge business requirements, short timelines, limited resources then we have to make sure that our testing must cover the most essential functions. Establishing each chunk of functionalities relative importance lets us sequence our construction of business and test scenarios to provide the greatest quality at the lowest cost.
The RBT process can be described by these major steps:
- Step1 - Define all requirements in terms of Risk associated to them
- Step2 - Based on risk assessment, prioritize the requirements
- Step3 - Plan and define tests according to requirement prioritization
- Step4 - Execute test according to prioritization and acceptance criteria.
Here we will see how the concepts, discussed in earlier chapters, ease the process of defining test strategy for a project having tight test schedule and budget with lots of requirements to be tested. The first two steps are common for development stream and test stream, however Step 3 and 4 are specifically dedicated to testing.
- Step 1 – Various risks associated with the applications are identified by the stakeholders of the project. The stake holders here should be a mix of business and technical team – that means, involving people from various departments and with various backgrounds: for example, the client, customers, business experts, technical experts, project manager, project Leader, users, developers and infrastructure representative. The risks can be identified in a number of ways whichever is considered suitable to the stakeholders:
- By using past experience
- Checklists of recurrent risks
- By holding workshop
The “impact analysis” for each identified risk is done in order to understand the consequences.
- Step 2 – Once all the possible risks and their impacts are analyzed, then project manager has to get the requirements prioritized. To achieve this- combination of Pareto Principle and various prioritization techniques (discussed above) are used by the stakeholders. Priority of the requirements should be agreed upon and should be updated in functional requirement document; the same should also be conveyed to the development and the test teams.
- Step 3 – Once we have the Requirements with priority tagged with them, test activities should be started keeping the priority of the requirements in mind. Business scenarios and test scenarios for the “must have” requirements must be derived first and then for “Should”, “Could” and for “Won’t have”, if time permits. Here while the test scenarios are being written, we must set priority to each test scenario. This priority can be set with the common understanding of tester and test lead/ manager. Each test case should have its priority – “Must test”, “Should test”, Could test”, “Won’t Test”- applying Pareto principle (here I have used MoSCoW priority just for example).
Wondering, how to have a “Could test” priority test case for a “Must have” requirement? Here pops the “quality attributes” & “quality criteria” in to picture. We have to consider which Quality attributes are in scope of testing and their respective weightage with respect to Quality Criteria set for the release. Test cases for the quality attributes with high weightage and having minimum limit in quality criteria should be given high priority and vise vice.
Figure – 2 - Requirement and Test Specification Diagram.
- Case 1 - if Functional quality attribute has high weightage and quality criteria quotes Critical functionality – 0 defects, then this means that the application should be tested for functionality having no critical functional issues. Hence test cases representing this criterion should have “Must test” priority. A functional test case which covers the navigation or User Interface requirement will have “Could test” priority.
- Case 2 - If “Usability” quality attribute has high weightage and quality criteria quotes 0 Cosmetic – defects (Most unlikely case), then this means that the application should be tested for usability having no usability/cosmetic/ GUI issues. Hence test cases representing this criterion should have “Must test” priority.
- Step 4: If any of the identified risk realizes by this time (When test execution is scheduled), then there is quite good chances of schedule slippage from development side. In this case, since final deadline to customer can’t be changed, obviously the “Test execution Schedule” would be shrunken. In such scenario, Test manager again has to apply the Pareto principle and has to finalize the scope of testing in the reduced timeline that will ensure least risk and highest quality. Test scenarios/ Cases that have “Must test” priority would be executed first to ensure basic Quality criteria are met. Then if time permits, test cases with “Should test” priority would be executed and so on. Concept here is to test the most important features first, ensuring the bare minimum quality requirements, and then if time and budget permits then widening the scope of testing to lesser important requirements.
Test Techniques Helpful for carrying RBT:
Test techniques that have specific purpose of capturing issues from most error prone areas are the best suited for RBT:
- Boundary Value analysis
- Equivalence partitioning
- Decision tables
- Path Flow testing
- Some amount of exploratory / Experience based testing
- Others, based on type of testing need to be performed
Risk Based Test Reporting:
Risk based Test reporting provides all the Stakeholders to understand the Project’s current status in a better and more understandable way. Test manger produces a Test report that includes Rick coverage matrix, based on risks identified (discussed earlier). The main objective of risk based test reporting is to be able to display the test coverage of risks, i.e - which of the identified risks have been tested so far and which of them have not yet been.
(Figure from: www.cs.tut.fi/tapahtumat/testaus04/schaefer.pdf)
These reports basically help in decision making for the release. If the release is made today, then what are the identified risks which might realize, since they are not yet addressed by the test team? At the same time – how safer are we, i.e – how many risks are covered by testing?
Advantages of RBT:
The Most important advantages of this approach are as follows:
- Linking the product risks to the requirements identifies gaps.
- With limited time, money and qualified resources, testing concentrates on the most important matters first, thus delivering the most optimal test.
- The focus lies on risks to the business instead of just the functionality of the information system.
- During testing, communication (Test Reporting) takes place in a language (risks) that all stakeholders understand, hence involvement of testing is greater in comparison to mere list of issues communicated.
- It offers a negotiating instrument to client and test manager alike when available means are limited.