Risk-based testing in agile projects
Lecture 06
What is risk?
Copyright By PowCoder代写 加微信 powcoder
Why risk-based testing is necessary
Impact of failures
Probability of defects
Assessing quality risks in Agile Projects
Iterative lightweight quality risk analysis
Risk poker
User-story risk visualization
Validating risk charts
Testing according to risks
Estimating testing effort based on content and risk
Planning Poker
Play the game
“Possibility of a negative or undesirable outcome or event..”
Level of risk
Determined by two factors:
Larger impact higher risk
Higher probability of a negative event higher risk
Based on these factors,
The level of risk is the impact caused if the bad event could occur multiplied by the probability of the bad event occurring.
Why risk-based testing is necessary?
Testing effort increases exponentially if we strive to execute more and more tests by applying stronger testing criteria.
Example: 10 input fields each with 4 equivalence partitions. If we must cover all pairs of partitions across all possible pairs of fields, we need at least 16 tests, or 4 * 4. To cover all triples requires 64 tests or 4 *4 *4. To cover all combinations of partitions requires 1,048,576 test or 4 raised to the power 10.
Thus, the impact of failure is also exponential.
Testing is weak most of the bugs are found in production the cost of fixing (them) increases exponentially.
Based on this, an optimum can be reached, when we increase testing cost which in turn reduce failure cost.
For different same size user stories the failure cost is different.
It depends on the complexity of the user story.
More testing effort is needed for complex stories to reach the optimum.
If a story is more risky – either due to a higher probability of bugs, a higher criticality of the story or both, then stronger tests need to be applied.
The risks can be analysed and the necessary testing priority and effort can be determined prior to coding.
Failure costs differ for each story
Different testing methods in parallel
To avoid combinatorial explosions, rather than simply trying to increase the percentage of potential tests we run we use different testing methods in parallel.
In such cases, the testing cost with respect to the detected failures does not increase exponentially.
Different methods will detect different sets of bugs, though the sets do partially overlap.
Risk-Based Testing
Test techniques vary in thoroughness of testing.
Risk-based testing guides
the determination of test priority and allocation of test effort, and
the selection of test design techniques.
For riskier user stories, we must
use stronger coverage criteria,
and apply a wider range of test design techniques.
Impact of failures
Catastrophic
negligible
Catastrophic failure
A failure is catastrophic if the software stops working entirely.
Causes damage to people or environment.
Stops other programs or crashes the entire host system.
occur when:
one of the key features does not work or works badly, especially if no workaround is possible to restore the missing or erroneous functionality.
the failure happens frequently in a very important feature and only a very time-consuming work around resolves the problem.
some data is lost or corrupted and restoring the data is very time-consuming.
Critical failure
Marginal Failure
A failure is marginal
if the user can solve the problem by a fairly quick and simple workaround and also
if some data is lost but can be restored easily.
Negligible Failure
If the problem does not affect functionality, but just makes the product less appealing to the user.
Impact classification
Parts of the code that are almost always executed after the software is started.
Part of the product code that most users are using frequently, but may be not during each usage session.
OCCASIONAL
An area of the code that an average user may never visit, but is needed for a more serious or experienced user occasionally.
An area of code that most users never use and which is used only if customers do very unusual activities.
During quality risk analysis, we should predict where we expect the code to contain many, moderate, few or almost no defects.
Probability of defects or likelihood of defects
Complexity
New technology, methods and tools
Changed Code
Time Pressure
New Team members
What should we consider when trying to predict the probability of defects.
Iterative lightweight quality risk analysis
Risk Poker
User Story Risk Visualisation
Validating Risk Charts
Testing according to Risks
Assessing quality risks in agile projects
Involves: business stakeholders, technical stakeholders and developers.
The discussion is about what worries the stakeholders from quality point of view.
The specific quality risks are identified and assessed during iteration planning.
Iterative lightweight quality risk analysis
Iterative lightweight quality risk analysis
Identify quality risks
Assess level of risk
Estimate test effort
Mitigate risks through test design, implementation, and execution
Quality risk analysis template
A whole-team approach to achieve optimal balance between
sufficient quality and acceptable risk, and
the available constraints in time and resources.
Scrum Master organises the risk poker session, but will not participate in the game. However, the product owner will participate.
The product owner represents all the stakeholders and will be able to provide necessary input, most importantly about the impact.
The question for each backlog item is:
What level of risk should we associate with this user story?
Risk Poker
We consider only two factors:
Probability
Some risks are not in user story level. For example, performance, security, or usability testing.
Risk Poker
During the game every player has four coloured cards, green, yellow, orange and red.
In order of increasing risk (green is least risky and red is most risky)
The colours represent the estimated risk level for both impact and probability.
Impact is estimated first and then likelihood.
Risk poker
How to risk poker?
The risk points are calculated as the product of the two scores.
For example, if 4 members had orange, orange, orange and yellow and we have 3 associated with orange and 2 with yellow, then the risk score, is 2 *3 = 6
Risk Score
When all the risk estimations of user stories are available, they can be visualised so they can be seen in relation to each other.
useful to see the big picture of risks.
Easy comparison.
Important to compare risks estimated in different meanings.
Sheet of paper
Excel Sheet
User story risk visualization
Risk chart example
Agile team validates the risk charts.
Two methods:
Risk scores using histograms – which can be helpful during release plan but will not work during iteration planning (since there are not enough stories)
Establish appropriate exemplars— which are user stories with agreed upon risk ratings.
One exemplar for each impact and likelihood rating, which means four-five exemplars for both (impact, and likelihood depending on scale 1-4 or 5
Validating risk charts
Story risk Test Technique
Green 1-4 Exploratory Testing
Yellow 5-8 Exploratory Testing, defect prevention (Agile testing SBE or ATDD)
Orange 9-12 Exploratory Testing, defect prevention, test design Automated or systematic based on test selection criteria, test automation, normal code coverage such as decision coverage.
Red > 12 Exploratory Testing, defect prevention, test design Automated or systematic based on test selection criteria, test automation, review or static analysis, normal code coverage such as modified decision coverage or breaking the user story into smaller ones.
Testing according to risks
Automated test design : Model-based testing
Systematic test design: EP, BV testing
Extents of testing:
Risk based Test methods selection
Opportunity
Report Bugs
After getting risk scores for each user story, the next step is to estimate testing effort.
In Agile methods, testing effort is estimated as part of the overall implementation effort, based on risk and the size of the user story.
Estimating testing effort based on content and risk
Agile assessment tool- Poker
Planning poker – example
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com