An Analytical Approach to Setting Severity of Bugs
Setting up correct severity and priority of reported bugs is a major action point in our Software Testing Life Cycle (STLC). It is of utmost importance because depending on the severity and priority level the order of bug fixing is planned. Usually in an SDLC environment, testers or QA Engineers assign the severity of a bug while reporting it in a bug tracking tool and the Product Owner or someone from the Project Leadership analyzes the bug and sets the priority.
Lets see the generalized definition of these two terms:
Bug severity means the impact the bug has on the application under test (AUT). It is associated with determining the complexity of this issue, how long it might take to fix this issue, when this issue should be fixed for release etc.
Bug priority is directly associated with when and in which order the bug should be fixed and released. Usually, a defect that has severe impact on the system functionality is given greater priority.
Depending on the impact criteria of a bug and the order of fixing a bug, we can categorize the severity and priority levels in various types. Lets visualize the categories as below:
Pictorial 1: Different levels of Severity and Priority
In this article we will try to establish a relationship matrix to determine bug severity based on the priority of a bug and discuss an analytical approach to calculating the severity level of a bug. Even though setting the bug priority is not the responsibility of a tester, testers should possess clear idea about the probable priority of a bug. Again, based on the impact level of a particular bug it is relatively easy to guess the priority.
Severity vs Priority Relationship Matrix
In order to establish a relationship matrix, we need to relate severity and priority based on the urgency of the fixing of the issue. Based on the fixing urgency and the impact on the system we can create a relationship matrix as below:
Pictorial 2: Severity vs Priority Relationship Matrix
In this matrix we have related the six levels of severity with the four levels of priority. Based on the priority and impact of the bug, we have assigned severity levels.
Our Analytical Approach
Our analytical approach to setting bug severity is completely based on satisfying the conditions of aforementioned matrix. In order to determine which condition a bug fulfills, we will focus on three questions. They are:
- What is the impact of the issue?
- What is the category of the issue?
- When this issue should be fixed and released?
In our STLC typically these questions have some fixed options.
Pictorial 3: Options of Impact Question
Pictorial 4: Options of Category Question
Pictorial 5: Options of Release Question
When a bug is found our task will be to simply get the answers of these three questions and find out the combined outcome of our answers. We can then map the outcome with our matrix and determine the severity level. Lets discuss each of the severity levels with sample example.
The bugs which have the most significant/critical impact on the system will be assigned as BLOCKER. Significant/critical impact means that these issues block the further flow in the system, block the corresponding feature or block a related feature. Usually these features need to be prioritized to be fixed as soon as possible. Hence they are of critical urgency to be fixed and released immediately or within the current release cycle. As real life example we can mention if a feature like user registration or login is non-functional, the corresponding bug will carry blocker severity.
Pictorial 6: A Sample Flow of BLOCKER Severity Bug
HIGHEST severity issues do not block any feature flow but they hamper our further testing. In a sense these bugs block our test execution in deeper flows. These bugs can have extensive or significant impact but are not as urgent as blocker bugs to be fixed and released immediately. In case a bug has been reported from client side or a bug has been found by live end users which does not belong to blocker category, must be assigned as highest severity.
Pictorial 7: A Sample Flow of HIGHEST Severity Bug
If a bug does not meet the complete requirement or the implementation cannot be verified against the SRS, they usually have moderate or large impact in the system. The type of bugs which causes the system not to behave as it is supposed to or generates wrong error message in a negative test scenario or use case, lie in this category. These bugs can have medium or high urgency to be fixed and release within the current cycle or sprint. These types of bugs belong to high severity category.
Pictorial 8: A Sample Flow of HIGH Severity Bug
In our STLC we regularly face some bugs which occur in non-functional items like placeholders, labels etc. or in UI action flows. These bugs can occur due to incomplete feature release or due to confusion in requirements. These issues usually have moderate impact on the system. These issues usually lie in category medium. One universal example of severity vs. priority is that if company logo is wrong it is of high priority. These type of issues will also belong to medium severity.
Pictorial 9: A Sample Flow of MEDIUM Severity Bug
We can encounter lots of bugs which occur in functional and non-functional flow and they do not have large impact on the system. They usually are of less urgency in the pecking order. These bugs having minor and insignificant impact on the system but should be fixed before the system is rolled out in production can be categorized as of low severity.
Pictorial 10: A Sample Flow of LOW Severity Bug
Bugs which have the properties of a low severity bug and can be deferred in a later release or can be marked as a bug which will not have much significant impact if it is not fixed before production roll out, can be assigned as of lowest category. Typically in the whole SDLC, the number of lowest severity bugs will be lowest.
Pictorial 11: A Sample Flow of LOWEST Severity Bug
Severity Calculator Application
We have built a sample severity calculator application to calculate bug severity. It is a simple Java GUI application. You can find the application here.
To run this application, go to the source folder where the jar has been downloaded and open command prompt and run the following command:
java -jar severity-calculator-1.0.jar
This application has a UI like this:
This application has a simple user interface containing the three questions and the corresponding options. Selecting the answers to the questions will calculate and display the severity. The calculation logic has been derived simply to satisfy the matrix established in this article. Answering all the questions is mandatory. After selecting all the options, clicking on the Calculate button will show the calculated severity and your selected answers.
Our next target is to improve the UI of this application. To improve the user experience and the overall logical operation of calculating severity, interested people are invited to collaborate on this project. The project can be found on this github link.
P.S: In this article, we have deliberately avoided the discussion of severity vs. priority. Lots of resourceful articles can be found online on this topic.
If you want to catch a detailed writeup on bug lifecycle and bug reporting, check out this article.