Bug Reporting 102
Authors: Santhosh Tuppad & Pradeep Soundararajan
Reviewer: uTest
Introduction: One way that we often identify a good tester is by reviewing the quality of his or her bug reports. A lot of credibility is at stake for a tester in the process and output of bug reporting. We have witnessed the respect for testers improves or falls apart based on this criteria, so we hope to convey the most important points of bug reporting here.
In this course, we share the practices of reporting bugs that are of great value to the most important stakeholders. Our intention is to help the tester community: to help you learn a few things that will likely boost your credibility or re-iterate what you maybe already know (but perhaps do not practice consistently). If you already practice these on a regular basis, kudos to you!
Audience of your bug report & understanding their point of view
Bug reports often touch multiple audiences. In this section we explore who the audiences are and what's important to them.
Programmers: It is usually obvious for testers that their target audience is developers. However, most testers pay little attention to understanding what the developers' needs are. Testers who were once programmers might understand what a developer is looking for when reviewing a bug report, but that doesn’t necessarily mean that these testers are likely to report bugs in a way that caters to this audience. Ultimately, it comes down to who has acquired and practiced these skills.
A programmer might look for information to:
- Reproduce a bug
- Be able to understand the bug in a different view than reported
- Perform an investigation of what is causing the defect
- Look for information about the bug and its co-relation to other bugs reported
- Understand which part of code to touch to fix the bug
Co-testers: Although testers may be on the same team, they could be split across different continents and time zones. We often come across situations where a tester - different from the one who reported the bug - has been asked to further investigate the bug or test for the fix. Therefore, a poorly written bug report could misguide a co-tester from the same testing team.
A co-tester might look for information to:
- Reproduce a bug in the same or different environment
- Add more investigation notes to the bug
- Provide additional information when it is deferred or rejected or even otherwise
- Respond to a developer’s comment on the bug
Test/Dev/Product Manager/Customers: This segment of the audience is mostly decision-makers or those who influence them. These people usually are in a situation where they don’t have time to read an entire bug report before making a decision. Finding and reporting a bug is one part - making it useful to this audience is another.
A test/dev/product manager might look for information to:
- Make a decision of adding / not adding a specific module to the upcoming release
- Make a decision of ship/no ship of the entire product
- Understand how much more development / bug fixing work is needed
- Plan a future release
- Help the customer plan releases to their customers
Bug report elements and their significance
There are many elements that constitute the completeness of a bug report. This section introduces the most important ones and those that many testers struggle with.
Summary or Subject: The purpose of a summary or subject line is to convey the nature of the bug in as few words as possible (without being ambiguous). Those who review and triage bugs may need to quickly assess whether a bug needs to be prioritized for immediate fixes. A developer might want to learn if the bug pertains to the code he or she has written. A manager might want to know if the bug is in a specific module to which a release is being planned. Writing a concise yet meaningful bug report is indeed an important skill.
Some organizations have begun standardizing bug report subject lines. The driver behind this is to help various stakeholders better understand the bugs at first glance. After all, we cannot expect all testers to naturally write the summary or bug report subject in similar fashion. A common approach to standardization has been to include: 1) the area [in the application] in which the defect occurs followed by a hyphen (“-“) and 2) a brief description of the bug.
Description or Actions Performed: This section should not simply be a copy-and-paste from the test case (provided one exists). It could contain information such as the action items performed and what was observed. Testers should strive to be context-driven in this section.
Some organizations use bug reporting templates that have limited flexibility in terms of the options and fields for their respective bug tracking systems. However, a tester can report relevant and important information about a bug in the Description section. This may include: risk to the user, cost of not fixing the bug, how the bug could impact business and other items related to bug advocacy.
Test Setup & Test Environment: There may not be a dedicated section called “Test Setup” or “Test Environment” in most bug reporting templates and tracking systems, but this is certainly important – not just for you but for other audiences. Setup information can be provided as an additional note in the Description section. Information on Setup and Environment could involve configuring the system, making hardware level changes, disabling or enabling something relevant to the test or bug being reported.
Severity: There must be at least half a million pages on the internet discussing Severity and Priority. Instead of adding to the debate, the only advice here is to be “reasonable.” Leave the priority to the business people and decision-makers. Explain why something is of high/medium severity in situations where it may not be obvious to your audience. Lastly, be open to understanding what others think.
Drafting and Publishing Bug Reports
There are a few things a tester should consider when a bug is found, investigated and ready to start drafting a report.
Looking for Duplicates
Why spend time on writing a bug report that is already written? If a tester finds a colleague or a fellow tester on the project has already reported the bug, then the strategy shifts from reporting to appending information on the already reported bug. This enhances the importance of the bug that is reported and ensures the management has lesser time dealing with duplicates during triage. This is a way of respecting the time of your audience.
Use auto spell and grammar check
Spell and Grammar check is extremely important to those testers whose English is of the second language. If you speak, write and read good English, no matter where you are born, English is one of those first languages for you.
Spell and Grammar checks are not fool proof. One of the examples we have encountered in a real time project is that of a report from a tester which reads like this, “X module feature is miss spelled”. You notice misspelled and miss spelled are meaningful words for the software although miss spelled means different thing to us from misspelled.
Use screenshots
Screenshots help most people understand a bug quickly. To help the audiences of a bug report, you should add screenshots to your bug reports. Using JPG or PNG formats is optimal. As an added bonus, testers may also include text and callouts in the screenshot itself using programs such as Jing.
Use video for long steps
Some bugs are hard to describe or there could be language barriers between the test and dev teams. Usage of video recording of the bug is getting increasingly popular. The only danger associated with it is that – only few testers know for what bugs a video shall be helpful. It looks obvious, but it is not. When steps to reproduce are lengthy and hard to explain, usage of a video is wise. If the dev team is from a different country whose English is as bad as that of the test team (J) then it is a wise idea to have a video of a bug that needs a lot of explanation. While videos could be of help to the developer audience, it could be a pain for others. So, a short description of the problem coupled with a video is the wisest choice we have made so far. We don’t suspect that you could be wiser than us.
Attaching document(s) of error logs
If the program logs actions and interactions of the system then it is likely integrated to be of help in bug fixing times. The log file could be huge and a tester who is context driven would copy paste only the relevant info in a text file and attach the same to the bug report.
Usage of Compression Tools
Screenshots, Videos, Log files and other relevant files to a bug could eat up a lot of space and is a trouble for the audience to download each and every single file attached to a specific bug. Usage of compression tools, such as WinZip or 7Zip can help in such situations. The worst tester would copy a single line of log; make a txt file and then zip the file. Well, it happened.
Naming convention for screenshots, videos, and documents
While naming image file or video file or document file, it is wise to follow a naming convention that goes well with audience. This helps in organizing and tracking these files easily. A worst tester would have a file crash.jpg and a good one would be likely to have something like Bug_id_Product_module_feature_typeofproblem.jpg
Checklist of publishing a bug report:
- Look for duplicates
- Check for meaningfulness of entries in all Elements of the bug report
- Try modeling as different audience and ask if the bug report is really useful to them
- Make a list of mistakes in the past with bug reporting and run it through the report
- Check for attachments, their sizes and relevancy to the context
- Save a copy of your bug report in MS Word or other editors as everyone knows bug tracking systems, crash, too.
- Finding bugs in the bug report and fix them
