When is a bug not a bug?

Working as Designed? Or Not Designed as It Should Work? Understanding when a bug is not a bug.

One of the hardest things for new testers to know is where the line between a valid bug stops and good feedback begins. This can be tricky at times even for experienced testers. Sometimes what one developer acknowledges as less than optimal and approves, another rejects as a “WAD” “works as designed.”

A bug is simply defined as, “Something not working as designed.” Too often though new testers fall into the trap of reporting, “Not designed as it should work.” The former will surely mean a payout, the later may be rejected depending on how valuable the client feels that input is. While they sound similar the distinction is important. One has found something broken; the other has found something not optimal. Even farther away from certainty is reporting “Not designed how I like it.” While you may have some valid thoughts on how things should work, every product has stages where different types of input are needed.

Here are a few examples all from the same page:

This above finding is accepted. It is clear the back button was intended to work and this is something Not Working as Designed.

This is a much more gray issue:

The search engine clearly lacks sophistication to handle a single letter at the end missing (Bryan should be Bryant).But it is not broken, it is just poorly designed.

The search engine is working if you type in the right name. It may be approved based on what the customer said was important to focus on in the Scope and Instructions.

Here is a third issue raised from the same search engine – nothing has been entered into the search field and it has been submitted.

This one will likely be rejected as “Works as Designed.” There is no negative outcome and it does not interfere with the user interface. This also is the same behavior that will be produced on most major search engines, with the page being re-served for a chance to retry your entry. The tester is making suggestions based on “How they would design it.” This is called a UX bug – for “user expectation” – and differs from a UI (User Interface) Bug which prevents something on the website from being used properly.

Each client will come to the table wanting a specific level of input. The scope and Instructions are designed to spell that out and to let you know the key areas important to test.

Let’s try a couple more. This is listed as the client’s focus:

“Please make sure that all the links and functionality of the website are working by testing as a regular user. Let us know if anything interferes with the user experience on the site.”
The tester submits the following:

While the tester has found a valid bug, they have not found anything useful to the developers’ instructions, which were to “test like a regular user.” This type of testing is called negative testing or limit testing.

In this situation a user was playing around and entered a couple hundred characters to see the reaction. There would be no user expectation of any sort and the page did not crash. Although the finding is true, it will be rejected as it lacks any practical reason for spending resources to fix it.

Some cycles you will be encouraged to try and break the product. It is important to understand that field verification is usually only useful where it creates a significant negative outcome. Something like a phone number that can’t be used later on because there were too few digits or when it causes a dramatic malfunction like the page crashes or freezes.

Now consider this one:

Will it be accepted or rejected?

Keep in mind:

  • Nothing is broken
  • The schedule displayed when requested
  • The date is accurate
  • It does not affect the user’s ability to use the site.
  • The focus was functionality and user interface.

While it is true the user has found something not optimal, the empty line on the schedule did not interfere with the user experience or functionality.

Under another cycle where the customer focus is the entire look and feel of the site it might have be approved. That is why it can’t be reiterated enough that you need to read scope and instructions thoroughly to understand the purpose of the cycle. Reading through the approved and rejected bug reports will also help you understand the customer’s focus better.

Here again, is your marching order:

“Please make sure that all the links and functionality of the website are working by testing as a regular user. Let us know if anything interferes with the user experience on the site.”

A tester found this problem:

This one should be approved. It interferes with the user’s ability to use and enjoy the site as they have to browse back to nba.com.

Based on the same instructions should the following find be approved (the user then sees the results page after clicking vote)?

Only if the customer specified you had to vote to be allowed to see the results will it be accepted. Most, online polls operate this way. Here the tester is assuming their own specs and reporting them as bugs.

“Once upon a time bug”

There is another example of a non-bug that should be discussed and that is the “once upon a time bug.” Even if a bug is in scope, if it can’t be reproduced a second time (and doesn’t present a show stopping outcome) it is going to be of low priority and will likely be rejected.

Often what looks like a bug initially turns out to be a localized problem with your own browser. The smartest way to avoid rejections on things like broken images, buttons that don’t respond, video sync errors, (essentially, anything related to an element that might not have fully downloaded) is to clear your browser cache, restart your browser and retest before submitting.

Consider the following:

Questions to Ask:

Here’s a list of things you can ask yourself if you are not sure if it is a bug: They move from probably yes it’s a bug in green to mainly feedback (likely not a bug) in red.

  • Is something not working as designed?
  • Did the developer suggest this as a key area to look at?
  • Does it interfere with the user’s ability to use the site?
  • Can you make it happen more than once?

  • If only once…does it create a significant negative outcome?
  • Does it interfere with the user’s enjoyment of the product?
  • Is it done unconventionally or inconsistently?

  • Is it not the optimal way to do this?
  • Did YOU expect it to happen a different way?

If you follow the above guidelines you will find yourself getting more approvals, less rejections and less feedback approvals. Most cycles have a Team Test Leader and a customer reviewing the discussion thread to answer your concerns. Feel free to use them to help focus your testing in a manner that is productive to all.

In the event of a rejection, just remember that the customer isn’t necessarily saying you are wrong! They are just making a decision that your finding doesn’t meet their requirements of something needing attention at this stage in time.

Join the conversation about this topic in our forums here: http://forums.utest.com/viewtopic.php?f=55&t=3179