Benefits of Automation Testing 101

Author: Iwona Pekala
Date: June 23rd 2011

 

Saving time using test automation

Test automation can save time in multiple ways. Testing should be automated, mainly when particular tests should be run repeatedly. This happens in following situations:

  • when particular area can change all the time during development process
  • when on each build should be run BVT[1] test, before further tests can be performed
  • when regression tests should be run from time to time
  • during regular testing of fix packs or service packs, to ensure that nothing was broken

Automating some areas leaves more time for testing other areas which can’t be automated. As a result, there is lower risk that some areas was skipped during testing due to lack of time. Also it gives a chance to perform more detailed test in non-automated areas.  As a result product can be delivered with better quality.

Automation is the most usable for BVT test, as its scope is usually small, so its automation shouldn’t take much time. It’s also the kind of test which is run on daily basis or even more often. Generally, the more often you use your automation tests, the more time it saves. It also leads to conclusion that areas which are tested once or twice shouldn’t be automated. In such cases, automating tests can take more time than it saves later. It’s also important to start automating when the code becomes stable and design can’t be changed. Starting automation too early can cause the need for rewrites, due to the changes.

Advantages of removing human factor from testing

There are some cases, when it’s better to perform tasks automatically, as it can give better results. When one area is tested by one person repeatedly, there is a risk that after longer period the person can’t find any defect. People easily get used to bad behavior. It causes danger, not only for repeatedly tested area, but also for other components for the software. A bored person won’t be paying attention to the details.

Automated tests are usually very sensitive. One very small change can cause everything to fail. It can be annoying sometimes, as not every change means defect, but it can catch new defects earlier than it would done during manual testing.

Automation is also good for catching bugs which are hard to reproduce. Automated tests always have the same steps, while the tester can forget some of his (or her) previous actions. As a result, the defect can’t be reproduced. Automation gives a guarantee that each test case is always executed in the same way and in the same environment. When executing a test case manually, some steps could be missed, or intentionally omitted. This can result in not only missing some bugs, but also can lead to reporting invalid bug (omitted steps can cause unexpected behavior).


Automated tests of data calculation

Usually test automation has a lot of advantages, but it’s not always necessary. There are some situations when it can’t be avoided. Let’s imagine a situation, that software calculates something, which is based on a lot of data. One number result based on 100 000 input data. Such a thing can’t be tested manually. Additionally, calculations in such situation are usually based on more than one formula, or can contain some dependencies. In this case, a lot of errors can be made during implementation. The process of automation in such a situation can include generating a lot of sets of data, including boundary cases.

In next step, all calculations algorithms from the design should be implemented by testers. It seems to be copying developers work, but it’s usually simpler, as in the process of automation only calculation part is implemented.

Finally, calculation results must be compared. At the beginning there will be a lot differences between results, as both developers and testers usually make mistakes during implementation. But after eliminating errors in both codes, results should be the same.

Is there any way to check complicated calculations without implementing it twice? I can answer this question giving an example. Assume that one of the software’s functions is to calculate linear regression (http://en.wikipedia.org/wiki/Linear_regression). We have a set of data and the result. Can we check if it’s correct just by looking at it? We can’t be sure. We can check how it looks on the chart. It may look good, but we can’t still be sure that it’s okay. The only way to be sure about is to calculate it independently. Calculating it manually could take hours, or days. The only rational method is to implement it again.

graphrational.jpg


[1] Build verification test - http://en.wikipedia.org/wiki/Build_Verification_Test