Automation Tools - Selenium 102
Author: Brian Rock
Introduction: This course describes the basic fundamentals of implementing an automated test framework using Selenium RC 1 and the Java/C#/Python programming language. This course makes several assumptions, among them:
- You are familiar with C#, Java or Python enough to write a unit test using NUnit, JUnit, or PyUnit; respectively.
- You have either Visual Studio or Eclipse installed on your system.
Background: Web-based applications are becoming the norm for customer facing products being delivered by software development companies today. Moreover, the complexities of these web-based applications are increasing to the point that they are on par with traditional thick client applications.
The testing effort of web-based applications has also become on par with traditional installed client programs. To make matters worse, the containers in which these applications execute, the browser, can be interchanged somewhat freely between the common variants; namely, Internet Explorer, FireFox and Chrome. This increases the testing effort for these applications. Taking into account the various operating systems on which the browsers can run, the required testing effort increases even more.
Automated software testing suites have commonly been purported to be the “silver bullet” for the testing problem. Unfortunately, the automated software to test the software is sometimes as complex and convoluted as the software it is testing. So, what are we to do, write an automated software test package to test the automated software test package that is testing the software :) I think not!
This is where Selenium RC 1 comes to the rescue! Mostly that is. Unfortunately, there is no “Silver Bullet” and all automated test software needs maintaining and reviews by knowledgeable software engineers. Moreover, automated software testing will never be able to replace the value of a good live software test engineer. Never.
Selenium is the closest alloy we have at the moment to solve this problem. Selenium RC 1 is an open source API that has been ported to a number of languages. Selenium RC 1 exposes an API that allows you to communicate commands to a Selenium server which in turn generates calls to a browser. These calls either request an action is executed, i.e. clicking a button or entering text, or they request information is returned, i.e. checking if text is present or returning the value of a specific node in the html.
This ability to drive a browser by issuing commands through an API coupled with a good unit test framework, like JUnit and its assertion statements; provides an effective testing framework for writing a maintainable and robust automated software testing package.
This course will be broken up into two parts, the first of which being mostly theory (Selenium 102) and the second of which focusing on implementation in each of three languages mentioned previously (Selenium 103). In this installment we will focus on the basics of xUnit Testing, the importance of creating a shared repository for object recognition strings, and some tips for using xUnit frameworks.
In the next installment we will implement and execute some test scenarios in each language using Selenium RC1 and Selenium RC 2 (webdriver) and discuss some tips&tricks.
Basics of xUnit Test: When we mention xUnit testing we are referring to the basic patterns of writing effective unit and system level automated tests. Almost every language today has its own version of an xUnit testing Framework for automated testing. Python has its unittest package, commonly referred to as pyunit, C# has nUnit and TeamTest, and Java has Kent Beck’s JUnit Framework. We will attempt to summarize the basics behind xUnit and highlight a single pattern - the four phased test - rather than analyze each testing framework.
Each framework attempts to provide an abstracted package of functionality for setting up, executing, validating and tearing down (or cleaning up) after the test. In addition to this core functionality, these frameworks also commonly provide services to automated execution and gathering of results.
As with most software problems that have been solved and analyzed before, there is usually an agreed upon best practice for solving a particular problem, these solutions are referred to as a software pattern. When writing an automated software test using an xUnit framework the pattern I typically follow what is called a Four Phase Test Patter (GOF1). The four phases consist of employing a Fresh/Shared fixture (i.e. the Application Under Test), exercising the application under test, verifying the result, and tearing it down.
- Set up the Fixture : This is the before picture that is required for the system under test to exhibit the expected behavior as well as anything you need to put in place to be able to observe the actual outcome.
- Exercise : This is when we interact with the system to generate the output to verify
- Verification: In this step we verify whether the expected outcome has been obtained.
- Tear Down: We tear down the test fixture to put the world back into the state in which we found it.
Setting up the fixture is usually a common set of steps that can factored out into a shared function or an abstract class for easier reuse in multiple test cases.
The exercising and the verification steps go hand in hand. This is the real meat of the test case and is the area that we are most concerned with testing. These steps should only be as complicated as needed to exercise the focused area under test. Every attempt should be made to keep these steps atomic so that defect masking doesn’t occur.
The final phase is the tear-down and clean up. From a repeatability standpoint, every change to a complicated system results in a new system which can lead to instability. In other words, we must attempt to get the system back into a known state after each test if possible. At the least, we need to get it back to a common state. This step is often overlooked but is very important.
Basic Project Structure: When automating in Selenium RC 1 I usually follow a standard project structure regardless of what language I am writing the automated suite in. The folder structure is highlighted below with a brief explanation of each item.
Project Root Folder
-Src
--TestPackages
--TestUtil
--config file
-Results
-ObjectRepository
The project root folder represents the name of the project and is generally what I use as the project name for source control. The src folder contains all the project source files, organized by testing package and all configuration files needed for the test to properly run.
The TestUtil folder generally contains classes or modules that are generalized for reuse throughout the solution. Such modules like those relating to fixture setup and tear down usually are stored here.
The Results folder is where the results of the automated testing are dropped. Depending on the language this may be just something that needs configured, such as for Junit/Nunit; or a bit more work will be needed to write a custom reporting framework such as a csv report for Pyunit.
Finally, the ObjectRepository folder will contain all files, either .txt and .xml, containing the xpath or other tags to recognize and locate the object while testing.
There are many other possible project folder structures that can be implemented, some are more complex and some are not. I personally try to keep the solution as simple as possible while still accomplishing the goals of the project.
Also of note, is that I am not providing or following a definition of general automation framework. There are many of these frameworks already defined (Hybrid, Keyword, Data Driven, etc …) and documented in various sources. In general, the documentation of these frameworks doesn’t provide much value, so instead, I am just focusing on what I generally employ which is a mix of several ‘frameworks’. You can call it what you will.
Importance of the Object Repository: When using Selenium RC 1 coupled with a xUnit test framework, you will need to manage the object recognition strings that will tell your test cases where the objects are located so that they can be accessed and manipulated during the tests. There are two schools of thought here; you can leave the xpath/locator strings directly in the source code of you can create a library of such strings and build a method to access them. There are pros and cons to each approach.
The pros to creating an object repository are that you are abstracting one source of change out of your tests. This will make your test code a bit more robust in that the automation test solution will be able to handle minor UI changes without having to change the source code of the solution. This will allow the user to update the locater strings in the separate file rather than opening up the source code and manually changing the strings, when the AUT is modified.
The implementation of an object repository is somewhat trivial. The location strings are stored in an external text file as either key/value pairs or in a property file. Then, in the source code you access the location string by using a method that first loads all of the strings into a map object and then accesses the members of the map by the key passed to the method, returning the value to the code.
The cons to this approach lies in the added complexity to the automation test solution. This complexity is especially evident when trying to debug failing tests. As long as the AUT hasn’t changed drastically, the location strings are usually the culprit for test failures with Selenium RC 1. However, this added complexity is usually worth the added effort once the automated solution reaches a certain number of scenarios, usually anything more than 50.
Importance of Naming Test Appropriately: Since the names of the tests are generally used in any reporting framework to highlight when a test fails, it is important to name the test expressively. I generally provide the basic scenario details and any modifiers to that scenario in the test name. For example, here are three test names that I would use when writing tests to validate the login functionality of an application. Specifically, I would test logging in with valid credentials, invalid credentials (i.e. incorrect password) and finally, with a missing password.
- testLoginValidCreds()
- testLoginValidUserBadPassword()
- testLoginValidUserBlankPassword()
Using test names in this fashion make it trivial to localize the scope of a defect within an automated test.
Importance of Test Validation Focus: Another, often overlooked, aspect of automated test scenarios is the concept of defect localization. If we are following the pattern of a four phase test the validation section should be rather small compared to the other sections. When we are automating a test the validation focus of the test should be specific.
What this means, is that we do not want to write a test that validates several separate items within a single scenario. The item that we are validating should be clearly listed within the test name to allow us to easily understand what failed in the test. All logic to set up the shared fixture should be abstracted to a separate utility function already, so in pseudo-code, our tests should look roughly like the following:
def testSomeUserFunctionInApplication():
#load shared Fixture
self.login();
self.loadFunctionPageB();
#exercise application and capture value
selenium.do_some_action();
selenium.click_button(self.object_reposi.load(key));
actual = selenium.capture_screen_value();
#assert / validate value against expected
self.assertTrue(actual,expected,”Value didn’t match,test failed”);
#clean up / tear down
self.removeLastAction();
The above Python-like pseudo code demonstrates a simple four phase test pattern containing a well-focused name and a focused validation point that would aide in defect localization. This test would allow the user, when ran and if failed, to easily identify the error that the test exposed in the application with very little debugging.
Please don’t dwell on the pseudo code above. In the next installment, we will have actual code and it will make much more sense.
