Custom Search

Sunday, May 10, 2009

Automating Tests with Segue Silk

This article, originally written in 1997 and updated in 2003, discusses architecting and designing your Silk scripts and also touches on some of the more advanced features of Silk such as custom classes and methods, data driven testing, and the @ operator.

Up Front Design
Test automation projects are programming projects. Period. No matter how cool the test tool interface looks, all test automation is programming at some level or another. This isn't a statement about Segue SilkTest in particular: test automation is still programming no matter which of the GUI test automation tools you choose (Mercury, Rational, Compuware, etc.).
Because many test automation tools do a good job of hiding the complexity, it's tempting to dive right in and begin automating without having any kind of strategy or plan in mind. In fact, if you've never used Segue Silk before you might want to do just that so you can begin to understand the capabilities and limitations of the tool. Go ahead. Experiment. Create some scripts, then try to run them against subsequent builds.
I'll wait here.
Oh good. You're back. What you probably found is that you could create scripts quickly. It didn't take long before you could generate an impressive amount of activity on the screen. Then when you tried to play those same scripts on later builds everything broke. You may even have had to start over. Without even a little up front design, test automation scripts are too fragile, too difficult to maintain.
So let's back up a step and talk about the up front design work you should do before beginning a serious test automation project.
I strongly recommend starting with a test automation strategy document and a test design specification. The strategy document will address questions such as

What tool(s) are we using?
What's our general approach to test automation?
What products or projects will we use test automation on?
What is the goal of automation?
How will we measure progress toward that goal?
What are some of the benefits we expect to reap?
What are some of the risks we foresee?
What will it take to make test automation a success?
Who will be responsible for creating, maintaining, and executing the automated scripts?

This document should be short: somewhere between 1 - 5 pages. The point of writing the document is to explain the test automation effort at a high level. This will be the most widely read test automation document. When the VP of Development wants to know what you're doing, you can hand him this document.
Next you want to begin thinking through some of the design issues with your test automation infrastructure: a framework that describes the application including reusable classes, methods, and functions. At the beginning of your automation effort, plan to spend the vast majority of your automation resources building your infrastructure. The investment will pay off in the end. Creating new scripts will be easier if there is a firm foundation on which they can rest. Existing scripts will be more maintainable if common objects and code is located in just one place rather than copied and pasted all over.
It's important to note that while it is a good idea to design your test automation infrastructure at the beginning, you do not need to implement it all at once. You can begin with a basic framework and add to it as you see the need. As long as you don't change the logical names of your window objects, you won't have to rewrite your test scripts to accommodate the changes to your infrastructure. In SilkTest that means thinking about:

Your application framework
Addition classes, methods, and functions you create
Your test cases, scripts, and suites of scripts
The file structure for your include files, scripts, etc.

Create a separate document to specify these kinds of details. Note that I am not suggesting that you design and specify all aspects of your infrastructure up front. These are things to think about now and throughout your test automation effort. This is a living document. You'll continue updating the design specification throughout the automation project. But by designing at least part of the infrastructure up front, you have a roadmap to follow, a checklist of programming tasks, and you can better measure your progress against your plan.
As with writing any kind of specification writing, there is a balance point between not enough detail and too much. Over-specifying your design may result in a document that is so long, no one will read it. Furthermore, the more implementation detail you put into the design specification, the more likely the design specification will be out of date the minute you begin implementation. Implementation details that sounded good when you were writing may not work nearly as well in practice. So make your specification as brief as possible while still covering the essential elements of the design.

Creating Your Application Framework

The SilkTest wizard will walk you through creating your application framework. This is the point in the project where you begin creating the abstract layer between your software under test and your test scripts. Not sure what I mean by abstract layer? I explain it in my article Making the Right Choice (PDF). Not sure you need an abstract layer? Trust me: you do. The test tools on the market that don't support or encourage the notion of an abstract layer are doing both themselves and their users a disservice. In a nutshell, the abstract layer is one of the keys to making your scripts maintainable.
Although you could declare all the windows in your application up front, I recommend taking a staged approach. By declaring only the windows you need at any given time, you'll find problems with your declarations earlier. And there will be problems. Segue Silk is a powerful tool, but nothing is magic. I've had projects where halfway through declaring my windows I discovered that I'd chosen the wrong identification method (chose Window ID when I should have chosen Index, etc.). By declaring only the handful of windows I was working on up front, I had less work to redo than if I'd done all my window declarations up front.
Once you have declared at least some of the windows in your application, you might want to look at ways of refining your framework to make it more maintainable, flexible, and robust. Some suggestions, including creating custom classes, methods, and functions, appear in the sections below.

Determining Your File Structure

By now you've started creating a framework. You probably have everything in a single include (.inc) file.
If you are testing a reasonably complex application, you may want to separate your framework into different files to minimize the amount of redundant maintenance you need to do. It can be difficult to determine where one file should end and another begin. Here are some lessons learned the hard way:
Do put all declarations associated with the same module or executable in a single file. This makes it easier to manage the include file for the module or executable.
If you are testing a series of executables with common elements, create a separate file to hold common elements. Each module will then require (at least) the two include files: its own and the common include file used by all modules.
Do not put functions into the same files with object declarations, particularly if you are working on more than one release of your application at a time. The executable statements in functions often stay the same from release to release, but object declarations may differ, particularly if your developers are fond of rearranging the interface on a regular basis. It's a good idea to pull functions into a general use library file. Include classes and their associated methods in the general use library file.
Consider creating separate library files for each executable or module if one common library file gets too long or if the executables are on different release schedules.
Make sure that the module/executable include files are dependent on the general include file, but not vice versa.
In general, the goal of any file structure is to minimize the number of places in which you need to make a particular change. You need to balance granularity with ease-of-use: separating everything into different files may give you greater control over what code is shared between projects and releases, but it also makes it very difficult to track which files contain what code.
Finally, just in case you haven't already done this: get source control for your tests. Also make sure that everyone creating and editing scripts knows how to use the source control system. Source control for scripts is vital regardless of which automated test tool you choose.

Creating Classes and Class Hierarchies

In order to make your infrastructure more maintainable, it's a good idea to refine your framework by creating classes for similar windows. If you don't have an object-oriented programming background, think of classes as a template. Creating a class to describe common elements in your program allows you to specify the common details just once.

In the design stage, you want to identify good candidates for classes. To do this:
Identify windows that are visually similar.
Do they serve a similar purpose?
Do they contain similar (or identical) fields and buttons?
Use the Record Window Declarations menu option in SilkTest to look at the underlying window declarations. Windows that look alike on the surface don't always look alike to SilkTest.
When you record the windows, do the resulting declarations look similar?
What are the differences? Different logical names don't matter, different tags matter a little, radically different structures will make classing a given window virtually impossible.
Once you've identified candidates for custom classes, you can create your custom classes using the Classes option on the Record menu, or you can create them manually using the winclass keyword like so:
winclass CustomDialog : DialogBox
(See the SilkTest documentation for details.)

The most straightforward use of classes is to reduce redundant window declarations. Creating classes also allows you to write methods that will be inherited by all windows that are members of the class.
You can also create a hierarchy of classes so related but different objects can still inherit characteristics from a common parent. Hierarchies of classes are useful if:
You have a number of windows that are similar but that fall into different categories. For example, if there are a number of places in your application where you get a File Open dialog, and a number of places where you get a Save dialog. Both dialogs may have elements in common (list of directories, file name field, etc.), and unique elements (Save button vs. Open, etc.).
A similar window appears in several executables or modules. You might want to create a parent class for all the related windows in the main library file, then create separate classes for each executable or module.

Designing Functions and Methods

Chances are there are actions you perform repeatedly while testing. For example, if you are testing a word processor, you probably have to open files over and over again. This kind of action is an ideal candidate for a function or a method. By pulling out common code into reusable functions and methods you improve the maintainability of your test scripts and also improve productivity.
Creating general use methods and functions is a good way to reduce the size of your scripts and increase the reusability of your code. When writing methods and functions:
Keep the method or function as generic as possible by passing parameters.
Consider passing a record rather than separate variables if multiple variables are needed.
With methods, remember that you can add more functionality at the window or subclass level by using the derived::method_name() syntax.
With functions, remember that you can always call functions from other functions. Consider creating sets of functions that can work together rather than creating a single function that does everything but is less flexible.

Passing records instead of separate variables makes the method or function more maintainable. If an additional variable is needed later, adding it to the record rather than as a new parameter to the method or function means that the programming interface to the method or function doesn't change.

Is It a Method or a Function?

One of the major design decisions is whether to make a series of actions a method or a function. The difference between a function and a method is that a method can only act on an object. You can create methods for individual objects as well as classes. Functions can act on objects as well, but more often you'll create functions to act on data or collections of objects.
In deciding whether a series of actions should become a function or a method, consider the advantages of methods over functions:
The this keyword allows you to reference otherwise ambiguous objects
Inheritance and polymorphism give you a cleaner programming interface
Because a method can only act on an object, you have built in error checking

Using the This Keyword: The this keyword allows you to refer to an object without knowing the name of the object. For example, in the generic word processing application mentioned above, you might want to have a method to find a file or directory in any browser window. Since the method is supposed to work in any browser window, you won't know ahead of time the name of the browser window. Simply substitute the word "this" whenever you would use the name of the window. For example, you might end up with a line of code in your method like:
this.FileList.Select(sFileName)


Using Inheritance and Polymorphism: When you add a method to a class, all objects declared as members of that class automatically have access to the method; they inherit it. This means that you only need to write a method once and you can use it throughout your tests. However, there may be times when you need to modify a method for one member of the class. You can define a method for a particular object with the same name as the class method and different functionality. The new method overrides the class method. This is called polymorphism.
Note: SilkTest has imposed a limitation on inheritance. You cannot inherit custom methods within the SilkTest-supplied hierarchy of classes. For example, the SilkTest class DialogBox is derived from the SilkTest class ChildWin. If you create a method for the ChildWin class, you would expect members of DialogBox to inherit it, but they don't. In order to use the same method for both the ChildWin and DialogBox classes, you must create the method twice, once for each class. This limitation is not a bug: Segue introduced the limitation to make it easier to debug script problems involving built-in classes.
Built in Error Checking: When you create a method, you can only call that method when there is an object on which the method can act. When you create a function, you can call the function at any time. QA Partner's error recovery system ensures that your script will recover if an object referenced in the function or method is not found. But if you use methods where it makes sense, QA Partner will recognize the error earlier with a method, because the statement calling the method includes the name of the object.

Using the "@" Redirection Operator


For all you C/C++ programmers out there, the "@" operator is the closest thing to a pointer in 4Test. You can use @ to dynamically refer to a field on a form or in a record or as a function pointer, to dynamically decide which function to call based on the value of a variable. This is very powerful and allows you to create very general functions, methods, and test cases.

For example, consider the following trivial example:
// create a record datatype with two fieldstype DATARECORD is record STRING sField1 INTEGER iField2

main() // use our new record datatype DATARECORD rData = {"wombat", 42} STRING sIt // iterate thru the record fields for each sIt in FieldsOfRecord(DATARECORD) print("Field: {sIt}, Val: {rData.@sIt}")
The final line prints out both sIt and @sIt. The @ operator causes the value of sIterate to be substituted. The results of the above script are:
Field: sField1, Val: wombatField: iField2, Val: 42
To use the @ operator to automatically fill in the fields on a form, simply create a record with fields that have the same name as the fields on the form. Then use something like the following to iterate through the fields and place their contents in the fields on the form:
Populate(DATARECORD rData) STRING sIt for each sIt in FieldsOfRecord(DATARECORD) MyWindow.@sIt.SetText(rData.@sIt)
You can also use the @ operator to dynamically call functions according to the value of a variable. The syntax to do this looks like:
@(VariableName)(Parameters go here)
For example, the following trivial example function calls the function specified by the sFunctionName variable using the parameter specified by the aData variable and returns the value returned by the function called.
ANYTYPE MyFunc(STRING sFuncName, ANYTYPE aData) return @(sFuncName)(aData)
Although trivial, this example shows you how you could write a wrapper function to call one of several other functions according to the value in a variable.

Generalizing Your Test Cases: Use Data Driven Testing


Segue SilkTest now supports data driven testing as of the 6.0 release. This is HUGE news. In the past, you had to write the code to make your scripts open and read a file. While the code wasn't that bad to write, it was something of a pain and it discouraged people from getting the most out of their test automation effort.
Now, however, you can use the Data Drive Testcase option on the Tools menu to make any test case data driven. Further, the test case can pull the data from a spreadsheet or a database, making it much easier to maintain your test data.
For more information in Data Driven Testing with SilkTest 6.0 see the SilkTest documentation.

Tips for Incorporating SilkTest Scripts into Your Testing Process

All the automation in the world won't help if the automated and manual efforts are not integrated. Here are some tips for incorporating results into your testing process:

Make the first field in test data records passed to a test case a description of the test case. This gives you traceability from the automated script to the test documentation.
Include the name of the automated test script or test case in the test documentation for each test. This gives you traceability from the test documentation to the automated script.
Use the LogError, LogWarning, and print functions to add important information to the results file. This will help both in tracking results and in debugging problems.
Don't automatically log any error encountered to your problem tracking system. Test automation can point you in the general direction of a problem but further investigation is usually required to identify the actual bug.
If a different group of testers is running the automated tests than creating them, ensure those testers get training in SilkTest too. It will help them demystify their results and find false negatives (and false positives!) more quickly.