CMM Overview

1 The Process MaturityFramework

After two decades of unfulfilled promises about productivity and quality gains from applying new software methodologies and technologies, industry and government organizations are realizing that their fundamental problem is the inability to manage the software process [DoD87]. The benefits of better methods and tools cannot be realized in the maelstrom of an undisciplined, chaotic project. In many organizations, projects are often excessively late and double the planned budget [Siegel90]. In such instances, the organization frequently is not providing the
infrastructure and support necessary to help projects avoid these problems. Even in disciplined organizations, however, some individual software projects produce excellent results. When such projects succeed, it is generally through the heroic efforts of a dedicated team, rather than through repeating the proven methods of an organization with a mature software process. In the absence of an organization-wide software process, repeating results depends entirely on having the same individuals available for the next project. Success that rests solely on the availability of specific individuals provides no basis for long-term productivity and quality improvement throughout an organization. Continuous improvement can occur only through focused and sustained effort towards building a process infrastructure of effective software engineering and management practices.

1.1 Immature Versus Mature Software Organizations

Setting sensible goals for process improvement requires an understanding of the difference between immature and mature software organizations. In an immature software organization, software processes are generally improvised by practitioners and their management during the course of the project. Even if a software process has been specified, it is not rigorously followed or enforced. The immature software organization is reactionary, and managers are usually focused on solving immediate crises (better known as fire fighting). Schedules and budgets are routinely exceeded because they are not based on realistic estimates. When hard deadlines are
imposed, product functionality and quality are often compromised to meet the schedule.
In an immature organization, there is no objective basis for judging product quality or for solving product or process problems. Therefore, product quality is difficult to predict. Activities intended to enhance quality such as reviews and testing are often curtailed or eliminated when projects fall behind schedule.
On the other hand, a mature software organization possesses an organization-wide ability for managing software development and maintenance processes. The software process is accurately communicated to both existing staff and new employees, and work activities are carried out
according to the planned process. The processes mandated are fit for use [Humphrey91b] and consistent with the way the work actually gets done. These defined processes are updated when necessary, and improvements are developed through controlled pilot-tests and/or cost benefit analyses. Roles and responsibilities within the defined process are clear throughout
the project and across the organization.
In a mature organization, managers monitor the quality of the software products and customer satisfaction. There is an objective, quantitative basis for judging product quality and analyzing problems with the product and process. Schedules and budgets are based on historical performance and are realistic; the expected results for cost, schedule, functionality, and quality of the product are usually achieved. In general, a disciplined process is consistently followed because all of the participants understand the value of doing so, and the necessary infrastructure exists to support the process.
Capitalizing on these observations about immature and mature software organizations requires construction of a software process maturity framework. This framework describes an evolutionary path from ad hoc, chaotic processes to mature, disciplined software processes. Without this framework, improvement programs may prove ineffective because the necessary foundation for supporting successive improvements has not been established. The software process maturity framework emerges from integrating the concepts of software process, software process capability, software process performance, and software process maturity, all of which are defined in succeeding paragraphs.


1.2 Fundamental Concepts Underlying Process Maturity

According to Webster's dictionary, a process is "a system of operations in producing something ... a series of actions, changes, or functions that achieve an end or result." The IEEE defines a process as "a sequence of steps performed for a given purpose" [IEEE-STD-610]. A software process can be defined as a set of activities, methods, practices, and transformations that
people use to develop and maintain software and the associated products (e.g., project plans, design documents, code, test cases, and user manuals). As an organization matures, the software process becomes better defined and more consistently implemented throughout the organization.
Software process capability describes the range of expected results that can be achieved by following a software process. The software process capability of an organization provides one means of predicting the most likely outcomes to be expected from the next software project the organization undertakes.
Software process performance represents the actual results achieved by following a software process. Thus, software process performance focuses on the results achieved, while software process capability focuses on results expected. Based on the attributes of a specific project and the context within which it is conducted, the actual performance of the project may not reflect
the full process capability of the organization; i.e., the capability of the project is constrained by its environment. For instance, radical changes in the application or technology undertaken may place a project' s staff on a learning curve that causes their project's capability, and performance, to fall short of the organization's full process capability.
Software process maturity is the extent to which a specific process is explicitly defined,managed, measured, controlled, and effective. Maturity implies a potential for growth in capability and indicates both the richness of an organization's software process and the consistency with which it is applied in projects throughout the organization. The software process is well-understood throughout a mature organization, usually through documentation and training, and the process is continually being monitored and improved by its users. The capability of a mature software
process is known. Software process maturity implies that the productivity and quality resulting from an organization’s software process can be improved over time through consistent gains in the discipline achieved by using its software process.
As a software organization gains in software process maturity, it institutionalizes its software process via policies, standards, and organizational structures. Institutionalization entails building an infrastructure and a corporate culture that supports the methods, practices, and procedures of the business so that they endure after those who originally defined them have gone.


2 The Five Levels of Software Process Maturity

Continuous process improvement is based on many small, evolutionary steps rather than revolutionary innovations [Imai86]. The CMM provides a framework for organizing these evolutionary steps into five maturity levels that lay successive foundations for continuous process improvement. These five maturity levels define an ordinal scale for measuring the
maturity of an organization's software process and for evaluating its software process capability. The levels also help an organization prioritize its improvement efforts.

A maturity level is a well-defined evolutionary plateau toward achieving a mature software process. Each maturity level provides a layer in the foundation for continuous process improvement. Each level comprises a set of process goals that, when satisfied, stabilize an important component of the software process. Achieving each level of the maturity framework
establishes a different component in the software process, resulting in an increase in the process capability of the organization.

Organizing the CMM into the five levels shown in Figure 2.1 prioritizes improvement actions for increasing software process maturity. The labeled arrows in Figure 2.1 indicate the type of process capability being institutionalized by the organization at each step of the maturity
framework.





The following characterizations of the five maturity levels highlight the primary process changes made at each level:

1) Initial The software process is characterized as ad hoc, and occasionally even chaotic. Few processes are defined, and success depends on individual effort.

2) Repeatable Basic project management processes are established to track cost, schedule, and functionality. The necessary process discipline is in place to repeat earlier successes on projects with similar applications.

3) Defined The software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization. All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software.

4) Managed Detailed measures of the software process and product quality are collected. Both the software process and products are quantitatively understood and controlled.

5) Optimizing Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.

Test Plan Guidelines

This is a sample of an outline for a test plan. It has been designed for medium to small test projects, and thus is fairly lightweight. It is by necessity general, because each enterprise, each development group, each testing group, and each development project is different. This outline should be used as a set of guidelines for creating your own standard template; add to it or subtract from it as you find appropriate. Bear in mind that it is generally better to have an excess of detail in the template—detail which can be removed when creating a specific test plan—than to have to remember to add something that is not in the template.

(Looking for a heavy-duty test plan? The government and the military are good sources. Try the one on the IRS Web site:

Make sure to fill in the running headers and footers with the product name, draft numbers, revision dates, and page numbers; this is important in places with lots of test projects on the go. Make sure to include the author’s name, too, so that errors or questions can be addressed to the right person.

1. OVERVIEW
1.1. PRODUCT NAME
1.2. PRODUCT REVISION
1.3. PROJECT LEADS
1.3.1. Marketing Lead (or other customer representative)
1.3.2. Program Manager
1.3.3. Development Lead
1.3.4. Test Lead
1.3.5. Build and Release Control Engineer
1.3.6. Legal representative

Include names, phone numbers, and email addresses for each. Note that this table will differ for a particular company or group. The goal is to ensure that anyone walking into the company or into the test role can easily identify and contact the people he/she needs to reach.


1.4. TEST PROJECT STAFF
1.4.1. Test requirements designers
1.4.2. Test case designers
1.4.3. Test personnel
1.4.3.1. For manual (i.e. non-automated) tests
1.4.3.2. For automated tests
1.4.3.3. Test automation programmers
1.4.4. Documentation reviewers
1.4.5. Legal reviewer

Include names, phone numbers, and email addresses for each. Note that there may be several people in each role, that one person may be obliged to fill multiple roles, and that some roles (e.g. legal reviewer) won’t be required for all projects.


1.5. PRODUCT OVERVIEW

This could also be “description of the change requirements” for maintenance projects.
1.5.1. Cut and paste from requirements document or specification a brief summary description of the product or change, or describe the project as understood by the developers—if the latter, make sure that there is agreement and sign-off from the customer.

1.6. TRACKING AND REPORTING SYSTEMS

1.6.1. Identify the defect tracking system in use.
1.6.2. Identify the manner and schedule by which defect reports are expected to be delivered to developers.
1.6.3. Identify parties that may have access to the tracking system.
1.6.4. Identify the change control system.
1.6.5. Identify the means by which the team is to be notified of changes to the requirements, the product, the test plan, etc.


2. TESTING SYNOPSIS

2.1. Items to be tested
2.1.1. Refer to the functional requirements that specify the features and functions to be tested. The description of the change need not be excessively detailed when there is a complete description to refer to in some other document. On the other hand, if there is no reasonable specification available, more detail is called for here.

2.2. Items not to be Tested
2.2.1. List the features and functions that will not be covered in this test plan. Identify briefly the reasons for leaving them out.


2.3. System Requirements
2.3.1. This section should be filled out in detail for new projects. For existing maintenance tasks, a simple cross-reference to the document describing existing system requirements is fine. Note any changes to previous system requirements, especially when support for a given product or platform is being dropped.

2.3.2. If there is a system requirement that could be unclear, make it specific; for example, for Web-based projects, identify not only the supported browsers but also the minimum versions of the supported browsers.


2.4. Standards/Reference material
2.4.1. List any standards or other reference material used in the creation of this test plan.
2.4.2. Identify standards for acceptance criteria, defect severity, testable specifications, and so on. (These standards may have to be created, or adapted from time to time; the first use of this test plan will require more work than later iterations.)


2.5. Glossary
2.5.1. In cases where terminology could be unfamiliar or open to interpretation, provide a list defining the unclear terms.
2.5.2. Obtain agreement on these terms from all interested parties.
2.5.3. Note: If no one is forthcoming with the information you need, make something up; they might not have done their jobs from the outset, but they’ll be happy to correct your work! You will have achieved the goal, which is clarity and agreement.


3. TYPES OF TESTING


3.1. ACCEPTANCE TESTING


3.1.1. Detail a set of acceptance criteria—conditions that must be met before testing can begin. A smoke test should represent the bare minimum of acceptance testing.
3.1.2. As noted above, the ideal is to create a separate document for acceptance criteria that can be reused and referred to here. If any particular, specialized test cases not listed in that document will be used, refer to them here.

3.2. FEATURE LEVEL TESTING


This is the real meat of the test plan. The test categories below are filled in itemizing categories of tests, along with references to the test library or catalog. Individual test cases should not be listed here; test requirements generally should not be either; the details should exist elsewhere and can be cross-referenced.
3.2.1. Task-Oriented Functional Tests
3.2.1.1. This is a detailed section, listing tests requirements for program features against functional specifications, user guides or other design related documents. If there are test matrices available listing these features and their interdependence (and there should be), refer to them.
3.2.2. Forced-Error Tests
3.2.2.1. Provide or refer to a list of all error conditions and messages. Identify the tests that will be run to force the program into error conditions.
3.2.3. Boundary Tests
3.2.3.1. Boundary tests—tests carried out at the lines between valid and invalid input, acceptable and unacceptable system requirements (such as memory, disk space, or timing), and other tests at the limits of performance—are the keys to eliminating duplication of effort. Identify the types of boundary tests that will be carried out. Note that such tests can also fall into the categories outlined below, so this section may be removed, or made a sub-section of those categories.
3.2.4. Integration Tests
3.2.4.1. Identify components or modules that can be combined and tested independently to reduce dependence on system testing. Identify any test harnesses or drivers that need to be developed.
3.2.5. System-Level Tests
3.2.5.1. Specify the tests will be carried out to fully exercise the program as a whole to ensure that all elements of the integrated system function properly. Note that when unit and integration testing have been properly performed, the dependence upon system testing can be reduced.
3.2.6. Real World User-Level Test
3.2.6.1. In contrast to types of testing designed to find defects, identify tests that will demonstrate the successful functioning of the program as you expect the customer to use it. What type of workflow tests will be run? What type of “real work” will be carried out using the program?
3.2.7. Unstructured Tests
3.2.7.1. Specify the amount of ad-hoc or exploratory testing that will be carried out. Identify the scope and the time associated with this form of testing.
3.2.8. Volume Tests
3.2.8.1. Indicate the types of tests will be carried out to see how the program deals with very large amounts of data, or with a large demand on timely processing. Note that these tests can rarely be performed without automation; identify the automation tools, test harnesses, or scripts that will be used. Ensure that the programs developed for the test automation effort are accompanied by their own sets of requirements, specifications, and development processes.
3.2.9. Stress Tests
3.2.9.1. Identify the limits under which the program is expected to perform. These may include number of transactions per unit time, timeouts, memory constraints, disk space constraints, and so on. Volume tests and stress tests are closely related; you may consider wrapping both into the same category.
3.2.9.2. How will the product be tested to push the upper functional limits of the program? Will specific tools or test suites be used to carry out stress tests? Ensure that these are reusable.
3.2.10. Performance Tests
3.2.10.1. Refer to the functional requirements that specify acceptable performance. Identify the functions that need to be measured, and the tests needed to show conformance to the requirements.

3.3. REGRESSION TESTING

3.3.1. At each stage of new development or maintenance, a subset of the regression test library should be run, focusing on the feature or function that has changed from the previous version. Unit, integration, and system tests are all viable places for regression testing. For small maintenance fixes, identify this subset. A good version control system can allow the building of older versions of the software for comparative purposes.

3.3.2. In the final phase of a complete development cycle, a full regression test cycle is run. Identify the test case libraries and suites that will be run.

3.3.3. Whether a subset or a full regression test run, existing test scripts, matrices and test cases should be used, whether automation is available or not. Identify the documents that describe the details. Emphasize regression tests for functions that are new or that have changed, for components that have had a history of vulnerability, for high-risk defects, and for previously-fixed severe defects.


3.4. CONFIGURATION AND COMPATIBILITY TESTING

3.4.1. If applicable, identify the types of software and hardware compatibility tests that will be carried out.

3.4.2. List operating systems, software applications, device drivers etc. that the product will be tested with or against.

3.4.3. List hardware environments required for in-house testing.


3.5. DOCUMENTATION TESTING/ONLINE HELP TESTING

3.5.1. Documentation and online help testing will be carried out to verify technical accuracy of documented material.

3.5.2. If a license agreement is included in or displayed by the product, or the portion of it to which this test plan refers, ensure the correct one is being used (see the next item below).


3.6. COPYRIGHTS AND LICENSE AGREEMENT

3.6.1. Identify any copyright notices displayed by the program. Verify that they are accurate and up to date.

3.6.2. In cases where an End-User License Agreement (EULA) is displayed by the program, which EULA will be used in this product? Provide a link to the file. Ensure that it is the consistent with the one included in the product.

3.6.3. Receive sign-off from the legal department that this is the correct EULA for this product.


3.7. UTILITY, TOOL KIT, AND COLLATERAL TESTS

3.7.1. If there are any additional products or components to be included in the final product, or on the distribution media, list the types of tests that will be carried out, and the extent to which they shall be performed.


3.8. INSTALL/UNINSTALL TESTS

3.8.1. How will deployment and installation be tested?
3.8.2. How will the uninstallation or rollback process be tested?
3.8.3. Since some form of deployment is required for all software products, what generic installation and uninstallation test catalogs will be used or adapted for these tests?


3.9. CODE COVERAGE

3.9.1. What tools or processes will be used to assure that each line of code is run at least once during testing?
3.9.2. Have the developers performed coverage tests during unit or integration testing? Have they provided the results of these tests? Have they provided source code, test harnesses, or test tools?
3.9.3. Are there plans to cover all code during regression testing? If not, why not?


3.10. YEAR 2000 AND DATE COMPLIANCE

3.10.1. Identify date and time values that are accepted, calculated, and output by the program. Pay attention both to hard dates and timespans.
3.10.2. What tests, if any, will be carried out to make sure the program will continue to work correctly when dates on both sides of the year 2000 are processed?
3.10.3. What tests, if any, will be run to ensure that other forms of date processing are done correctly?


3.11. INTERNATIONALIZATION

3.11.1. For products intended for global markets, what tests will be carried out to make sure the product can be easily localized (that is, adapted for a specific local market)? For products intended for Asian markets, what tests will be performed to verify that the program correctly handles multiple-byte character sets?


4. TEST SCHEDULE AND RESOURCES


4.1. Identify the estimated effort required to execute the test plan. Include a both a range and a confidence level. Use the guidelines on pp. 3.23 – 3.44 of the course book to plan and review estimates.

4.2. Identify the resources available to carry out the test plan.

4.3. Identify time or resource constraints that will lead to a risk of the test project falling behind schedule, below expected scope, or below expected quality. Cross-reference this with the Unresolved Issues and Risks section later in this document.

4.4. If any testing is to be handled by another entity, such as another department or a third party test lab, identify them. List names and contact information at the beginning of this document. List the specific tasks they will be assigned to carry out. Include references to contracts with these people, and ensure that contracts are approved and signed.


5. TEST PHASES AND COMPLETION CRITERIA

5.1. Detail the planned test cycles and phases; these should be linked to the development plan for the project. Specify the type of testing being done in each phase. Typically unit testing will be done by the developer of the code, and need not be covered in detail in the test plan. Integration and system testing phases should be detailed here.

5.2. Outline the criteria for assessing the severity of found defects. List expectations for setting the priorities on resolving them. Collaborate with the developer(s), project managers, and the customer representatives on this.

5.3. Identify in advance the criteria that must be fulfilled before each stage of testing can be considered complete. Make these specific, measurable, and decidable; otherwise, expectations will differ and time will be wasted on discussion and debate.

5.4. If there are to be staged releases of system testing – typically alpha for internal releases, beta for limited releases to external test sites, and final releases – sometime called “gold master”, define them. Define acceptance standards for each phase. Ideally these should be in a separate document that can be referred to here

Bear in mind that there is a chance that the standards set here are subject to being overruled by some authority or another; for example, a product may ship with a higher than satisfactory number of minor defects, at the behest of a marketing department or CFO that wants the product released with time as the most important consideration. Be prepared to accept such decisions dispassionately, but also be prepared to record them as failures to fulfill the standards set and agreed upon in advance. Companies and individuals can forget easily and repeat mistakes when there is no record of breached agreements and their consequences; people learn and improve more easily when records of successes and failures are available.


6. UNRESOLVED ISSUES AND RISKS

6.1. Identify issues that have yet to be decided as of this draft of the plan. Note these as risks to the schedule, scope, or quality of the test effort.

6.2. Identify other risks that may have an impact on the success of the plan. Use the risks outlined in the course book and the attached speaker notes as a guideline to identifying common risks. Refer also to the Software Project Survival Guide (Steve McConnell), which includes a good list of risks for every phase of development. When assessing risk, don’t be optimistic; the quality of the test plan and the risk assessment is weakened by failure to assess risk realistically.


7. TEST PLAN REVIEW


7.1. Include plans for review of this test plan. Identify the parties to review and approve the document, either within the test group or with another set of developers or test engineers. Look at sample test plan checklists, such as that on p. 3.45 – 3.47 of the course book, or those in Software Project Survival Guide. Use ideas from these checklists to develop your own checklists, appropriate to the size and scope of the product. Identify here the checklist(s) that will be used.

7.2. Meet with developers and customers or customer representatives to ensure that the test plan meets their requirements.

What is Alpha Testing?

Alpha testing is the software prototype stage when the software is first able to run. It will not have all the intended functionality, but it will have core functions and will be able to accept inputs and generate outputs. An alpha test usually takes place in the developer's offices on a separate system.

Any software that will be run on customer equipment must first be reviewed and approved by EPRI Corporate Software Quality. Be sure to discuss any exceptions with the Software Quality Manager for your EPRI sector.

The EPRI project manager utilizes this prototype to evaluate and provide input to the developer as the design evolves. Although software is usually not distributed to users prior to beta testing, selected end-users may also be involved. Usually, the most complex or most used parts of the code are developed more completely in the alpha, in order to enable early resolution of design questions.

The project manager collaborates with the developer to determine specific goals for alpha testing, and to integrate the results into evolving project plans.

In-depth software reliability testing, installation testing, and documentation testing are not done at alpha test time, as the software is only a prototype. Alpha tester feedback forms are not used, although the developer does request feedback on specific aspects of the software.

Setting Customer Expectations

If customers will receive alpha software, they must understand that the product will be only minimally functional, and is likely to have problems. The required review by EPRI Corporate Software Quality checks for the most serious problems, but customers must still be prepared for unexpected or frustrating experiences.

A letter or e-mail accompanying alpha software is strongly recommended. This letter should make customers aware of:

• The software's limited functionality at this early stage
• The likelihood of experiencing problems

Their mission: Find problems and provide feedback

Quality Team Involvement

It is understood that alpha software is not fully functional. However, EPRI Corporate Software Quality tests all software that is to be sent to users or run on customer equipment so that possible serious errors can be removed. Examples of such errors are software behavior that compromises a user's system, or presence of a virus. This pre-testing of alpha software that is sent to users will protect the reputation of EPRI software with customers.

If any end-users or customers will receive software for alpha evaluation, or if software will be run on customer equipment, be sure to send it first to EPRI Corporate Software Quality. This prior review and approval is required. Be sure to discuss any exceptions with the Software Quality Manager for your EPRI sector.

In order not to delay project schedules, EPRI Corporate Software Quality:

• Uses e-mail notes rather than structured reports on alpha evaluations.
• Works with project managers to complete alpha evaluations within the time needed.

If desired, the EPRI Corporate Software Quality will provide additional early usability feedback on an alpha stage software code, such as feedback on the proposed navigation scheme, and the look and feel of the software.

In general, expect a 6-8 day turnaround time for EPRI software testing. However, during peak periods such as September through December, turnaround may take longer depending upon the testing queue. Testing time will also vary with software size and complexity.

Common Problems

• Customer expectations surrounding Alpha software have not been appropriately set, and customers are disappointed at the limited functionality and problems encountered.
• Software sent to users or run on customer equipment is not tested by EPRI Corporate Software Quality beforehand.
  

In Software Development there are a few different phases of software releases: Alpha Release, Beta Release, and Final Release. Let’s break it down...

Alpha Software

Alpha Software is the first major release of software for the public to use. Since it is the first major release, it often times has many unknown bugs.

If you use alpha software you should not be dependent on your computer as alpha software could possible cause unexpected things to happen with your system(s). It is also a good idea to have a general understanding of software testing.

Why release buggy software to the public? By releasing this software to the public, we can reach a substantial amount of people who will be willing to submit detailed information about their experience with the software as well as bug reports. Once we receive a bug report, we immediately create a fix or workaround for the next release of the software.

At SodaBush, we do a great deal of testing before we release any Alpha Software. This way we limit the amount of bugs in what should normally be a very buggy phase of software.

Beta Software

Beta Software is the second major release of software for the public to use. It works much the same as alpha software except that it should, hopefully, have fewer bugs, be more stable, and easier to work with. Keep in mind that no software ever is fully debugged. But, with your help, we can try to remove as many bugs as possible, creating a better experience for everyone using the software overall.

Final Software

Final Software is the final major release of software for the public to use. This is the phase which all software one day reaches. Final release software has been tested on several machines and has many, hopefully all, of the major bugs fixed.

During this phase of software development software becomes widely available as a freeware or shareware product.

Telling the Difference

On our site, we label all of our software accordingly. If you see “Alpha” after the name of a piece of software, it would mean that the software is currently alpha testing. (i.e. “Hyper Color v1.00 Alpha 1.”) Likewise with “Hyper Color v1.00 Beta 1,” you will know it is beta software. Software which does not indicate alpha or beta is a final version.

Bottom Line

What is the difference between Alpha, Beta, and Final software? When you boil it down, you can simply view the differences as...

Final = Least Buggy
Beta = Buggy
Alpha = Most Buggy

a stage of software development where the software is first tested for bugs by real users. In contrast to beta testing, alpha software is usually assumed to have some significant bugs or unimplemented portions.

Step Overview (http://www.epri.com/eprisoftware/processguide/testinteg.html)

The main objective of testing is to show with a high level of confidence that the software application meets the acceptance criteria: function, performance, usability, features, and capabilities.
The developer is expected to thoroughly test software for both functionality and usability before submitting it to EPRI for final acceptance testing.

Software testing tests the software itself, but it also tests documentation, operator or user interfaces, and reports. What is also indirectly tested are the development team, the technical and project management, the application development tools, and the software development rules and guidelines. In sum, much more is tested than just the software product itself.

Tasks In This Step

These are the tasks of the Test and Integration step. For more background, see the Background Material section below.

• Alpha Test by Developer - Describes alpha testing, and when EPRI Corporate Software Quality review is required.
• Alpha Test (Required if Alpha version going to customers) - Describes alpha testing, and when EPRI Corporate Software Quality review is required.
• Beta Test (Required for all software going to customers) - Overview, implementation tips, downloadable Beta Tester Feedback form.

Common Problems

• Alpha software that is run outside the developer's shop is not reviewed by EPRI Corporate Software Quality before it is sent to customers.
• Beta testing is not done, or is not complete enough to identify key problems.

Insufficient usability testing is done by the developer prior to submitting final software to EPRI Corporate Software Quality.

Unit Testing

1. Introduction
Unit testing is the testing of individual components (units) of the software. Unit testing is usually conducted as part of a combined code and unit test phase of the software lifecycle, although it is not uncommon for coding and unit testing to be conducted as two distinct phases.
The basic units of design and code in Ada, C and C++ programs are individual subprograms (procedures, functions, member functions). Ada and C++ provide capabilities for grouping basic units together into packages (Ada) and classes (C++). Unit testing for Ada and C++ usually tests units in the context of the containing package or class.
When developing a strategy for unit testing, there are three basic organisational approaches that can be taken. These are top down, bottom up and isolation. These three approaches are described and their advantages and disadvantages discussed in sections 2, 3, and 4 of this paper. The concepts of test drivers and stubs are used throughout this paper. A test driver is software which executes software in order to test it, providing a framework for setting input parameters, executing the unit, and reading the output parameters. A stub is an imitation of a unit, used in place of the real unit to facilitate testing.
An AdaTEST or Cantata test script comprises a test driver and an (optional) collection of stubs. Using AdaTEST or Cantata to implement the organisational approaches to unit testing presented in this paper is discussed in section 5.



2. Top Down Testing

2.1. Description
In top down unit testing, individual units are tested by using them from the units which call
them, but in isolation from the units called. The unit at the top of a hierarchy is tested first, with all called units replaced by stubs. Testing continues by replacing the stubs with the actual called units, with lower level units being stubbed. This process is repeated until the lowest level units have been tested. Top down testing requires test stubs, but not test drivers. Figure 2.1 illustrates the test stubs and tested units needed to test unit D, assuming that units A, B and C have already been tested in a top down approach.
A unit test plan for the program shown in figure 2.1, using a strategy based on the top
down organisational approach, could read as follows:

Step (1)
Test unit A, using stubs for units B, C and D.
Step (2)
Test unit B, by calling it from tested unit A, using stubs for units C and D.
Step (3)
Test unit C, by calling it from tested unit A, using tested units B and a stub for unit D.
Step (4)
Test unit D, by calling it from tested unit A, using tested unit B and C, and stubs for units
E, F and G. (Shown in figure 2.1).
Step (5)
Test unit E, by calling it from tested unit D, which is called from tested unit A, using tested
units B and C, and stubs for units F, G, H, I and J.
Step (6)
Test unit F, by calling it from tested unit D, which is called from tested unit A, using tested
units B, C and E, and stubs for units G, H, I and J.
Step (7)
Test unit G, by calling it from tested unit D, which is called from tested unit A, using tested
units B, C, E and F, and stubs for units H, I and J.
Step (8)
Test unit H, by calling it from tested unit E, which is called from tested unit D, which is
called from tested unit A, using tested units B, C, E, F and G, and stubs for units I and J.
Step (9)
Test unit I, by calling it from tested unit E, which is called from tested unit D, which is
called from tested unit A, using tested units B, C, E, F, G and H, and a stub for units J.
Step (10)
Test unit J, by calling it from tested unit E, which is called from tested unit D, which is
called from tested unit A, using tested units B, C, E, F, G, H and I.

2.2. Advantages

Top down unit testing provides an early integration of units before the software integration phase. In fact, top down unit testing is really a combined unit test and software integration strategy. The detailed design of units is top down, and top down unit testing implements tests in the sequence units are designed, so development time can be shortened by overlapping unit testing with the detailed design and code phases of the software lifecycle. In a conventionally structured design, where units at the top of the hierarchy provide high level functions, with units at the bottom of the hierarchy implementing details, top down unit testing will provide an early integration of 'visible' functionality. This gives a very requirements oriented approach to unit testing. Redundant functionality in lower level units will be identified by top down unit testing, because there will be no route to test it. (However, there can be some difficulty in distinguishing between redundant functionality and untested functionality).

2.3. Disadvantages

Top down unit testing is controlled by stubs, with test cases often spread across many stubs. With each unit tested, testing becomes more complicated, and consequently more expensive to develop and maintain. As testing progresses down the unit hierarchy, it also becomes more difficult to achieve the good structural coverage which is essential for high integrity and safety critical applications,
and which are required by many standards. Difficulty in achieving structural coverage can also lead to a confusion between genuinely redundant functionality and untested functionality. Testing some low level functionality, especially error handling code, can be totally impractical. Changes to a unit often impact the testing of sibling units and units below it in the hierarchy. For example, consider a change to unit D. Obviously, the unit test for unit D would have to change and be repeated. In addition, unit tests for units E, F, G, H, I and J, which use the tested unit D, would also have to be repeated. These tests may also have to change themselves, as a consequence of the change to unit D, even though units E, F, G, H, I and J had not actually changed. This leads to a high cost associated with retesting when changes are made, and a high maintenance and overall lifecycle cost. The design of test cases for top down unit testing requires structural knowledge of when the unit under test calls other units. The sequence in which units can be tested is constrained by the hierarchy of units, with lower units having to wait for higher units to be tested, forcing a 'long and thin' unit test phase. (However, this can overlap substantially with the detailed design and code phases of the software lifecycle).
The relationships between units in the example program in figure 2.1 is much simpler than would be encountered in a real program, where units could be referenced from more than one other unit in the hierarchy. All of the disadvantages of a top down approach to unit testing are compounded by a unit being referenced from more than one other unit.

2.4. Overall

A top down strategy will cost more than an isolation based strategy, due to complexity of testing units below the top of the unit hierarchy, and the high impact of changes. The top down rganisational approach is not a good choice for unit testing. However, a top down approach to the integration of units, where the units have already been tested in isolation, can be viable.

3. Bottom up Testing


3.1. Description

In bottom up unit testing, units are tested in isolation from the units which call them, but using the actual units called as part of the test. The lowest level units are tested first, then used to facilitate the testing of higher level units. Other units are then tested, using previously tested called units. The process is repeated until the unit at the top of the hierarchy has been tested. Bottom up testing requires test drivers, but does not require test stubs. Figure 3.1 illustrates the test driver and tested units needed to test unit D, assuming that units E, F, G, H, I and J have already been tested in a bottom up approach.

A unit test plan for the program shown in figure 3.1, using a strategy based on the bottom
up organisational approach, could read as follows:
Step (1)
(Note that the sequence of tests within this step is unimportant, all tests within step 1 could
be executed in parallel.)
Test unit H, using a driver to call it in place of unit E;
Test unit I, using a driver to call it in place of unit E;
Test unit J, using a driver to call it in place of unit E;
Test unit F, using a driver to call it in place of unit D;
Test unit G, using a driver to call it in place of unit D;
Test unit B, using a driver to call it in place of unit A;
Test unit C, using a driver to call it in place of unit A.
Step (2)
Test unit E, using a driver to call it in place of unit D and tested units H, I and J.
Step (3)
Test unit D, using a driver to call it in place of unit A and tested units E, F, G, H, I and J.
(Shown in figure 3.1).
Step (4)
Test unit A, using tested units B, C, D, E, F, G, H, I and J.


3.2. Advantages

Like top down unit testing, bottom up unit testing provides an early integration of units before the software integration phase. Bottom up unit testing is also really a combined unit test and software integration strategy. All test cases are controlled solely by the test driver, with no stubs required. This can make unit tests near the bottom of the unit hierarchy relatively simple. (However, higher level unit tests can be very complicated). Test cases for bottom up testing may be designed solely from functional design information, requiring no structural design information (although structural design information may be useful in achieving full coverage). This makes the bottom up approach to unit testing useful when the detailed design documentation lacks structural detail. Bottom up unit testing provides an early integration of low level functionality, with higher level functionality being added in layers as unit testing progresses up the unit hierarchy. This makes bottom up unit testing readily compatible with the testing of objects.

3.3. Disadvantages

As testing progresses up the unit hierarchy, bottom up unit testing becomes more complicated, and consequently more expensive to develop and maintain. As testing progresses up the unit hierarchy, it also becomes more difficult to achieve good structural coverage. Changes to a unit often impact the testing of units above it in the hierarchy. For example, consider a change to unit H. Obviously, the unit test for unit H would have to change and be repeated. In addition, unit tests for units A, D and E, which use the tested unit H, would also have to be repeated. These tests may also have to hange themselves, as a consequence of the change to unit H, even though units A, D and E had not actually changed. This leads to a high cost associated with retesting when changes are made, and a high maintenance and overall lifecycle cost. The sequence in which units can be tested is constrained by the hierarchy of units, with higher units having to wait for lower units to be tested, forcing a 'long and thin' unit test phase. The first units to be tested are the last units to be designed, so unit testing cannot overlap with the detailed design phase of the software lifecycle. The relationships between units in the example program in figure 2.2 is much simpler than would be encountered in a real program, where units could be referenced from more than one other unit in the hierarchy. As for top down unit testing, the disadvantages of a bottom up approach to unit testing are compounded by a unit being referenced from more than one other unit.


3.4. Overall

The bottom up organisational approach can be a reasonable choice for unit testing, particularly when objects and reuse are considered. However, the bottom up approach is biased towards functional testing, rather than structural testing. This can present difficulties in achieving the high levels of structural coverage essential for high integrity and safety critical applications, and which are required by many standards. The bottom up approach to unit testing conflicts with the tight timescales required of many software developments. Overall, a bottom up strategy will cost more than an isolation based strategy, due to complexity of testing units above the bottom level in the unit
hierarchy and the high impact of changes.

4. Isolation Testing

4.1. Description

Isolation testing tests each unit in isolation from the units which call it and the units it calls.Units can be tested in any sequence, because no unit test requires any other unit to have been tested. Each unit test requires a test driver and all called units are replaced by stubs. Figure 4.1 illustrates the test driver and tested stubs needed to test unit D.

A unit test plan for the program shown in figure 4.1, using a strategy based on the isolation
organisational approach, need contain only one step, as follows:
Step (1)
(Note that there is only one step to the test plan. The sequence of tests is unimportant, all
tests could be executed in parallel.)
Test unit A, using a driver to start the test and stubs in place of units B, C and D;
Test unit B, using a driver to call it in place of unit A;
Test unit C, using a driver to call it in place of unit A;
Test unit D, using a driver to call it in place of unit A and stubs in place of units E, F and G,
(Shown in figure 3.1);

Test unit E, using a driver to call it in place of unit D and stubs in place of units H, I and J; Test unit F, using a driver to call it in place of unit D; Test unit G, using a driver to call it in place of unit D;
Test unit H, using a driver to call it in place of unit E; Test unit I, using a driver to call it in place of unit E; Test unit J, using a driver to call it in place of unit E.

4.2. Advantages

It is easier to test an isolated unit thoroughly, where the unit test is removed from the complexity of other units. Isolation testing is the easiest way to achieve good structural coverage, and the difficulty of achieving good structural coverage does not vary with the position of a unit in the unit hierarchy. Because only one unit is being tested at a time, the test drivers tend to be simpler than for bottom up testing, while the stubs tend to be simpler than for top down testing. With an isolation approach to unit testing, there are no dependencies between the unit tests, so the unit test phase can overlap the detailed design and code phases of the software lifecycle. Any number of units can be tested in parallel, to give a 'short and fat' unit test phase. This is a useful way of using an increase in team size to shorten the overall time of a software development.
A further advantage of the removal of interdependency between unit tests, is that changes to a unit only require changes to the unit test for that unit, with no impact on other unit tests. This results in a lower cost than the bottom up or top down organisational approaches, especially when changes are made. An isolation approach provides a distinct separation of unit testing from integration testing,
allowing developers to focus on unit testing during the unit test phase of the software lifecycle, and on integration testing during the integration phase of the software lifecycle. Isolation testing is the only pure approach to unit testing, both top down testing and bottom up testing result in a hybrid of the unit test and integration phases. Unlike the top down and bottom up approaches, the isolation approach to unit testing is not affected by a unit being referenced from more than one other unit.

4.3. Disadvantages

The main disadvantage of an isolation approach to unit testing is that it does not provide any early integration of units. Integration has to wait for the integration phase of the software lifecycle. (Is this really a disadvantage?). An isolation approach to unit testing requires structural design information and the use of both stubs and drivers. This can lead to higher costs than bottom up testing for units near the bottom of the unit hierarchy. However, this will be compensated by simplified testing for units higher in the unit hierarchy, together with lower costs each time a unit is changed.

4.4. Overall

An isolation approach to unit testing is the best overall choice. When supplemented with an appropriate integration strategy, it enables shorter development timescales and provides the
lowest cost, both during development and for the overall lifecycle. Following unit testing in isolation, tested units can be integrated in a top down or bottom up sequence, or any convenient groupings and combinations of groupings. However, a bottom up integration is the most compatible strategy with current trends in object oriented and object biased designs. An isolation approach to unit testing is the best way of achieving the high levels of structural coverage essential for high integrity and safety critical applications, and which are required by many standards. With all the difficult work of achieving good structural coverage achieved by unit testing, integration testing can concentrate on overall functionality and the interactions between units.

5. Using AdaTEST and Cantata

A unit test will be repeated many times throughout the software lifecycle, both during the development part of the lifecycle and later during maintenance. A test harness such as AdaTEST or Cantata can be used to automate unit tests, resulting in unit tests which are easy to repeat and have a low cost of repetition, and reducing risk of human error.
AdaTEST and Cantata test scripts comprise a test driver and an (optional) collection of stubs. daTEST and Cantata can be used with any of the organisational approaches to unit testing described by this paper, or with any combination of organisational approaches, enabling the developer to adopt a testing strategy best suited to the needs of a project. Two related papers are available from IPL:

• Achieving Testability when using Ada Packaging and Data Hiding Methods

• Testing C++ Objects

The paper "Testing C++ Objects" also provides detail about how the complexity of separate class and containment hierarchies leads to problems with a bottom up approach to unit testing. It describes how an isolation approach to unit testing is the only practical way to deal with separate class and containment hierarchies.

6. Conclusion

In practice, it is unlikely that any single approach to unit testing can be used exclusively. Typically, an isolation approach to unit testing is modified with some bottom up testing, in which the called units are a mixture of stubs and already tested real units. For example, it makes more sense for a mathematical function to be used directly, provided that it has already been tested and is unlikely to change. The recommended strategy is:

• Base your unit test strategy on the isolation approach, then integrate groups of
tested units bottom up.

• Compromise by incorporating some bottom up where it is convenient (for example, using real perators, mathematical functions, string manipulation etc.), but remember the potential impact of changes.

This will result in the lowest cost; both to develop unit tests, and to repeat and maintain tests following changes to units, whilst also facilitating the thorough test coverage necessary to achieve reliable software. Remember that unit testing is about testing units, and that integration testing is about testing the interaction between tested units.

Automating Tests with Segue Silk

This article, originally written in 1997 and updated in 2003, discusses architecting and designing your Silk scripts and also touches on some of the more advanced features of Silk such as custom classes and methods, data driven testing, and the @ operator.

Up Front Design
Test automation projects are programming projects. Period. No matter how cool the test tool interface looks, all test automation is programming at some level or another. This isn't a statement about Segue SilkTest in particular: test automation is still programming no matter which of the GUI test automation tools you choose (Mercury, Rational, Compuware, etc.).
Because many test automation tools do a good job of hiding the complexity, it's tempting to dive right in and begin automating without having any kind of strategy or plan in mind. In fact, if you've never used Segue Silk before you might want to do just that so you can begin to understand the capabilities and limitations of the tool. Go ahead. Experiment. Create some scripts, then try to run them against subsequent builds.
I'll wait here.
Oh good. You're back. What you probably found is that you could create scripts quickly. It didn't take long before you could generate an impressive amount of activity on the screen. Then when you tried to play those same scripts on later builds everything broke. You may even have had to start over. Without even a little up front design, test automation scripts are too fragile, too difficult to maintain.
So let's back up a step and talk about the up front design work you should do before beginning a serious test automation project.
I strongly recommend starting with a test automation strategy document and a test design specification. The strategy document will address questions such as

What tool(s) are we using?
What's our general approach to test automation?
What products or projects will we use test automation on?
What is the goal of automation?
How will we measure progress toward that goal?
What are some of the benefits we expect to reap?
What are some of the risks we foresee?
What will it take to make test automation a success?
Who will be responsible for creating, maintaining, and executing the automated scripts?

This document should be short: somewhere between 1 - 5 pages. The point of writing the document is to explain the test automation effort at a high level. This will be the most widely read test automation document. When the VP of Development wants to know what you're doing, you can hand him this document.
Next you want to begin thinking through some of the design issues with your test automation infrastructure: a framework that describes the application including reusable classes, methods, and functions. At the beginning of your automation effort, plan to spend the vast majority of your automation resources building your infrastructure. The investment will pay off in the end. Creating new scripts will be easier if there is a firm foundation on which they can rest. Existing scripts will be more maintainable if common objects and code is located in just one place rather than copied and pasted all over.
It's important to note that while it is a good idea to design your test automation infrastructure at the beginning, you do not need to implement it all at once. You can begin with a basic framework and add to it as you see the need. As long as you don't change the logical names of your window objects, you won't have to rewrite your test scripts to accommodate the changes to your infrastructure. In SilkTest that means thinking about:

Your application framework
Addition classes, methods, and functions you create
Your test cases, scripts, and suites of scripts
The file structure for your include files, scripts, etc.

Create a separate document to specify these kinds of details. Note that I am not suggesting that you design and specify all aspects of your infrastructure up front. These are things to think about now and throughout your test automation effort. This is a living document. You'll continue updating the design specification throughout the automation project. But by designing at least part of the infrastructure up front, you have a roadmap to follow, a checklist of programming tasks, and you can better measure your progress against your plan.
As with writing any kind of specification writing, there is a balance point between not enough detail and too much. Over-specifying your design may result in a document that is so long, no one will read it. Furthermore, the more implementation detail you put into the design specification, the more likely the design specification will be out of date the minute you begin implementation. Implementation details that sounded good when you were writing may not work nearly as well in practice. So make your specification as brief as possible while still covering the essential elements of the design.

Creating Your Application Framework

The SilkTest wizard will walk you through creating your application framework. This is the point in the project where you begin creating the abstract layer between your software under test and your test scripts. Not sure what I mean by abstract layer? I explain it in my article Making the Right Choice (PDF). Not sure you need an abstract layer? Trust me: you do. The test tools on the market that don't support or encourage the notion of an abstract layer are doing both themselves and their users a disservice. In a nutshell, the abstract layer is one of the keys to making your scripts maintainable.
Although you could declare all the windows in your application up front, I recommend taking a staged approach. By declaring only the windows you need at any given time, you'll find problems with your declarations earlier. And there will be problems. Segue Silk is a powerful tool, but nothing is magic. I've had projects where halfway through declaring my windows I discovered that I'd chosen the wrong identification method (chose Window ID when I should have chosen Index, etc.). By declaring only the handful of windows I was working on up front, I had less work to redo than if I'd done all my window declarations up front.
Once you have declared at least some of the windows in your application, you might want to look at ways of refining your framework to make it more maintainable, flexible, and robust. Some suggestions, including creating custom classes, methods, and functions, appear in the sections below.

Determining Your File Structure

By now you've started creating a framework. You probably have everything in a single include (.inc) file.
If you are testing a reasonably complex application, you may want to separate your framework into different files to minimize the amount of redundant maintenance you need to do. It can be difficult to determine where one file should end and another begin. Here are some lessons learned the hard way:
Do put all declarations associated with the same module or executable in a single file. This makes it easier to manage the include file for the module or executable.
If you are testing a series of executables with common elements, create a separate file to hold common elements. Each module will then require (at least) the two include files: its own and the common include file used by all modules.
Do not put functions into the same files with object declarations, particularly if you are working on more than one release of your application at a time. The executable statements in functions often stay the same from release to release, but object declarations may differ, particularly if your developers are fond of rearranging the interface on a regular basis. It's a good idea to pull functions into a general use library file. Include classes and their associated methods in the general use library file.
Consider creating separate library files for each executable or module if one common library file gets too long or if the executables are on different release schedules.
Make sure that the module/executable include files are dependent on the general include file, but not vice versa.
In general, the goal of any file structure is to minimize the number of places in which you need to make a particular change. You need to balance granularity with ease-of-use: separating everything into different files may give you greater control over what code is shared between projects and releases, but it also makes it very difficult to track which files contain what code.
Finally, just in case you haven't already done this: get source control for your tests. Also make sure that everyone creating and editing scripts knows how to use the source control system. Source control for scripts is vital regardless of which automated test tool you choose.

Creating Classes and Class Hierarchies

In order to make your infrastructure more maintainable, it's a good idea to refine your framework by creating classes for similar windows. If you don't have an object-oriented programming background, think of classes as a template. Creating a class to describe common elements in your program allows you to specify the common details just once.

In the design stage, you want to identify good candidates for classes. To do this:
Identify windows that are visually similar.
Do they serve a similar purpose?
Do they contain similar (or identical) fields and buttons?
Use the Record Window Declarations menu option in SilkTest to look at the underlying window declarations. Windows that look alike on the surface don't always look alike to SilkTest.
When you record the windows, do the resulting declarations look similar?
What are the differences? Different logical names don't matter, different tags matter a little, radically different structures will make classing a given window virtually impossible.
Once you've identified candidates for custom classes, you can create your custom classes using the Classes option on the Record menu, or you can create them manually using the winclass keyword like so:
winclass CustomDialog : DialogBox
(See the SilkTest documentation for details.)

The most straightforward use of classes is to reduce redundant window declarations. Creating classes also allows you to write methods that will be inherited by all windows that are members of the class.
You can also create a hierarchy of classes so related but different objects can still inherit characteristics from a common parent. Hierarchies of classes are useful if:
You have a number of windows that are similar but that fall into different categories. For example, if there are a number of places in your application where you get a File Open dialog, and a number of places where you get a Save dialog. Both dialogs may have elements in common (list of directories, file name field, etc.), and unique elements (Save button vs. Open, etc.).
A similar window appears in several executables or modules. You might want to create a parent class for all the related windows in the main library file, then create separate classes for each executable or module.

Designing Functions and Methods

Chances are there are actions you perform repeatedly while testing. For example, if you are testing a word processor, you probably have to open files over and over again. This kind of action is an ideal candidate for a function or a method. By pulling out common code into reusable functions and methods you improve the maintainability of your test scripts and also improve productivity.
Creating general use methods and functions is a good way to reduce the size of your scripts and increase the reusability of your code. When writing methods and functions:
Keep the method or function as generic as possible by passing parameters.
Consider passing a record rather than separate variables if multiple variables are needed.
With methods, remember that you can add more functionality at the window or subclass level by using the derived::method_name() syntax.
With functions, remember that you can always call functions from other functions. Consider creating sets of functions that can work together rather than creating a single function that does everything but is less flexible.

Passing records instead of separate variables makes the method or function more maintainable. If an additional variable is needed later, adding it to the record rather than as a new parameter to the method or function means that the programming interface to the method or function doesn't change.

Is It a Method or a Function?

One of the major design decisions is whether to make a series of actions a method or a function. The difference between a function and a method is that a method can only act on an object. You can create methods for individual objects as well as classes. Functions can act on objects as well, but more often you'll create functions to act on data or collections of objects.
In deciding whether a series of actions should become a function or a method, consider the advantages of methods over functions:
The this keyword allows you to reference otherwise ambiguous objects
Inheritance and polymorphism give you a cleaner programming interface
Because a method can only act on an object, you have built in error checking

Using the This Keyword: The this keyword allows you to refer to an object without knowing the name of the object. For example, in the generic word processing application mentioned above, you might want to have a method to find a file or directory in any browser window. Since the method is supposed to work in any browser window, you won't know ahead of time the name of the browser window. Simply substitute the word "this" whenever you would use the name of the window. For example, you might end up with a line of code in your method like:
this.FileList.Select(sFileName)

Using Inheritance and Polymorphism: When you add a method to a class, all objects declared as members of that class automatically have access to the method; they inherit it. This means that you only need to write a method once and you can use it throughout your tests. However, there may be times when you need to modify a method for one member of the class. You can define a method for a particular object with the same name as the class method and different functionality. The new method overrides the class method. This is called polymorphism.
Note: SilkTest has imposed a limitation on inheritance. You cannot inherit custom methods within the SilkTest-supplied hierarchy of classes. For example, the SilkTest class DialogBox is derived from the SilkTest class ChildWin. If you create a method for the ChildWin class, you would expect members of DialogBox to inherit it, but they don't. In order to use the same method for both the ChildWin and DialogBox classes, you must create the method twice, once for each class. This limitation is not a bug: Segue introduced the limitation to make it easier to debug script problems involving built-in classes.
Built in Error Checking: When you create a method, you can only call that method when there is an object on which the method can act. When you create a function, you can call the function at any time. QA Partner's error recovery system ensures that your script will recover if an object referenced in the function or method is not found. But if you use methods where it makes sense, QA Partner will recognize the error earlier with a method, because the statement calling the method includes the name of the object.

Using the "@" Redirection Operator

For all you C/C++ programmers out there, the "@" operator is the closest thing to a pointer in 4Test. You can use @ to dynamically refer to a field on a form or in a record or as a function pointer, to dynamically decide which function to call based on the value of a variable. This is very powerful and allows you to create very general functions, methods, and test cases.

For example, consider the following trivial example:
// create a record datatype with two fieldstype DATARECORD is record STRING sField1 INTEGER iField2

main() // use our new record datatype DATARECORD rData = {"wombat", 42} STRING sIt // iterate thru the record fields for each sIt in FieldsOfRecord(DATARECORD) print("Field: {sIt}, Val: {rData.@sIt}")
The final line prints out both sIt and @sIt. The @ operator causes the value of sIterate to be substituted. The results of the above script are:
Field: sField1, Val: wombatField: iField2, Val: 42
To use the @ operator to automatically fill in the fields on a form, simply create a record with fields that have the same name as the fields on the form. Then use something like the following to iterate through the fields and place their contents in the fields on the form:
Populate(DATARECORD rData) STRING sIt for each sIt in FieldsOfRecord(DATARECORD) MyWindow.@sIt.SetText(rData.@sIt)
You can also use the @ operator to dynamically call functions according to the value of a variable. The syntax to do this looks like:
@(VariableName)(Parameters go here)
For example, the following trivial example function calls the function specified by the sFunctionName variable using the parameter specified by the aData variable and returns the value returned by the function called.
ANYTYPE MyFunc(STRING sFuncName, ANYTYPE aData) return @(sFuncName)(aData)
Although trivial, this example shows you how you could write a wrapper function to call one of several other functions according to the value in a variable.

Generalizing Your Test Cases: Use Data Driven Testing

Segue SilkTest now supports data driven testing as of the 6.0 release. This is HUGE news. In the past, you had to write the code to make your scripts open and read a file. While the code wasn't that bad to write, it was something of a pain and it discouraged people from getting the most out of their test automation effort.
Now, however, you can use the Data Drive Testcase option on the Tools menu to make any test case data driven. Further, the test case can pull the data from a spreadsheet or a database, making it much easier to maintain your test data.
For more information in Data Driven Testing with SilkTest 6.0 see the SilkTest documentation.

Tips for Incorporating SilkTest Scripts into Your Testing Process

All the automation in the world won't help if the automated and manual efforts are not integrated. Here are some tips for incorporating results into your testing process:

Make the first field in test data records passed to a test case a description of the test case. This gives you traceability from the automated script to the test documentation.
Include the name of the automated test script or test case in the test documentation for each test. This gives you traceability from the test documentation to the automated script.
Use the LogError, LogWarning, and print functions to add important information to the results file. This will help both in tracking results and in debugging problems.
Don't automatically log any error encountered to your problem tracking system. Test automation can point you in the general direction of a problem but further investigation is usually required to identify the actual bug.
If a different group of testers is running the automated tests than creating them, ensure those testers get training in SilkTest too. It will help them demystify their results and find false negatives (and false positives!) more quickly.