Testing Strategy Template

Introduction

One of the basic principles of the testing process is to plan for testing early in the development life cycle. For large projects, this starts in the analysis phase with the formulation of the Testing Strategy, which you later translate into the lower-level Testing Plan.

The Testing Strategy provides overall guidance and direction for the rest of the testing process. Preparing a Testing Strategy helps the project team think through the work at a high level, concentrating on the requirements of the testing events, the approach that will be used, and the resources necessary to complete the testing.

Testing stages in the Testing Strategy include:

· Unit Testing—The first testing is to ensure that the component meets expectations in terms of features and functionality. In almost all cases, the person that develops the component also does unit testing.

· Integration Testing—After the components have been unit tested, they are executed together. This testing verifies that the interfaces work correctly and that data is processed in its entirety as expected.

· System Testing—This is where the most rigorous testing takes place, to ensure that the solution meets the technical and environmental needs of the customer and that the solution behaves as it should. Many individual tests can be performed under the system test umbrella. These include stress testing, usability testing, security testing, performance testing, recovery testing, and multisite testing.

· Acceptance Testing—This is the final set of tests to ensure that the solution meets the business objectives and requirements. The business customers may be responsible for this testing. It requires their active participation to work with the solution, as they will when it moves to production status.

This template is designed to help you draft your own Testing Strategy. Note that the shaded text in this template is explanatory only; you should delete these comments and the example text from your version of the document.

Project Overview

Briefly discuss the project background and objectives. The appropriate information can be copied from the Project Definition document.

Business Risks

These are high-level risks of the project that will affect the overall testing strategy. For instance, the risk of doing business on the Internet may drive the need for rigorous system tests of firewalls, technical architecture, and security. The risks can be classified in terms of high, medium, and low, depending on the nature and impact of the problem. For each high and medium risk, identify which elements in the overall testing approach will help ensure that the potential problem does not occur.

Risk Area	Level (H/M/L)		Risk Plan
Example Since this is our first system doing e-commerce on the Internet, there is a concern that security will not be tight enough.		H	· Rigorously test security with our initial partners. · Bring in subject matter experts to validate our security design. · Set up tests for people to try to break in to the system or get around security features.

Testing Milestones

This section contains the overall schedule at a major milestone level. The associated details will be broken down further in the Testing Plan. Since this document is created in the analysis phase, these dates are subject to later revision.

Testing Milestone	Date Completed
Example
Testing Strategy	06/01/YYYY
Testing Plan	07/01/YYYY
Unit Testing	10/01/YYYY
Integration Testing	11/01/YYYY

Testing Environment

Think through the technologies and facilities needed for the testing process. If the overall testing environment needs are understood up front, it will be easier to put the environment in place after the Test Plan is created. In addition, some parts of the environment may not be available and may need to be planned for and acquired well in advance.

Example

The developers will conduct unit testing from their offices, with no additional software required. Integration testing will also be conducted in the developer’s workspace, based on who is assigned to coordinate the major system components. System testing and user testing will be coordinated in the project war room. The war room will have six desktop machines installed, with Internet access and our standard testing tools. For system testing, we will also need access to the servers in the operations center….

Testing Approach

Describe the testing process at a high level, including how you will conduct unit testing, integration testing, system testing, and acceptance testing. This is where fundamental decisions are made about the type of testing that makes sense for your project. For instance, if you are implementing a packaged solution, the approach may start in system testing, with the vendor providing close support. If you are doing iterative development cycles, the testing approach will reflect this overall development life cycle. For system testing, define the major testing events such as stress testing, security testing, disaster-recovery testing, usability testing, and response-time testing.

Unit Testing Approach

Describe the approach for conducting unit testing to ensure that each component works according to specifications.

Example

Each developer is responsible for unit testing his or her programs as they are completed. The project manager will consider each assigned component to be completed when it has been developed and successfully unit tested. The developers will keep all test data, along with a document stating the expected results, so that these same cases can be used in the integration tests….

Integration Testing Approach

Integration testing validates that the individually tested components work correctly together.

Example

Specific developers will be assigned to coordinate the integration tests. The project manager will assign the specific parts of the system to be tested. The prior test cases created during unit testing will be used as the basis for integration testing, with new test cases added to fully test all the interfaces and dependencies. All databases should be loaded with a set of test data so that all logic can be initially tested. Reports will be created from the files that would normally be sent to our customers so that the data can be validated….

System Testing Approach

System testing is the time for fully testing all features and functions and for ensuring that production conditions are fully tested. This can include stress testing, requirements validation, security testing, customer testing, testing of the training material, etc.

Example

System testing will include many separate tests to ensure that the application will work as expected in production. The test cases from integration testing will be brought forward as the basis for system testing. During system testing, we will conduct a stress test to generate the number of transactions we feel are normal for one day and feed them through the online system within an hour. This will be in an environment that mirrors the ultimate production environment. As many tests as possible will be conducted with a subset of our trading partners, so that we can simulate the actual Web transactions going from our application to our partner’s environment….

Acceptance Testing Approach

Describe the approach for conducting business unit customer acceptance testing to ensure that the system satisfies requirements for operational functionality, procedures, and usability.

Example

The customers will have access to all of the test data that has been generated previously, but they will ultimately be responsible for generating whatever sets of transactions are required to test the system to their satisfaction. For all intents and purposes, the system will be placed in a production status, including hardware and software. They will coordinate testing with the partners as well and ensure that the resulting transactions are as they expect….

Optional Sections

The following sections can be added to the Testing Strategy if it provides more perspective and value to the reader or if the information is needed within your company.

Business Assumptions

Document any assumptions surrounding the testing strategy.

· Assumption #1

· Assumption #2

· Assumption #3

· Etc.

Testing Objectives

The project should have a set of overall objectives to be achieved. Usually these overall objectives can be broken down into a specific set of testing objectives that will be accomplished through the testing process. You can state these in terms of the overall testing results, or you can break them down into objectives for the unit, integration, system, and acceptance testing. (Remove this comment section from final document.)

Testing Organization

In this section, define the high-level organization structure for testing. This might be especially helpful for large projects where many people will be involved in various aspects of the testing process. It also would be good if an independent testing organization has responsibility for portions of the testing process. An organization chart works well here, or you can just list the various groups involved. At this point, you may not have specific names, but you should know what groups will be needed.

Testing Event	Organizations/People Involved
Example
Unit Testing	Developers
Integration Testing	Developers, DBAs, project manager
System Testing	Testing organization, developers, partners, server administration,…
Acceptance Testing	Developers, customers,…

Test-Design Specification

1. Purpose.

To specify refinements of the test approach and to identify the features to be tested by this design and its associated tests.

2. Outline.
A test-design specification shall have the following structure:
(1) Test-design-specification identifier
(2) Features to be tested
(3) Approach refinements
(4) Test identification
(5) Feature pass/fail criteria

The sections shall be ordered in the specified sequence. Additional sections may be included at the end. If some or all of the content of a section is in another document, then a reference to that material may be listed in place of the corresponding content. The referenced material must be attached to the test-design specification or available to users of the design specification.


Details on the content of each section are contained in the following sections.


2.1 Test-Design-Specification Identifier.

Specify the unique identifierassigned to this test-design specification. Supply a reference to the associated test plan, if it exists.


2.2 Features to be Tested.

Identify the test items and describe the features and combinations of features which are the object of this design specification. Other features may be exercised, but need not be identified.
For each feature or feature combination, a reference to its associated requirements in the
item requirement specification or design description should be included.


2.3 Approach Refinements.

Specify refinements to the approach described in the test plan. Include specific test techniques to be used. The method of analyzing test results should be identified (for example, comparator
programs or visual inspection).

Specify the results of any analysis which provides a rationale for test-case selection. For example, one might specify conditions which permit a determination of error tolerance (for example, those conditions which distinguish valid inputs from invalid inputs).

Summarize the common attributes of any test cases. This may include input constraints that must be true for every input in the set of associated test cases, any shared environmental needs, and any shared special procedural requirements, and any shared case dependencies.



2.4 Test Identification.

List the identifier and a brief description of each test case associated with this design. A articular
test case may be identified in more than one test design specification. List the identifier and a brief description of each procedure associated with this test-design specification.



2.5 Feature Pass/Fail Criteria.

Specify the criteria to be used to determine whether the feature or feature combination has passed or failed.

Guidelines For Matrics Calculations

Metrics Used In Testing
In this tutorial you will learn about metrics used in testing, The Product Quality Measures -

1. Customer satisfaction index,
2. Delivered defect quantities,
3. Responsiveness (turnaround time) to users,
4. Product volatility,
5. Defect ratios,
6. Defect removal efficiency,
7. Complexity of delivered product,
8. Test coverage,
9. Cost of defects,
10. Costs of quality activities,
11. Re-work,
12. Reliability and Metrics for Evaluating Application System Testing.


The Product Quality Measures:

1. Customer satisfaction index
This index is surveyed before product delivery and after product delivery (and on-going on a periodic basis, using standard questionnaires).The following are analyzed:
Number of system enhancement requests per year
Number of maintenance fix requests per year
User friendliness: call volume to customer service hotline
User friendliness: training time per new user
Number of product recalls or fix releases (software vendors)
Number of production re-runs (in-house information systems groups)

2. Delivered defect quantities
They are normalized per function point (or per LOC) at product delivery (first 3 months or first year of operation) or Ongoing (per year of operation) by level of severity, by category or cause, e.g.: requirements defect, design defect, code defect, documentation/on-line help defect, defect introduced by fixes, etc.

3. Responsiveness (turnaround time) to users
Turnaround time for defect fixes, by level of severity
Time for minor vs. major enhancements; actual vs. planned elapsed time

4. Product volatility
Ratio of maintenance fixes (to repair the system & bring it into compliance with specifications), vs. enhancement requests (requests by users to enhance or change functionality)

5. Defect ratios
Defects found after product delivery per function point.
Defects found after product delivery per LOC
Pre-delivery defects: annual post-delivery defects
Defects per function point of the system modifications

6. Defect removal efficiency
Number of post-release defects (found by clients in field operation), categorized by level of severity
Ratio of defects found internally prior to release (via inspections and testing), as a percentage of all defects All defects include defects found internally plus externally (by customers) in the first year after product delivery

7. Complexity of delivered product
McCabe's cyclomatic complexity counts across the system
Halstead’s measure
Card's design complexity measures
Predicted defects and maintenance costs, based on complexity measures

8. Test coverage
Breadth of functional coverage
Percentage of paths, branches or conditions that were actually tested
Percentage by criticality level: perceived level of risk of paths
The ratio of the number of detected faults to the number of predicted faults.

9. Cost of defects
Business losses per defect that occurs during operation
Business interruption costs; costs of work-arounds
Lost sales and lost goodwill
Litigation costs resulting from defects
Annual maintenance cost (per function point)
Annual operating cost (per function point)
Measurable damage to your boss's career

10. Costs of quality activities
Costs of reviews, inspections and preventive measures
Costs of test planning and preparation
Costs of test execution, defect tracking, version and change control
Costs of diagnostics, debugging and fixing
Costs of tools and tool support
Costs of test case library maintenance
Costs of testing & QA education associated with the product
Costs of monitoring and oversight by the QA organization (if separate from the development and test organizations)

11. Re-work
Re-work effort (hours, as a percentage of the original coding hours)
Re-worked LOC (source lines of code, as a percentage of the total delivered LOC)
Re-worked software components (as a percentage of the total delivered components)

12. Reliability
Availability (percentage of time a system is available, versus the time the system is needed to be available)
Mean time between failure (MTBF).
Man time to repair (MTTR)
Reliability ratio (MTBF / MTTR)
Number of product recalls or fix releases
Number of production re-runs as a ratio of production runs


Metrics for Evaluating Application System Testing:

Metric = Formula

Test Coverage = Number of units (KLOC/FP) tested / total size of the system. (LOC represents Lines of Code)

Number of tests per unit size = Number of test cases per KLOC/FP (LOC represents Lines of Code).

Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria

Defects per size = Defects detected / system size

Test cost (in %) = Cost of testing / total cost *100

Cost to locate defect = Cost of testing / the number of defects located

Achieving Budget = Actual cost of testing / Budgeted cost of testing

Defects detected in testing = Defects detected in testing / total system defects

Defects detected in production = Defects detected in production/system size

Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of
acceptance defects found after delivery) *100

Effectiveness of testing to business = Loss due to problems / total resources processed by the system.

System complaints = Number of third party complaints / number of transactions processed

Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10

Source Code Analysis = Number of source code statements changed / total number of tests.

Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation

Test Execution Productivity = No of Test cycles executed / Actual Effort for testing

Software Metrics

Software metrics are an integral part of the state-ofthe-practice in software engineering. More and more customers are specifying software and/or quality metrics reporting as part of their contractual requirements. Industry standards like ISO 9000 and industry models like the Software Engineering Institute’s (SEI) Capability Maturity Model Integrated (CMMI®) include measurement. Companies are using metrics to better understand, track, control and predict software projects, processes and products.

The term software metrics means different things to different people. When we buy a book or pick up an article on software metrics, the topic can vary from project cost and effort prediction and modeling, to defect tracking and root cause analysis, to a specific test coverage metric, to computer performance modeling. These are all examples of metrics when the word is used as a noun. I prefer the activity based view taken by Goodman. He defines software metrics as, "The continuous application of measurement-based techniques to the software development process and its products to supply meaningful and timely management information, together with the use of those techniques to improve that process and its products." [Goodman-93] Figure 1, illustrates an expansion of this definition to include software-related services such as installation and responding to customer issues. Software metrics can provide the information needed by engineers for technical decisions as well as information required by management.

If a metric is to provide useful information, everyone involved in selecting, designing, lementing,
collecting, and utilizing it must understand its definition and purpose.

Function Point Analysis

Introduction To Function Point Analysis
Software systems, unless they are thoroughly understood, can be like an ice berg. They are becoming more and more difficult to understand. Improvement of coding tools allows software developers to produce large amounts of software to meet an ever expanding need from users. As systems grow a method to understand and communicate size needs to be used. Function Point Analysis is a structured technique of problem solving. It is a method to break systems into smaller components, so they can be better understood and analyzed.

Function points are a unit measure for software much like an hour is to measuring time, miles are to measuring distance or Celsius is to measuring temperature. Function Points are an ordinal measure much like other measures such as kilometers, Fahrenheit, hours, so on and so forth

Reboot! Rethinking and Restarting Software Development - Online book
Online self paced function point training.

Human beings solve problems by breaking them into smaller understandable pieces. Problems that may appear to be difficult are simple once they are broken into smaller parts -- dissected into classes. Classifying things, placing them in this or that category, is a familiar process. Everyone does it at one time or another -- shopkeepers when they take stock of what is on their shelves, librarians when they catalog books, secretaries when they file letters or documents. When objects to be classified are the contents of systems, a set of definitions and rules must be used to place these objects into the appropriate category, a scheme of classification. Function Point Analysis is a structured technique of classifying components of a system. It is a method to break systems into smaller components, so they can be better understood and analyzed. It provides a structured technique for problem solving.

In the world of Function Point Analysis, systems are divided into five large classes and general system characteristics. The first three classes or components are External Inputs, External Outputs and External Inquires each of these components transact against files therefore they are called transactions. The next two Internal Logical Files and External Interface Files are where data is stored that is combined to form logical information. The general system characteristics assess the general functionality of the system.

Brief History
Function Point Analysis was developed first by Allan J. Albrecht in the mid 1970s. It was an attempt to overcome difficulties associated with lines of code as a measure of software size, and to assist in developing a mechanism to predict effort associated with software development. The method was first published in 1979, then later in 1983 . In 1984 Albrecht refined the method and since 1986, when the International Function Point User Group (IFPUG) was set up, several versions of the Function Point Counting Practices Manual have been published by IFPUG. The current version of the IFPUG Manual is 4.1. A full function point training manual can be downloaded from this website.


Objectives of Function Point Analysis
Frequently the term end user or user is used without specifying what is meant. In this case, the user is a sophisticated user. Someone that would understand the system from a functional perspective --- more than likely someone that would provide requirements or does acceptance testing.

Since Function Points measures systems from a functional perspective they are independent of technology. Regardless of language, development method, or hardware platform used, the number of function points for a system will remain constant. The only variable is the amount of effort needed to deliver a given set of function points; therefore, Function Point Analysis can be used to determine whether a tool, an environment, a language is more productive compared with others within an organization or among organizations. This is a critical point and one of the greatest values of Function Point Analysis.

Function Point Analysis can provide a mechanism to track and monitor scope creep. Function Point Counts at the end of requirements, analysis, design, code, testing and implementation can be compared. The function point count at the end of requirements and/or designs can be compared to function points actually delivered. If the project has grown, there has been scope creep. The amount of growth is an indication of how well requirements were gathered by and/or communicated to the project team. If the amount of growth of projects declines over time it is a natural assumption that communication with the user has improved.

Characteristic of Quality Function Point Analysis
Function Point Analysis should be performed by trained and experienced personnel. If Function Point Analysis is conducted by untrained personnel, it is reasonable to assume the analysis will done incorrectly. The personnel counting function points should utilize the most current version of the Function Point Counting Practices Manual,

Current application documentation should be utilized to complete a function point count. For example, screen formats, report layouts, listing of interfaces with other systems and between systems, logical and/or preliminary physical data models will all assist in Function Points Analysis.

The task of counting function points should be included as part of the overall project plan. That is, counting function points should be scheduled and planned. The first function point count should be developed to provide sizing used for estimating.


The Five Major Components
Since it is common for computer systems to interact with other computer systems, a boundary must be drawn around each system to be measured prior to classifying components. This boundary must be drawn according to the user’s point of view. In short, the boundary indicates the border between the project or application being measured and the external applications or user domain. Once the border has been established, components can be classified, ranked and tallied.

External Inputs (EI) - is an elementary process in which data crosses the boundary from outside to inside. This data may come from a data input screen or another application. The data may be used to maintain one or more internal logical files. The data can be either control information or business information. If the data is control information it does not have to update an internal logical file. The graphic represents a simple EI that updates 2 ILF's (FTR's).

External Outputs (EO) - an elementary process in which derived data passes across the boundary from inside to outside. Additionally, an EO may update an ILF. The data creates reports or output files sent to other applications. These reports and files are created from one or more internal logical files and external interface file. The following graphic represents on EO with 2 FTR's there is derived information (green) that has been derived from the ILF's

External Inquiry (EQ) - an elementary process with both input and output components that result in data retrieval from one or more internal logical files and external interface files. The input process does not update any Internal Logical Files, and the output side does not contain derived data. The graphic below represents an EQ with two ILF's and no derived data.

Internal Logical Files (ILF’s) - a user identifiable group of logically related data that resides entirely within the applications boundary and is maintained through external inputs.

External Interface Files (EIF’s) - a user identifiable group of logically related data that is used for reference purposes only. The data resides entirely outside the application and is maintained by another application. The external interface file is an internal logical file for another application.

Summary of benefits of Function Point Analysis

Function Points can be used to size software applications accurately. Sizing is an important component in determining productivity (outputs/inputs).
They can be counted by different people, at different times, to obtain the same measure within a reasonable margin of error.
Function Points are easily understood by the non technical user. This helps communicate sizing information to a user or customer.
Function Points can be used to determine whether a tool, a language, an environment, is more productive when compared with others.