Good test designs often require testing many different sets of valid and invalid input parameters, hardware/software environments, and system conditions. This results in a combinatorial explosion of test cases. For example, testing different combinations of possible hardware and software components on a typical PC could involve hundreds or even thousands of possible tests. The classic question for effective testing is always, “Given limited time and resources, which of the combinations should be tested?”

This tutorial describes the underlying problems and challenges in test case design for combinatorial testing and explains solutions using different testing techniques. An overview about supporting tools (free as well as commercial) including their features, characteristics and usage scenarios is given. Furthermore, successful experiences gained from real-world projects using the right tools are presented which clearly show the necessity and benefits of these testing techniques.

Outline:

Contents
 

1. Introduction – Examples and problem statement

• Example developer unit testing

• Example system black-box testing

• Example configuration testing

• Problem statement: Combinatorial testing

2. Method

• Approach

• Example

• Benefits

• Usage scenarios

• Approaches for automated calculation

• Orthogonal arrays and covering arrays

• Heuristic algorithms

• Influences and dependencies

• Points to watch and limitations

• Additional requirements

3. Real-world example installation testing

4. Tools

• AETG Web Service (Telcordia)

• AllPairs (James Bach)

• CTE XL (DaimlerChrysler)

• Combinatorial Test Services CTS (IBM alphaWorks)

• IBM Intelligent Test Case Handler (IBM alphaWorks)

• Jenny (Bob Jenkins)

• Pairwise Independent Combinatorial Testing PICT (Microsoft)

• Pro-Test (SigmaZone)

• TConfig (Alan Williams)

• Tool characteristics, calculation results and experiences

• Demo

5. References

6. Summary