Combinatorial Testing

 

Overview:

Combinatorial methods have been demonstrated to significantly reduce the cost and increase the quality of software and system testing. As part of NIST's mission to advance the science of testing and measurement, we have conducted extensive analyses of software failure in real-world systems. Combinatorial testing takes advantage of the interaction rule, which is based on our analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. The project has developed advanced algorithms and tools for combinatorial testing, including ACTS, to generate tests, and CCM, to measure combinatorial aspects of test quality. Tools have been obtained by more than 1,000 organizations, and the accompanying combinatorial testing tutorial downloaded by more than 27,000.

Project Contact:

Mr. Rick Kuhn
(301) 975-3337
kuhn@nist.gov