Combinatorial Testing for Software: an Adaptation of Design of Experiments

Software has become increasingly ubiquitous in tools and methods used for science, engineering, medicine, commerce, and human interactions. Extensive testing is required to assure that software works correctly. Combinatorial testing is a versatile methodology which is useful in a broad range of situations to detect faults in software. It is based on the insight that while the behavior of a software system may be affected by a large number of factors, only a few factors are involved in a failure-inducing fault. We discuss the development of combinatorial testing for software as adaptation of design of experiment methods. Combinatorial testing began as pairwise testing in which first orthogonal arrays and then covering arrays were used to make sure that all pairs of the test settings were tested. Subsequent investigations of actual software failures showed that pairwise (2-way) testing may not always be sufficient and combinatorial t-way testing for t greater than 2 may be needed. Until recently efficient tools for generating test suites for combinatorial t-way testing were not widely available. Combinatorial testing has become practical because efficient and free downloadable tools with support of constraints have become available.