Covering arrays or orthogonal arrays?

A question that testers frequently have concerns the type of test array to be used. Orthogonal arrays (OA) are well known in statistics, and have been applied to software testing. Should testers use orthogonal arrays or covering arrays? What is the difference between the two?

The short answer to the first question is that in nearly all cases, covering arrays are preferable for software testing. This is a consequence of the difference in definitions. In an orthogonal array, each combination occurs only once, but in a covering array, each combination occurs at least once. So in a covering array, some combinations will be tested several times, which normally doesn’t matter for software testing.

Relaxing the constraint on number of occurrences has several positive effects. Often, an OA matching the required combinatorial test structure does not exist. For example suppose four test factors have two test settings each and one has three test settings; thus the combinatorial test structure is 2⁴×3¹. An OA of strength 2 matching the test structure 2⁴×3¹ does not exist (it is mathematically impossible). Also, OA based test suites may include invalid test cases that cannot be executed. For example, suppose in interoperability testing one factor is operating system and another factor is browser. Since the browser Internet Explorer (IE) does not run on the operating system Linux, the pair {Linux, IE} is invalid. Since a test suite based on an OA includes all pairs of combinations the same number of times, this pair will be included in some test cases and those test cases would be invalid. A consequence of omitting those test cases is that some valid pairs of test setting may not be tested. The limitations of OAs led to the use of covering arrays (CAs), which have several advantages: (1) CAs can be constructed for any combinatorial test structure of unequal numbers of test settings. (2) If for a combinatorial test structure an OA exists, then a CA of the same or fewer test cases can be obtained. (3) CAs can be constructed for any required strength (t-way) testing, while OAs are generally limited to strength 2 and 3. (4) While generating test suites based on CAs, invalid combinations can be excluded.