Computer Security Resource Center

Computer Security Resource Center

Computer Security
Resource Center

This is an archive
(replace .gov by .rip)

Automated Combinatorial Testing for Software

Formal Methods and Combinatorial Testing

The field of formal methods covers a broad range of mathematically-based techniques for specifying and verifying properties of software and systems.  Formal methods can be very effective for certain classes of problems, but they have gained a reputation for enormous expense.  One of the greatest opportunities for cost-effective use of these methods is the union of formal methods with testing. When a formal specification can be used in generating expected test results, the cost of developing the specification can be offset by a great reduction in the otherwise high cost of producing a test oracle (determining what result should be expected for each test).  Even when formal proofs of software properties are used, testing will be needed,  because assumptions made in the proof may not hold when the specification was mapped to implemented code, a few lines in a proof may correspond to a large volume of library code. 

We have developed to approaches to combining formal methods with combinatorial testing:

PEV - Pseudo-Exhaustive Verification tool, which uses formally specified rules to split a covering array of all t-way combinations of parameter values into positive and negative cases, allowing fully automated testing. That is, tests containing all possible t-way combinations of parameter values are associated with the expected result for every test.  The tool allows the user to enter rules as logic predicates, which are then converted to k-DNF form by the tool, and used to determine the expected result for every test.

Model checking of covering arrays - In this approach, a formal specification is defined using the temporal logic language CTL, which is then used with the NuSMV model checker to determine the expected result for every test in a covering array.  The result is fully automated testing, i.e., both test data and expected output for every test.  This method has been implemented in the Access Control Policy Testing tool (see Downloadable Tools).


PEV is an easy-to-use tool with a GUI interface and a command-line version for incorporation into a tool chain. The rule specification and test generation process are summarized below. 

PEV test process overview


Rules are entered as logic predicates.  Currently the tool supports boolean variables and relations between numeric variables.

PEV rule entry

Tests are displayed and can be exported as comma separated value (.csv) files that are readable by other tools or any spreadsheet application.

PEV output




Created May 24, 2016, Updated October 07, 2019