Downloadable Tools
We are developing tools to support combinatorial testing. No license is required and there are no restrictions on distribution or use. If you would like a copy, just send us an email request. Please include the name of your company or university - this information helps us with maintaining management support for the project. All software is provided free of charge and will remain free in the future. NIST is an agency of the US Government.
Software:
Tutorials and guides:
- Practical Combinatorial Testing - tutorial-style introduction to combinatorial methods for software testing (pdf)
- ACTS User Guide - how to use the ACTS test generation tool (pdf)
- Combinatorial Coverage Measurement - explains various coverage measurements and how to use the tool for computing these (pdf)
Advanced Combinatorial Testing System (ACTS) can compute tests for 2-way through 6-way interactions. An easy-to-use GUI is included, in addition to a command line version suitable for use in scripts or system calls from another tool. A comparison of ACTS with similar tools shows that ACTS produces smaller test sets with the same degree of coverage) and is faster than others. ACTS is Java-based, so it will run on any Java platform.
For a detailed comparison of ACTS with other combinatorial testing tools, see spreadsheet and graphs here.
To request a copy, send email to Rick Kuhn - kuhn@nist.gov.
This software is provided free of charge, is in the public domain, and will remain free in the future. NIST is an agency of the US Government. Please Note: Prior to February 2009, the name "FireEye" was used for the ACTS tool. Some papers and articles on the site refer to FireEye; this is the same tool now called ACTS.
User guide for combinatorial testing tool here.
Who uses our combinatorial testing tool? We have more than 2200 users as of August 2015, in IT, defense, finance, telecom, and many other industries. Here is a breakdown of our user base.
User Feedback
Users have been very positive, and are applying ACTS to a wide variety of software.
- "The tool is pretty easy to use, and has already reduced our planned number of tests by 20% or so.
- "At present I am working with a development team to enable automated testing of a complex rating algorithm under development within a property/casualty insurance system. The system component under test accepts several hundred independent input variables which are used to produce rating factors as output. I have had some opportunity to use [NIST-ACTS] as a means to calculate test coverage and identify missing test cases in our current population of test data. I'm pleased to report that the combinatorial coverage data has proven useful to us in identifying gaps. Eventually, I would like to use this data to extend our test coverage as well -- at coverage strength 3 we're producing a volume of test cases that should be manageable when paired with automated test data generation. I can also foresee other uses which would pair automated testing tools with [NIST-ACTS] output to improve our test coverage for other systems."
- "Our feedback on [NIST-ACTS] is extremely positive. The tool implement in a very efficient way the algorithm of N-Wise reduction we were looking at. In fact, we would be interested in including your technology in our tool [ ]."
- "[NIST-ACTS] is a good application. It was easy to figure out and use. ... I would use this application again."
- "I've fund it pretty intuitive and greatly helped to reduce the test size. I'll continue to use this tool at next opportunity."
- "Not only does [NIST-ACTS] handle phenomenal combinatorial complexity without breaking a sweat, it is also very easy and straightforward to use. The user interface is very well thought out, intuitive, and delightfully uncluttered.
Combinatorial coverage measurement: a tool to measure combinatorial coverage of an existing test suite. This is currently an alpha release, but will eventually be open source.
To request a copy, send email to Rick Kuhn - kuhn@nist.gov.
User guide for coverage measurement tool here.
Access Control Policy Test (ACPT) tool allows policy authors to conveniently specify access control models (such as RBAC and Multi-Level models) and rules as well as access control properties. From the specified models and rules, the ACPT tool automatically synthesizes deployable policies in XACML and generates combinatorial tests to verify security policy implementations. Complete test cases are generated, consisting of test inputs and expected output for each set of inputs. ACPT uses ACTS to provide 2-way to 4-way combinatorial testing of policies.
To request a copy, send email to Vincent Hu - vhu@nist.gov.
.NET Configuration test file generator produces a collection of .NET config files from ACTS output. This tool can be used to provide combinatorial coverage for systems that have a large number of configuration options.
To request a copy, send email to Rick Kuhn - kuhn@nist.gov.
Sequence covering array generator can produce arrays for more efficient testing of event-driven systems. See chapter 5 of the Practical Combinatorial Testing tutorial, here, for an explanation of sequence covering arrays.
To request a copy, send email to Rick Kuhn - kuhn@nist.gov.
Defining variables in system under test
Option 1: Constraints among variables can be added.
Option 2: Different interaction strengths can be used for sets of variables.
Finished covering array can be output as Excel spreadsheet, text or XML.