Automated Combinatorial Testing for Software (ACTS)
Combinatorial testing is a proven method for more effective software testing at lower cost. The key insight underlying combinatorial testing’s effectiveness resulted from a series of studies by NIST from 1999 to 2004. NIST research showed that most software bugs and failures are caused by one or two parameters, with progressively fewer by three or more. This finding, referred to as the interaction rule, has important implications for software testing because it means that testing parameter combinations can provide more efficient fault detection than conventional methods. New algorithms compressing combinations into a small number of tests have made this method practical for industrial use, making it possible to do better testing at lower cost.
Read more
Our focus is on empirical results and real-world testing.
Quick Start -
It's easy to learn the basics of this advanced method!
- 3-page summary
- Combinatorial and pairwise testing tutorial (81 pgs.)
- Briefing slides
(approx. 60 min).
FREELY AVAILABLE SOFTWARE: Software on this site is free of charge and will remain free in the future. It is public domain; no license is required and there are no restrictions on use. NIST is an agency of the United States Government.
NEW - Introducing
Combinatorial Testing in a Large Organization: Experience
Report,
(poster); full paper in IEEE Computer, April 2015.
J. Hagar. D.R. Kuhn, R.N. Kacker, T. Wissink.
An extensive evaluation of combinatorial testing by one of the world's
largest aerospace firms. Describes the
experience of Lockheed Martin applying combinatorial methods in eight
pilot projects, with roughly 20% cost savings and 20% to 50% better coverage. Extended
abstract. Presented at 3rd International Workshop on Combinatorial Testing,
Cleveland, Mar. 31, 2014.
NEW - Oracle-free Testing - software testing normally requires that for each test, there is an
expected output, known as a test oracle. Combinatorial methods
make it possible to detect a significant number of faults without a
conventional test oracle. This seemingly impossible task is
achieved using two layers of covering arrays with equivalence classes
derived from specifications. Source code is not required.
Essentially, we are able to take advantage of the information latent in
equivalence classes to do extensive consistency checking, revealing
errors.
- Presentation
on this method from the NSF Research Experience for Undergraduates
- Paper
- Intl. Workshop on Combinatorial Testing, 2015 - introduces this
method; with illustrative examples
- Paper
- Intl. Workshop on Combinatorial Testing, 2016 (submitted) - related
method applied to access control rules
Book - We have published the first textbook on combinatorial methods in software testing, Introduction to Combinatorial Testing,
Rick Kuhn, Raghu Kacker, Yu "Jeff" Lei, with chapter contributions from Renee Bryce, Eduardo Miranda, Sreedevi Sampath, and George Sherwood. (CRC Press, ISBN 1466552298,
June 2013; 319 pgs). The book provides software testers, developers, and students with a self-contained tutorial on how to use these methods for real-world software.
"I thoroughly recommend it to anyone involved in the practice of
software testing." - ACM Computing Reviews
Recent Talks (Kuhn):
- Duke University, Oct. 30, 2015
- Carnegie Mellon, Feb 17, 2015;
- Loyola, April 13, 2015;
- NSF Research Experience for Undergraduates, East Carolina, June 29, 2015;
- NASA IV&V Workshop, September 9-11, 2014;
- NITRD/National Science Foundation, April 24, 2014;
- IEEE Intl Conference on Software Testing Keynote,
April 2, Cleveland;
- American Software Testing Qualifications Board Keynote, March 26, 2014, San Francisco
Video of
2013 NASA IV&V conference briefing on application of
combinatorial coverage measurement. (Charley Price
and Rick Kuhn)
Tool releases, August 2015: Version
2.93 of ACTS testing tool and Version 0.8 of CCM
combinatorial coverage measurement tool now available
Combinatorial Coverage Measurement -
NIST IR 7878 (released Sept. 2012)
Tools provided freely on this site are used by hundreds of organizations
around the world for software and hardware testing of reliability, safety, and security. Our research collaborations include the U. of Texas Arlington,
US Air Force, Carnegie Mellon, U. of North Texas, Johns Hopkins
University Applied Physics Lab, U. of Maryland Baltimore County, Centro Nacional de Metrologia of Mexico, NASA, and the U. of Maryland University College. This article explains how the approach is related to statistical Design of Experiments.
Combinatorial testing poster
summarizes the methodology.
Our research program includes:
- advanced covering array algorithms;
- combinatorial coverage measurement;
- cybersecurity testing;
- fault localization;
- distribution of interaction faults;
- testing web apps;
- application to modeling and simulation.
Some of our accomplishments to date include:
- empirical finding that software failures triggered by interactions of few variables (1 to 6);
- IPOG covering array algorithm and its variants, more efficient than other known algorithms;
- demonstrating effectiveness of test prioritization;
- demonstrating improved efficiency for modeling and simulation;
- access control testing automation;
- measurement science and tools for combinatorial coverage.
Rick
Kuhn or Raghu
Kacker
kuhn@nist.gov, raghu.kacker@nist.gov
Back to Top
We have 2,300 users as of June 2015, in nearly all major industries. Here is a breakdown of our user base.
1) Combinatorial testing tool, for generating test suites. Advanced
Combinatorial Testing System (ACTS) can compute tests for 2-way
through 6-way interactions. An easy-to-use GUI is included. A comparison
of ACTS with similar tools shows that ACTS produces
smaller test sets (with the same degree of coverage) and is faster than
others. ACTS was developed by NIST and the University of Texas at
Arlington. To request a copy, send email to Rick Kuhn. Please include your first
and last name, and company or university name (this helps us with
management support for the project!) ACTS won the 2009 Excellence in
Technology Transfer Award from the Federal Laboratory Consortium
Mid-Atlantic Region.
2) Combinatorial coverage measurement tool, for evaluating quality
of test suites. Useful for gaining the advantages of combinatorial
testing without disrupting existing test practice. The CCM
measurement tool can analyze existing tests for 2-way
through 6-way interactions they already have. An easy-to-use GUI is
included. CCM was developed by NIST and the Centro
Nacional de Metrologia of Mexico. To request a copy, send email to Rick Kuhn. Please include your first
and last name, and company or university name (this helps us with
management support for the project!)
3) Testing tutorial, Practical Combinatorial Testing,
NIST SP 800-142
(81 pages). This publication provides a self-contained tutorial
on using combinatorial testing for real-world software. It introduces
the key concepts and methods, explains use of software tools for
generating combinatorial tests, and discusses advanced topics. The
material is accessible to an undergraduate student of computer science
or engineering, and includes an extensive set of references to
papers that provide more depth on each topic. Oct. 2010. Public domain,
distribution unlimited; 81 pages.
4) Security policy testing tool. The Access Control
Policy Test (ACPT) tool allows policy authors to conveniently
specify access control models (such as RBAC and Multi-Level models) and
rules as well as access control properties. From the specified models
and rules, the ACPT tool automatically synthesizes deployable policies
in XACML and
generates combinatorial tests to verify security policy
implementations. Complete test cases are generated, consisting of test
inputs
and expected output for each set of inputs. ACPT uses ACTS to provide
2-way to 4-way combinatorial testing of policies. To request a copy,
send email to Vincent Hu
5) Web app testing tool. CPUT (Combinatorial-based
Prioritization of User-session-based Testsuites) applies combinatorial
methods to testing web applications. Test prioritization is used to
make web app testing much more manageable. The tool allows testers to
easily collect, prioritize, and reduce user-session-based test cases.
CPUT provides (1) guidance to users on how to configure their web
server to log important usage information, (2) automated parsing of web
logs into XML formatted test cases that can be used by test replay
tools, (3) automated prioritization of test cases by length-based and
combinatorial-based criteria, and (4) automated reduction of testcases
by combinatorial coverage.
Back to Top
Past Events
- C. Price, R. Kuhn, R. Forquer, A. Lagoy, R. Kacker, Evaluating the t-way Combinatorial Technique for Determining the Thoroughness of a Test Suite, NASA IV&V Workshop, 2013.
- 2nd International Workshop on Combinatorial Testing association with
5th IEEE International Conference on Software Testing, Verification and Validation (ICST 2013, March 18-22) Luxembourg.
- 1st
Asian Workshop on Combinatorial Testing (PPT in zip file)
- 1st
International Workshop on Combinatorial Testing In association with
5th IEEE International Conference on Software Testing, Verification and
Validation April 17-21, 2012, Montreal, Canada - The workshop was a
great success, with 10 presented, lots of lively discussion.
Presentations here.
- NIST/Carnegie Mellon University workshop, June 7, 2011, on
software verification was a great success, with more than 150
participants locally and via webcast.
- A survey
by Gary Gack found that nearly everyone who tried combinatorial testing
has found it useful!
- Combinatorial
coverage measurement was presented to NASA developers and
researchers at the NASA
IV&V Workshop , September 11-13, 2012.
- We gave talks at East Carolina University and Carnegie Mellon
University earlier this year.
- ACTS team members presented three papers at the First Intl
Workshop on Combinatorial Testing, April, 2012.
- 1,000th organization acquired ACTS tool in March 2012.
- We did a half-day tutorial on combinatorial methods at ICSM 2011
and lecture at Verify/ATI in September.
- Rick Kuhn and Raghu Kacker are presenting a half-day tutorial on
combinatorial methods at the IEEE International Conference on Software
Maintenance, ICSM 2011, September 26 in Williamsburg, Virginia.
- Rick Kuhn and Sreedevi Sampath are speaking at the Verify/ATI
conference on September 27, in Arlington, Virginia.
- Raghu Kacker presented seminars on combinatorial testing and ACTS
at Fraunhofer
Institute for Open Communication Systems (FOKUS) and Technical University
of
Berlin June 27 and 28, 2011.
- Advanced
test methods seminar in Pittsburgh at Carnegie Mellon University with
Prof. Eduardo Miranda in June, drew over 150 attendees in person and
via internet.
- ACTS profiled in IEEE Computer, February 2011, News
Briefs.
- Seminars and talks: NASA IV&V, August 31; Fraunhofer
Center - UMD, August 15; NASA IV&V, July 14; Carnegie Mellon Univ.,
June 7, Johns Hopkins University Applied Physics Lab, April 27;
Institute for Defense Analyses, April 6; Army Test and Evaluation
Command, March 30; Institute for Defense Analyses February
25;Carnegie-Mellon University February 9.
- Raghu Kacker gave invited talks at Indian Institute of
Technology, Madras, and Infosys Corp., January 2011.
- November issue of Logigear
magazine features an interview
(Rick Kuhn) and an article on combinatorial testing.
- ACTS
featured in SIGNAL, the flagship publication of the Armed
Forces Communications and Electronics Association.
- Rick Kuhn speaking on combinatorial testing at Dept
of Energy modeling and simulation workshop December 1, 2010.
- New
Help on Testing for Common Cause of Software Bugs, Government
Computer News, November 1, 2010. ACTS also featured in The
Engineer, Information
Week and TMC.net.
- ACTS project featured in October issue of Logigear Magazine.
- NIST initiated a Cooperative Research and Development Agreement
with Lockheed Martin Corp. in August 2010.
- 500th organization acquired ACTS tool in November 2010
- Rick Kuhn, Raghu Kacker, and Sreedevi Sampath conducted a
workshop on combinatorial testing for the US Army Test and
Evaluation Command in May.
- 300th organization acquired ACTS tool in February 2010
- Rick Kuhn gave an invited talk on combinatorial testing at Carnegie
Mellon University in January.
- ACTS received 2009 Excellence in Technology Transfer Award
from the Federal
Laboratory
Consortium Mid-Atlantic Region.
- Our new introductory article on
combinatorial testing was featured as one of the Highlights for the
August 2009 issue of IEEE Computer, the flagship publication of the
IEEE Computer Society.
Back to Top
Seminars and Talks
Quick introductions to Combinatorial Testing
- Practical
Applications of Combinatorial Testing, East Carolina
University, March 22,
2012.
- Combinatorial
Testing and Design of Experiments, TU Berlin, June 28, 2011.
- Combinatorial
Testing, Institute for Defense Analyses, April 6, 2011.
(approx.
2 hours)
- Combinatorial
Testing Seminar, US Army Test & Evaluation Command,
Aberdeen Proving Ground, May 17, 2010. (approx. 3 hours).
- Combinatorial
Testing, Carnegie-Mellon University Jan 26, 2010. (approx. 60
min.)
- Combinatorial
Testing Tutorial, National Defense Industrial
Association, Reston, VA, September 16-17, 2009. (approx. 30 min.)
Note: As US Government work, the work of Kuhn, Kacker, and
Black is not subject to US copyright. You may use any or all of
these slides freely (but we always appreciate it if you cite NIST!)
Conference Presentations
- Introducing
Combinatorial Testing in Large Organizations, ASTQB, March 2014
- Evaluating
the t-way Technique for Determining the Thoroughness of a Test Suite,
NASA IV&V Workshop, September 2013.
- Combinatorial coverage measurement NASA IV&V Workshop
September 11-13, 2012.
- Combinatorial
Methods for Event Sequence Testing, D.R. Kuhn, J. Higdon, J.
Lawrence, R.N. Kacker and Y. Lei, FIrst Intl Workshop on Combinatorial
Testing, April, 2012.
- Combinatorial
Testing of ACTS: A Case Study, M. Nouroz Borazjany, L. Yu, Y.
Lei, R.N. Kacker and D.R. Kuhn
- Isolating
Failure-Inducing Combinations in Combinatorial Testing using Test
Augmentation and ClassificationK. Shakya, T. Xie, N. Li, Y.
Lei, R.N. Kacker, D.R. Kuhn
- Rick Kuhn, Combinatorial
Methods for Discrete Event Simulation of a Grid Computer Network,
ModSim World 2009, October 14-17 2009, Virginia Beach, Virginia.
- Rick Kuhn, Raghu Kacker, Combinatorial
Testing Tutorial, Military Test and Evaluation, McLean, VA,
June 24-25, 2009. (approx. 45 min.)
- D.R. Kuhn, R. Kacker, Automated
Combinatorial Testing, VERIFY 2007, Crystal City, VA, October
29-30, 2007.
- P. Black, My
Life with Bugs, or Why I Believe in Combinatorial Testing,
American Society for Quality, Gaithersburg, MD October 30, 2007.
- Y. Lei, IPOG
- A General Strategy for t-Way Software Testing, IEEE
Engineering of Computer Based Systems conference, 2007.- describes
generalized IPO algorithm for constructing t-way covering arrays.
- D.R. Kuhn, V. Okun, Automated Combinatorial Testing for Software,
ISSRE 06, Raleigh, November 6, 2006.
- D. R. Kuhn, V. Okun, Pseudo-exhaustive
Testing For Software, 30th NASA/IEEE
Software Engineering Workshop, April 25-27, 2006 - proof-of-concept
experiment on pseudo-exhaustive testing.
- Y. Lei, In
Parameter Order: A Test Generation Strategy for Pairwise Testing,
6/21/05 - introduces IPO and outlines extension to pairwise test.
Performance comparison with other tools here.
- Presentation
Back to Top
Tutorial
- Combinatoral
and pairwise testing tutorial Practical Combinatorial
Testing, NIST SP 800-142, (October 2010) - a concise, easy to
read tutorial, for individual study or classroom use. Public domain,
distribution unlimited; 81 pages.
Papers
- Kuhn, D.R., Hu, V., Ferraiolo, D., Kacker, R., Lei, Y., Pseudo-exhaustive Testing of Attribute Based Access Control Rules, (Aug. 2015) Preprint, to be submitted Intl Workshop on Combinatorial Testing.
- Hagar, J. D., Wissink, T. L., Kuhn, D. R., & Kacker, R. N. (2015). Introducing Combinatorial Testing in a
Large Organization. IEEE Computer, (4), 64-72.
- DR Kuhn, R Bryce, F Duan, L Sh. Ghandehari, Yu Lei, RN Kacker, Combinatorial Testing, Theory and Practice, Advances in Computers, vol. 99 (to appear, 2015).
- Kuhn, D. R., Kacker, R. N., Lei, Y., & Torres-Jimenez, J. (2015, April). Equivalence class verification and oracle-free testing using two-layer covering arrays. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-4). IEEE.
- Duan, F., Lei, Y., Yu, L., Kacker, R. N., & Kuhn, D. R. (2015, April). Improving IPOG's vertical growth based on a graph coloring scheme. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-8). IEEE.
- Yu, L., Duan, F., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2015, April). Constraint handling in combinatorial test generation using forbidden tuples. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-9). IEEE.
- Ghandehari, L. S., Chandrasekaran, J., Lei, Y., Kacker, R., & Kuhn, D. R. (2015, April). BEN: A combinatorial testing-based fault localization tool. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-4). IEEE.
- D.R. Kuhn, R.N. Kacker and Y. Lei, Combinatorial Coverage as an Aspect of Test Quality, Mar/Apr 2015 CrossTalk: Journal of Defense Software Engineering. Article
(pdf)
Preprint
- D.R. Kuhn, R.N. Kacker and Y. Lei, Measuring and Specifying
Combinatorial Coverage of Test Input Configurations, submitted to Innovations
in Systems and Software Engineering: a NASA journal, [2015].
Preprint
- L. S. Ghandehari, J. Czerwonka, Y. Lei, S. Shafiee, R. Kacker and
R. Kuhn, An Empirical Comparison of Combinatorial and Random Testing,
Third International Workshop on Combinatorial Testing (IWCT
2014), in Proceedings of the Seventh IEEE International Conference on
Software, Testing, Verification and Validation (ICST 2014),
Cleveland, Ohio, March 31-April 4, 2014, pp. 68-77.
Abstract; DOI:
10.1109/ICSTW.2014.8; Preprint
- J. Hagar, R. Kuhn, R. Kacker, and T. Wissink, Introducing
Combinatorial Testing in a Large Organization: Pilot Project Experience
Report [poster], Third International Workshop on Combinatorial
Testing (IWCT 2014), in Proceedings of the Seventh IEEE International
Conference on Software, Testing, Verification and Validation (ICST 2014),
Cleveland, Ohio, March 31-April 4, 2014, p. 153.
Abstract; DOI:
10.1109/ICSTW.2014.70; Preprint
- R. Kuhn, R Kacker and Y. Lei, Estimating Fault Detection
Effectiveness [poster], Third International Workshop on
Combinatorial Testing (IWCT 2014), in Proceedings of the Seventh IEEE
International Conference on Software, Testing, Verification and
Validation (ICST 2014), Cleveland, Ohio, March 31-April 4, 2014, p.
154.
Abstract; DOI:
10.1109/ICSTW.2014.69; Preprint
- J. Torres-Jimenez, I. Izquierdo-Marquez, Survey of Covering
Arrays, 15th International Symposium on Symbolic and Numeric
Algorithms for Scientific Computing (SYNASC 2013), Timisoara,
Romania, September 23-26, 2013, pp. 20-27.
DOI: 10.1109/SYNASC.2013.10
- R.N. Kacker, D.R. Kuhn, Y. Lei, and J.F. Lawrence, Combinatorial
Testing for Software: an Adaptation of Design of Experiments, Measurement,
vol. 46, no. 9, November 2013, pp. 3745-3752.
DOI: 10.1016/j.measurement.2013.02.021
- X. Niu, C. Nie, Y. Lei, A.T.S. Chan, Identifying
Failure-Inducing Combinations Using Tuple Relationships, 2nd
International Workshop on Combinatorial Testing (IWCT 2013), in
Proceedings of the Sixth IEEE International Conference on Software,
Testing, Verification and Validation (ICST 2013), Luxembourg, March
18-22, 2013, pp. 271-280.
DOI: 10.1109/ICSTW.2013.38; Preprint
- D.R. Kuhn, I. Dominguez Mendoza, R.N. Kacker and Y. Lei. Combinatorial
Coverage Measurement Concepts and Applications, 2nd
International Workshop on Combinatorial Testing (IWCT 2013), in
Proceedings of the Sixth IEEE International Conference on Software,
Testing, Verification and Validation (ICST 2013), Luxembourg, March
18-22, 2013, pp. 352-361.
Abstract; DOI:
10.1109/ICSTW.2013.77; Preprint
- L.S.G. Ghandehari, M.N. Borazjany, Y. Lei, R.N. Kacker and D.R.
Kuhn, Applying Combinatorial Testing to the Siemens Suite, 2nd
International Workshop on Combinatorial Testing (IWCT 2013), in
Proceedings of the Sixth IEEE International Conference on Software,
Testing, Verification and Validation (ICST 2013), Luxembourg, March
18-22, 2013, pp. 362-371.
Abstract; DOI:
10.1109/ICSTW.2013.47; Preprint
- M.N. Borazjany, L.S.G. Ghandehari, Y. Lei, R.N. Kacker and D.R.
Kuhn, An Input Space Modeling Methodology for Combinatorial Testing,
2nd International Workshop on Combinatorial Testing (IWCT 2013),
in Proceedings of the Sixth IEEE International Conference on Software,
Testing, Verification and Validation (ICST 2013), Luxembourg, March
18-22, 2013, pp. 372-381.
Abstract; DOI:
10.1109/ICSTW.2013.48; Preprint
- D.R. Kuhn and R.N. Kacker, Measuring Combinatorial Coverage
of System State-space for IV&V [extended abstract], NASA
IV&V Workshop, Morgantown, West Virginia, September 11-13,
2012.
Abstract; Presentation
- S. Manchester, R. Bryce, S. Sampath, N. Samant, D. R. Kuhn and R.
Kacker, Applying Higher Strength Combinatorial Criteria to Test
Prioritization: a Case Study, to appear in: Journal of
Combinatorial Mathematics and Combinatorial Computing.
Preprint
- D.R. Kuhn, R.N. Kacker and Y. Lei. Combinatorial Testing,
Encyclopedia of Software Engineering, edited by Phillip A.
Laplante. Taylor & Francis, 2012.
Abstract; DOI: 10.1081/E-ESE-120048153; Preprint
- D.R. Kuhn, J.M. Higdon, J.F. Lawrence, R.N. Kacker and Y. Lei, Efficient
Methods for Interoperability Testing Using Event Sequences, CrossTalk
(Hill AFB): the Journal of Defense Software Engineering, vol. 25,
no. 4, July/August 2012, pp. 15-18.
Abstract; Article
- C. Montanez-Rivera, D.R. Kuhn, M. Brady, R.M. Rivello, J. Reyes
and M.K. Powers, Evaluation of Fault Detection Effectiveness for
Combinatorial and Exhaustive Selection of Discretized Test Inputs, Software
Quality Professional, vol. 14, no. 3, June 2012.
Abstract; Article; Preprint
- M.N. Borazjany, L. Yu, Y. Lei, R. Kacker and R. Kuhn, Combinatorial
Testing of ACTS: A Case Study, First International Workshop on
Combinatorial Testing, in Proceedings of the IEEE Fifth International
Conference on Software, Testing, Verification and Validation (ICST 2012),
Montreal, Quebec, Canada, April 17-21, 2012, pp. 591-600.
Abstract; DOI:
10.1109/ICST.2012.146
- K. Shakya, T. Xie, N. Li, Y. Lei, R. Kacker and R. Kuhn, Isolating
Failure-Inducing Combinations in Combinatorial Testing using Test
Augmentation and Classification, First International Workshop
on Combinatorial Testing, in Proceedings of the IEEE Fifth
International Conference on Software, Testing, Verification and
Validation (ICST 2012), Montreal, Quebec, Canada, April 17-21,
2012, pp. 620-623.
Abstract; DOI:
10.1109/ICST.2012.149; Preprint
- D.R. Kuhn, J.M. Higdon, J.F. Lawrence, R.N. Kacker and Y. Lei, Combinatorial
Methods for Event Sequence Testing, First International
Workshop on Combinatorial Testing, in Proceedings of the IEEE Fifth
International Conference on Software, Testing, Verification and
Validation (ICST 2012), Montreal, Quebec, Canada, April 17-21,
2012, pp. 601-609.
Abstract; DOI:
10.1109/ICST.2012.147; Preprint
- V.C. Hu, D.R. Kuhn, T. Xie and J. Hwang, Model Checking for
Verification of Mandatory Access Control Models and Properties, International
Journal of Software Engineering and Knowledge Engineering, vol. 21,
no. 1, February 2011, pp. 103-127.
Abstract; DOI:
10.1142/S021819401100513X; Preprint
- J. Lawrence, R.N. Kacker, Y. Lei, D.R. Kuhn, and M. Forbes, A
Survey of Binary Covering Arrays, Electronic Journal of
Combinatorics, vol. 18, no. 1, art. P84, April 7, 2011.
Abstract; Article
- D.R. Kuhn, R.N. Kacker and Y. Lei, Advanced Combinatorial
Test Methods for System Reliability, 2010 Annual Technical
Report of the IEEE Reliability Society, January 2011.
Article
- C. Montanez, D.R. Kuhn, M. Brady, R. Rivello, J. Reyes, M.K.
Powers, An Application of Combinatorial Methods to Conformance
Testing for Document Object Model Events, NIST Internal Report
(NISTIR) 7773, November 2010, 9 pp.
Abstract; NISTIR 7773
- R.C. Bryce, C.J. Colbourn and D.R. Kuhn, Finding Interaction
Faults Adaptively using Distance-Based Strategies, 18th IEEE
International Conference and Workshops on Engineering of Computer Based
Systems (ECBS), Las Vegas, Nevada, April 27-29, 2011, pp.4-13.
DOI:
10.1109/ECBS.2011.9
- J.R. Maximoff, M.D. Trela, D.R. Kuhn and R. Kacker, A Method
for Analyzing System State-space Coverage within a t-Wise Testing
Framework, 4th Annual IEEE International Systems Conference,
April 5-8, 2010, San Diego, California, pp. 598-603.
DOI: 10.1109/SYSTEMS.2010.5482481
- D.R. Kuhn, R. Kacker and Y.Lei, Random vs. Combinatorial
Methods for Discrete Event Simulation of a Grid Computer Network, MODSIM
World 2009, Virginia Beach, Virginia, October 14-16, 2009. In Selected
Papers Presented at MODSIM World 2009 Conference and Expo, edited
by T.E. Pinelli, NASA/CP-2010-216205, National Aeronautics and Space
Administration, pp. 83-88.
Abstract; Paper
- D.R. Kuhn,R. Kacker, Y. Lei, Combinatorial and Random Testing Effectiveness for a Grid
Computer Simulator
- R. Kuhn, R. Kacker, Y. Lei and J. Hunter, Combinatorial
Software Testing, IEEE Computer, vol. 42, no. 8, August
2009, pp. 94-96.
Abstract; DOI:
10.1109/MC.2009.253; Preprint
- R.C. Bryce, Y. Lei, D.R. Kuhn and R. Kacker, Combinatorial
Testing, Ch. 14 in Handbook of Research on Software Engineering
and Productivity Technologies: Implications of Globalization,
edited by M. Ramachandran and R. Atem de Carvalho, IGI Global, 2010,
pp.196-208.
DOI: 10.4018/978-1-60566-731-7.ch014
- V.C. Hu, D.R. Kuhn and T. Xie, Property Verification for
Generic Access Control Models, 2008 IEEE/IFIP International
Symposium on Trust, Security, and Privacy for Pervasive Applications
(TSP-08) in Volume 2 of Proceedings of the 2008 IEEE/IFIP International
Conference on Embedded and Ubiquitous Computing (EUC-08), Shanghai,
China, December 17-20, 2008, pp. 234-250.
Abstract; DOI:
10.1109/EUC.2008.22; Preprint
- M. Ellims, D. Ince and M. Petre, The Effectiveness of T-Way
Test Data Generation, 27th International Conference on Computer
Safety, Reliability, and Security (SAFECOMP 2008), Newcastle upon
Tyne, U.K., September 22-25, 2008. In Computer Safety, Reliability, and
Security, Lecture Notes in Computer Science 5219, pp.16–29.
DOI: 10.1007/978-3-540-87698-4_5
- M. Forbes, J. Lawrence, Y. Lei, R.N. Kacker, and D.R. Kuhn, Refining
the In-Parameter-Order Strategy for Constructing Covering Arrays, Journal
of Research of the National Institute of Standards and Technology, vol.
113, no. 5, September-October 2008, pp. 287-297.
Abstract; DOI:
10.6028/jres.113.022
- D.R. Kuhn, R. Kacker and Y. Lei, Automated Combinatorial Test
Methods: Beyond Pairwise Testing, CrossTalk (Hill AFB): the
Journal of Defense Software Engineering, vol. 21, no. 6, June 2008,
pp.22-26.
Abstract; Article
Comment: A fairly comprehensive tutorial on combinatorial testing and
automated test generation, with a worked example.
- R. Kessel and R. Kacker, A Test of Linearity Using Covering
Arrays for Evaluating Uncertainty in Measurement, Advanced
Mathematical and Computational Tools in Metrology and Testing (AMCTM
VIII), Paris, France, June 23-25, 2008, Series on Advances in
Mathematics for Applied Sciences vol. 78, pp.195-203.
Abstract; DOI: 10.1142/7212; Preprint
- R. Kuhn, Y. Lei and R. Kacker, Practical Combinatorial
Testing: Beyond Pairwise Testing, IT Professional, vol. 10,
no. 3, May-June 2008, pp.19-23.
Abstract; DOI:
10.1109/MITP.2008.54
Comment: Quick introduction to combinatorial testing.
- Y. Lei, R. Kacker, D. Kuhn, V. Okun and J. Lawrence, IPOG/IPOD:
Efficient Test Generation for Multi-Way Combinatorial Testing, Software
Testing, Verification, and Reliability, vol. 18, no. 3, September
2008, pp. 125-148.
Abstract; DOI: 10.1002/stvr.381
- Y. Lei, R.H. Carver, R. Kacker and D. Kung, A Combinatorial
Testing Strategy for Concurrent Programs, Software Testing,
Verification, and Reliability, vol. 17, no. 4, December 2007, pp.
207-225.
Abstract; DOI: 10.1002/stvr.369
- Y.Lei, R. Kacker, D.R. Kuhn, V. Okun and J. Lawrence, IPOG: a
General Strategy for T-way Software Testing, 14th Annual IEEE
International Conference and Workshops on the Engineering of
Computer-Based Systems (ECBS’07), Tucson, Arizona, March
26-29 2007, pp. 549-556.
Abstract; DOI:
10.1109/ECBS.2007.47; Preprint
- R. Bryce and C.J. Colbourn, Prioritized Interaction Testing
for Pair-wise Coverage with Seeding and Constraints, Information
and Software Technology, vol. 48, no. 10, October 2006, pp. 960-970.
DOI: 10.1016/j.infsof.2006.03.004
- R. Bryce, A. Rajan and M.P.E. Heimdahl, Interaction Testing in
Model-Based Development: Effect on Model-Coverage, 13th
Asia-Pacific Software Engineering Conference (APSEC 2006), Kanpur,
India, December 6-8, 2006, pp. 259-268.
DOI:
10.1109/APSEC.2006.42
- D. R. Kuhn, V. Okun, Pseudo-exhaustive
Testing for Software, 30th Annual IEEE/NASA Software
Engineering Workshop (SEW-30),
Columbia, Maryland, April 24-28, 2006, pp. 153-158.
Abstract; DOI:
10.1109/SEW.2006.26; Preprint
Comment: Proof-of-concept experiment on pseudo-exhaustive testing,
integrating automated test generation with combinatorial testing.
- R.C. Bryce, C.J. Colbourn and M.B. Cohen, A Framework of
Greedy Methods for Constructing Interaction Test Suites, 27th
International Conference on Software Engineering (ICSE 2005), St.
Louis, Missouri, May 15-21, 2005, pp. 146-155.
DOI: 10.1109/ICSE.2005.1553557
- M. Grindal, J. Offutt and S.F. Andler, Combination Testing
Strategies: a Survey, Software Testing, Verification and
Reliability, vol. 15, no. 3, March 2005, pp. 167-199.
DOI:
10.1002/stvr.319; Preprint
Comment: A survey of combinatorial testing methods and results.
- D.R. Kuhn, D.R. Wallace, A.M. Gallo, Jr., Software Fault
Interactions and Implications for Software Testing, IEEE
Transactions on Software Engineering, vol. 30, no. 6, June 2004,
pp. 418-421.
Abstract; DOI:
10.1109/TSE.2004.24
Comment: Investigates interaction level required to trigger faults in a
large distributed database system.
- D.R. Kuhn and M.J. Reilly, An Investigation of the
Applicability of Design of Experiments to Software Testing, 27th
Annual NASA Goddard/IEEE Software Engineering Workshop (SEW ’02),
Greenbelt, Maryland, December 5-6, 2002, pp. 91-95.
Abstract; DOI:
10.1109/SEW.2002.1199454; Preprint
Comment: Investigates interaction level required to trigger faults in
open source browser and server.
- D.R. Wallace and D.R. Kuhn, Failure Modes in Medical Device
Software: an Analysis of 15 Years of Recall Data, International
Journal of Reliability, Quality and Safety Engineering, vol. 8, no.
4, December 2001, pp.351-371.
Abstract; DOI:
10.1142/S021853930100058X; Preprint
Comment: Categorizes failures by their symptoms and faults, including
interaction level required to trigger faults in medical device software.
Back to Top
Combinatorial Methods for Modeling and Simulation
- D.R. Kuhn, R. Kacker and Y.Lei, Random vs.
Combinatorial Methods for Discrete Event Simulation of a Grid Computer Network, MODSIM
World 2009, Virginia Beach, Virginia, October 14-16, 2009. In
Selected Papers Presented at MODSIM World 2009 Conference and Expo,
edited by T.E. Pinelli, NASA/CP-2010-216205, National Aeronautics and
Space Administration, pp. 83-88.
Abstract; Paper
- R. Kessel and R. Kacker, A Test of
Linearity Using Covering
Arrays for Evaluating Uncertainty in Measurement, Advanced
Mathematical and Computational Tools in Metrology and Testing (AMCTM
VIII), Paris, France, June 23-25, 2008, Series on Advances in
Mathematics for Applied Sciences vol. 78, pp.195-203.
Abstract; DOI: 10.1142/7212; Preprint
Automated Test
Generation Using Model Checking
- V.C. Hu, D.R. Kuhn, T. Xie and J. Hwang, Model Checking for Verification of Mandatory Access Control Models and Properties, International Journal of Software Engineering and Knowledge Engineering, vol. 21, no. 1, February 2011, pp. 103-127.
Abstract; DOI: 10.1142/S021819401100513X; Preprint
- D.R. Kuhn, R. Kacker and Y. Lei, Automated Combinatorial Test Methods: Beyond Pairwise Testing, CrossTalk (Hill AFB): the Journal of Defense Software Engineering, vol. 21, no. 6, June 2008, pp.22-26.
Abstract; Article
Comment: A fairly comprehensive tutorial on combinatorial testing and automated test generation, with a worked example.
- V. Okun and P. E. Black, Issues in Software Testing with Model Checkers, submitted to 2003 International Conference on Dependable Systems and Networks (DSN 2003), San Francisco, California, June 22-25, 2003.
Preprint
Comment: Explains effective use of model checking to generate complete test cases.
- V . Okun, Specification Mutation for Test Generation and Analysis, PhD Dissertation, University of Maryland Baltimore County, 2004, 77 pp.
Dissertation
Comment: Both theoretical and experimental methods for selecting the most effective mutation operators for test generation.
- V . Okun, P.E. Black and Y. Yesha, Testing with Model Checkers: Insuring Fault Visibility, WSEAS Transactions on Systems, vol. 2, no. 1, January 2003, pp. 77-82.
Abstract; Paper
Comment: Describes specification-based mutation methods using a model checker to guarantee propagation of faults to visible outputs. Same paper was presented at 2002 WSEAS International Conference on System Science, Applied Mathematics & Computer Science, and Power Engineering Systems, Rio de Janeiro, Brazil, October 21-23, 2002.
- P .E. Black, V. Okun and Y. Yesha, Mutation Operators for Specifications, Fifteenth IEEE International Conference on Automated Software Engineering, Grenoble, France, September 11-15, 2000, pp. 81-88.
Abstract; DOI: 10.1109/ASE.2000.873653; Preprint
Comment: Sets of mutation operators that yield good test coverage at reduced cost compared with other operators.
Back to Top
Disclaimer: Certain software products are identified in this document.
Such identification does not imply recommendation by NIST, nor does it
imply that the products identified are necessarily the best available
for the purpose.Belorussian translation of our
project summary
here.