Automated Test Generation

Oracle-free Testing

Combinatorial methods make it possible to detect a significant number of faults without a conventional test oracle.  This seemingly impossible task is achieved using two layers of covering arrays with equivalence classes, as shown in this presentation. 

• Kuhn, D. R., Kacker, R. N., Lei, Y., & Torres-Jimenez, J. (2015, April). Equivalence Class Verification and Oracle-free Testing Using Two-layer Covering Arrays. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-4). IEEE.

Automated Test Generation

Automated test generation is of little value if it only generates data, without the expected results for each set of inputs.  The methods described below can generate both inputs and expected outputs. 

• Kuhn DR, D Yaga, Hu, V, Kacker RN, Lei Y. Pseudo-Exhaustive Testing of Rule Based Systems, 30th Intl Conference on Software Engineering and Knowledge Engineering, Redwood City, CA July 1-3, 2018.

• Kuhn DR, Hu V, Ferraiolo DF, Kacker RN, Lei Y. Pseudo-Exhaustive Testing of Attribute Based Access Control Rules. In2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) 2016 Apr 11 (pp. 51-58). IEEE.

• V.C. Hu, D.R. Kuhn, T. Xie and J. Hwang, Model Checking for Verification of Mandatory Access Control Models and Properties, International Journal of Software Engineering and Knowledge Engineering, vol. 21, no. 1, February 2011, pp. 103-127. Preprint

• D.R. Kuhn, R. Kacker and Y. Lei, Automated Combinatorial Test Methods: Beyond Pairwise Testing, CrossTalk (Hill AFB): the Journal of Defense Software Engineering, vol. 21, no. 6, June 2008, pp.22-26. Comment: A fairly comprehensive tutorial on combinatorial testing and automated test generation, with a worked example.
• D. R. Kuhn, V. Okun, Pseudo-exhaustive Testing for Software, 30th Annual IEEE/NASA Software Engineering Workshop (SEW-30), Columbia, Maryland, April 24-28, 2006, pp. 153-158.Abstract; DOI: 10.1109/SEW.2006.26; Preprint Comment: Proof-of-concept experiment on pseudo-exhaustive testing, integrating automated test generation with combinatorial testing.
• V. Okun and P. E. Black, Issues in Software Testing with Model Checkers (preprint), submitted to 2003 International Conference on Dependable Systems and Networks (DSN 2003), San Francisco, California, June 22-25, 2003. Comment: Explains effective use of model checking to generate complete test cases.
• V . Okun, Specification Mutation for Test Generation and Analysis, PhD Dissertation, University of Maryland Baltimore County, 2004, 77 pp. Comment: Both theoretical and experimental methods for selecting the most effective mutation operators for test generation.
• V . Okun, P.E. Black and Y. Yesha, Testing with Model Checkers: Insuring Fault Visibility, WSEAS Transactions on Systems, vol. 2, no. 1, January 2003, pp. 77-82. Comment: Describes specification-based mutation methods using a model checker to guarantee propagation of faults to visible outputs. Same paper was presented at 2002 WSEAS International Conference on System Science, Applied Mathematics & Computer Science, and Power Engineering Systems, Rio de Janeiro, Brazil, October 21-23, 2002.
• P .E. Black, V. Okun and Y. Yesha, Mutation Operators for Specifications, Fifteenth IEEE International Conference on Automated Software Engineering, Grenoble, France, September 11-15, 2000, pp. 81-88. Preprint Comment: Sets of mutation operators that yield good test coverage at reduced cost compared with other operators.