Combinatorial Testing

Assured Autonomy and Explainable AI Papers

Our conference and journal papers on assured autonomy and explainable AI.  We try to include links to the full papers, but for those not yet linked, please contact us for a copy:  kuhn@nist.gov. 

Papers

2022

Freeman, L. Batarseh, F., Kuhn, D. R., Raunak, M. S., & Kacker, R. N.,  The Path to Consensus on Artificial Intelligence Assurance, IEEE Computer (to appear, 2022)

Kuhn, D. R., Raunak, M. S., Prado, C., Patil, V, & Kacker, R. N.,  Combination Frequency Differencing for Identifying Design Weaknesses in Physical Unclonable Functions, (submitted for publication)

2021

Chandrasekaran, J., Lei, Y., Kacker, R., & Kuhn, D. R. (2021, April). A Combinatorial Approach to Explaining Image Classifiers. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 35-43). IEEE. 

Chandrasekaran, J., Lei, Y., Kacker, R., & Kuhn, D. R. (2021, April). A Combinatorial Approach to Testing Deep Neural Network-based Autonomous Driving Systems. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 57-66). IEEE.

Kampel, L., Simos, D. E., Kuhn, D. R., & Kacker, R. N. (2021). An exploration of combinatorial testing-based approaches to fault localization for explainable AI. Annals of Mathematics and Artificial Intelligence, 1-14.

Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2021). Combinatorial Coverage Difference Measurement. NIST Cybersecurity Whitepaper.

Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2021). Combinatorial Frequency Differencing. NIST Cybersecurity Whitepaper.

Lanus, E., Freeman, L. J., Kuhn, D. R., & Kacker, R. N. (2021, April). Combinatorial Testing Metrics for Machine Learning. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 81-84). IEEE.

Raunak, M. S., & Kuhn, R. (2021). Explainable Artificial Intelligence and Machine Learning. IEEE Computer, 54(10), 25-27.

Toohey, J. R., Raunak, M. S., & Binkley, D. (2021). From Neuron Coverage to Steering Angle: Testing Autonomous Vehicles Effectively. Computer, 54(8), 77-85.

Wu, J. C., & Kacker, R. N. (2021). Standard Errors and Significance Testing in Data Analysis for Testing Classifiers.

2020

Chandrasekaran, J., Feng, H., Lei, Y., Kacker, R., & Kuhn, D. R. (2020, August). Effectiveness of dataset reduction in testing machine learning algorithms. In 2020 IEEE International Conference On Artificial Intelligence Testing (AITest) (pp. 133-140). IEEE.

DR Kuhn, R Kacker, Y Lei, D Simos, "Combinatorial Methods for Explainable AI", Intl Workshop on Combinatorial Testing, Porto, Portugal, March 23-27, 2020.

Kuhn, R., Kacker, R. N., Lei, Y., & Simos, D. (2020). Input Space Coverage Matters. Computer, 53(1), 37-44.

2019

R. Kuhn, R. Kacker, An Application of Combinatorial Methods for Explainability in Artificial Intelligence and Machine Learning. NIST Cybersecurity Whitepaper, May 22, 2019. 

2018 and earlier

DR Kuhn, D Yaga, R Kacker, Y Lei, V Hu, Pseudo-Exhaustive Verification of Rule Based Systems, 30th Intl Conf on Software Engineering and Knowledge Engineering, July 2018.

D.R. Kuhn, I. Dominguez, R.N. Kacker and Y. Lei. "Combinatorial Coverage Measurement Concepts and Applications", 2nd Intl Workshop on Combinatorial Testing, Luxembourg, IWCT2013, IEEE, Mar. 2013.

Presentations

1. R. Kuhn, R. Kacker, Explainable AI, NIST presentation.  PDF Explainable AI   PPT Explainable AI  
2. D R Kuhn, R Kacker, Risk, Assurance, and Explainability for Autonomous Systems, Advancements in Test and Evaluation of Autonomous Systems (ATEAS) workshop, Dayton, OH, Oct, 2019. NIST presentation. 
3. R. Kuhn, Assured Autonomy - Problems and Possible Solutions, Hill AFB, Aug, 2021.