Examples of combinatorial coverage achieved by real-world test suites in various application domains.
Application |
Config |
t = 2 |
t = 3 |
t = 4 |
t = 5 |
t = 6 |
Reference |
Spacecraft control |
132754262 |
0.940 |
0.831 |
0.668 |
0.536 |
Maximoff, J. R., Kuhn, D. R., Trela, M. D., & Kacker, R. A method for analyzing system state-space coverage within a t-wise testing framework. In 2010 IEEE ICST. |
|
Spacecraft component control |
273241102 |
0.962 |
0.917 |
0.819 |
0.658 |
0.467 |
Price, C., Kuhn, R., Forquer, R., Lagoy, A., Kacker, R., Evaluating the t-way Combinatorial Technique for Determining the Thoroughness of a Test Suite, NASA IV&V Workshop, 2013. |
Aircraft system test |
243142 |
1.000 |
1.000 |
0.987 |
0.697 |
Kuhn, D. R., Kacker, R. N., & Lei, Y. (2015). Combinatorial coverage as an aspect of test quality. CrossTalk, 28(2), 19-23. |
|
Apache Lang | .686 | .440 | Felbinger, H., Wotawa, F., & Nica, M. (2018, April). Adapting unit tests by generating combinatorial test data. In 2018 IEEE ICST (ICSTW) (pp. 352-355). IEEE. | ||||
Apache Math | .628 | .301 | same | ||||
JFreeChart | .577 | .250 | same | ||||
Joda-Time | .500 | .250 | same | ||||
Data analytics |
0.160 |
0.030 |
Smith, R., et al., "Measuring Combinatorial Coverage at Adobe", 2019 IEEE ICST. |
||||
Data analytics |
0.650 |
0.390 |
same |
||||
Data analytics |
0.120 |
0.010 |
same |
||||
Data analytics |
0.120 |
0.010 |
same |
||||
Data analytics |
0.230 |
0.060 |
same |
||||
Data analytics |
0.270 |
0.090 |
same |
||||
Train control |
0.830 |
0.550 |
0.320 |
0.180 |
0.100 |
Fifo, M., Enoiu, E., & Afzal, W. (2019, April). On measuring combinatorial coverage of manually created test cases for industrial software. In 2019 IEEE ICST (ICSTW) (pp. 264-267). |
Security and Privacy: assurance, modeling, testing & validation
Technologies: software & firmware