U.S. flag   An unofficial archive of your favorite United States government website
Dot gov

Official websites do not use .rip
We are an unofficial archive, replace .rip by .gov in the URL to access the official website. Access our document index here.

Https

We are building a provable archive!
A lock (Dot gov) or https:// don't prove our archive is authentic, only that you securely accessed it. Note that we are working to fix that :)

Combinatorial Testing

Combinatorial coverage examples

Examples of combinatorial coverage achieved by real-world test suites in various application domains. 

Application

Config

t = 2

t = 3

t = 4

t = 5

t = 6

Reference

Spacecraft control

132754262

0.940

0.831

0.668

0.536

 

Maximoff, J. R., Kuhn, D. R., Trela, M. D., & Kacker, R. A method for analyzing system state-space coverage within a t-wise testing framework. In 2010 IEEE ICST.

Spacecraft component control 

273241102

0.962

0.917

0.819

0.658

0.467

Price, C., Kuhn, R., Forquer, R., Lagoy, A., Kacker, R., Evaluating the t-way Combinatorial Technique for Determining the Thoroughness of a Test Suite, NASA IV&V Workshop, 2013.  

Aircraft system  test

243142

1.000

1.000

0.987

0.697

 

Kuhn, D. R., Kacker, R. N., & Lei, Y. (2015). Combinatorial coverage as an aspect of test quality. CrossTalk28(2), 19-23.

Apache Lang   .686 .440       Felbinger, H., Wotawa, F., & Nica, M. (2018, April). Adapting unit tests by generating combinatorial test data. In 2018 IEEE ICST (ICSTW) (pp. 352-355). IEEE.
Apache Math   .628 .301       same
JFreeChart   .577 .250       same
Joda-Time   .500 .250       same

Data analytics

 

0.160

0.030

     

Smith, R., et al., "Measuring Combinatorial Coverage at Adobe", 2019 IEEE ICST. 

Data analytics

 

0.650

0.390

     

same

Data analytics

 

0.120

0.010

     

same

Data analytics

 

0.120

0.010

     

same

Data analytics

 

0.230

0.060

     

same

Data analytics

 

0.270

0.090

     

same

Train control

 

0.830

0.550

0.320

0.180

0.100

Fifo, M., Enoiu, E., & Afzal, W. (2019, April). On measuring combinatorial coverage of manually created test cases for industrial software. In 2019 IEEE ICST (ICSTW) (pp. 264-267). 

Created May 24, 2016, Updated August 04, 2022