Published: July 25, 2017
Author(s)
Sergiy Vilkomir (East Carolina University), Aparna Alluri (East Carolina University), Richard Kuhn (NIST), Raghu Kacker (NIST)
Conference
Name: 2017 IEEE International Conference on Software Quality Reliability and Security (QRS-C 2017)
Dates: July 25-29, 2017
Location: Prague, Czech Republic
Citation: Proceedings. 2017 IEEE International Conference on Software Quality, Reliability and Security (Companion Volume) (QRS-C 2017), pp. 61-68
Software testing criteria differ in their effectiveness, the numbers of test cases required, and the processes of test generation. Specific criteria often are compared to random testing, and in some cases, random testing shows a surprisingly high level of effectiveness. One reason that this is the case is that any random test set has a specific level of coverage according to any coverage criterion. Numerical evaluation of coverage levels of random testing according to various coverage criteria is an interesting research task and is important in understanding the relationship between different testing approaches. In this paper, we performed an experimental evaluation of the coverage levels of random testing for two criteria: MC/DC and combinatorial t-way testing. Our experiments showed that, when the number of random test cases increased, a high level of coverage was reached rapidly, both for MC/DC and t-way. However, many more random tests are required to reach 100% coverage. An unexpected result was that there were significant differences in the measurement of partial MC/DC coverage by various tools. The results may be used to select optimal methods for practical testing and develop new testing methods based on the integration of existing approaches.
Software testing criteria differ in their effectiveness, the numbers of test cases required, and the processes of test generation. Specific criteria often are compared to random testing, and in some cases, random testing shows a surprisingly high level of effectiveness. One reason that this is the...
See full abstract
Software testing criteria differ in their effectiveness, the numbers of test cases required, and the processes of test generation. Specific criteria often are compared to random testing, and in some cases, random testing shows a surprisingly high level of effectiveness. One reason that this is the case is that any random test set has a specific level of coverage according to any coverage criterion. Numerical evaluation of coverage levels of random testing according to various coverage criteria is an interesting research task and is important in understanding the relationship between different testing approaches. In this paper, we performed an experimental evaluation of the coverage levels of random testing for two criteria: MC/DC and combinatorial t-way testing. Our experiments showed that, when the number of random test cases increased, a high level of coverage was reached rapidly, both for MC/DC and t-way. However, many more random tests are required to reach 100% coverage. An unexpected result was that there were significant differences in the measurement of partial MC/DC coverage by various tools. The results may be used to select optimal methods for practical testing and develop new testing methods based on the integration of existing approaches.
Hide full abstract
Keywords
combinatorial testing; MC/DC; pairwise; random testing; coverage
Control Families
None selected