Application

Reference

Notes/Abstract

M Park, H Jang, T Byun, Yunja, "Property-based Testing for LG Home Appliances using Accelerated Software-in-the-Loop Simulation", Software Engineering in Practice (ICSE-SEIP '20), At Seoul

Vukovic, V., Djurkovic, J., Sakal, M., & Rakovic, L. (2020). An Empirical Investigation of Software Testing Methods and Techniques in the Province of Vojvodina. Tehnički vjesnik, 27(3), 687-696.

Li, X., Gao, R., Wong, W.E., Yang, C. and Li, D.,  Applying combinatorial testing in industrial settings. In 2016 IEEE Intl Conf on Software Quality, Reliability and Security (QRS) (pp. 53-60).

From January 2016 to February 2016, authors tested three real-life software systems using CT, and compared the results to errors that had been discovered using conventional methods, finding roughly 3X as many bugs in one-fourth of the time, for a 12X increase in test efficiency. 

AV Jayakumar, S Gautham, R Kuhn, B Simons, A Collins, T Dirsch, R Kacker, C Elks. Systematic Software Testing of Critical Embedded Digital Devices in Nuclear Power Applications, 31st Intl Symposium Software Reliability Engineering (ISSRE 2020)

Loyola Univ, NIST -

Cryptography - multiple companies

Mouha, N., Raunak, M.S., Kuhn, D.R. and Kacker, R., 2018. Finding bugs in cryptographic hash function  implementations. IEEE Trans Reliability, 67(3), pp.870-884.

Detected flaws in cryptographic software code, reducing the test set size by 700X as compared with exhaustive testing, while retaining the same fault- detection capability. 

Lockheed Martin -

Aerospace

Siemens -

Industrial controls

Ozcan, M., 2017, March. Applications of Practical Combinatorial Testing Methods at Siemens Industry Inc., Building Technologies Division. In 2017 IEEE Intl Conf on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 208-215). IEEE.

Applied combinatorial testing to industrial control systems, using mixed-strength covering arrays, “resulting in requiring fewer tests for higher strength coverage”. 

Adobe - 

Data analytics

Adobe - 

Data analytics

Smith, Riley, et al., "Applying Combinatorial Testing to Large-scale Data Processing at Adobe", 2019 IEEE Intl Conf on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 208-215). 

"In this paper, we report the practical application of combinatorial testing to the data collection, compression and processing components of the Adobe analytics product. Consequently, the effectiveness of combinatorial testing for this application is measured in terms of new defects found rather than detecting known defects from previous versions. The results of the application show that combinatorial testing is an effective way to improve validation for these components of Adobe Analytics."

LESSONS LEARNED GIVE RIGOUR AND EFFICIENCY TO ORDNANCE, AIRCRAFT AND SHIP TESTING 

"More recently the movement has incorporated high throughput testing (HTT) techniques using combinatorial methods to do highly efficient screening or validations, especially of simulations and to do more rigorous software verifications and cybersecurity protections with up to six-way combinations. The U.S. Defense methods have also spread to the U.S. Defense Industry initially for compliance purposes, meaning that these industries are now realising industry benefits."

T. McLean, Cyber Penetration Testing and Statistical Analysis in DT&E  Dataworks 2018, Defense and Aerospace Test and Analysis Wkshp

The project successfully implemented an unbiased and statistically based test methodology for Link 16 standards conformance that can ensure repeatability with a quantifiable increase in test space coverage. 

Using Combinatorial Test Design methods to select software test scenarios has repeatedly delivered large efficiency and thoroughness gains – which begs the questions: 

• Why are these proven methods not used everywhere?• Why do some efforts to promote adoption of new approaches stagnate?• What steps can leaders take to introduce successfully introduce and spread new test design methods?

Masuda, S., Nakamura, H. and Kajitani, K., 2018. Rule-based searching for collision test cases of autonomous vehicles simulation. IET Intelligent Transport Systems, 12(9), pp.1088-1095.

Smith, B., Feather, M. and Huntsberger, T., A Hybrid Method of Assurance Cases and Testing for Improved Confidence in Autonomous Space Systems. In 2018 AIAA Information Systems-AIAA Infotech@ Aerospace(p. 1981).

Abstract. We are investigating a new test development method that aims to maximize the confidence to be achieved by combining Assurance Cases with High Throughput Testing (HTT). Assurance Cases, developed for safety-critical systems, are a rigorous argument that the system satisfies a property (e.g., the Mars rover will not tip over during a traverse). They integrate testing, analysis, and environmental and operational assumptions, from which the set of conditions that testing must cover is determined. In our method, information from the Assurance Case is used to determine the test coverage needed, and then input to HTT to generate the minimal test suites needed to provide that coverage.

Garn, B., Kapsalis, I., Simos, D.E. and Winkler, S., On the applicability of combinatorial testing to web application security testing: a case study. Proc 2014 Wkshp Joining AcadeMiA and Industry Contributions to Test Automation and Model-Based Testing. ACM.

This paper reports on a case study done for evaluating and revisiting a recently introduced combinatorial testing methodology used for web application security purposes. It further reports on undertaken practical experiments thus strengthening the applicability of combinatorial testing to web application security testing.

Sagi, Bhargava Rohit. "Experimental Design in Game Testing." Rochester Institute of Technology (2016).

Combinatorial testing is a method of experimental design that is used to generate test cases and is primarily used for commercial software testing. In addition to the discussion of the implementation of combinatorial testing techniques in video game testing, we present a method for finding combinations resulting in video game bugs.

Mälardalen University

Redavid, C., & Farid, A. (2011). An overview of game testing techniques. Västerås: sn.

"It's always an important issue for testers and software managers to decide how much testing should be enough [11] Game quality has to be good enough for customer, but testing has to stop before the release date. It is neither practical nor economical, to test every possible combination of game event, conguration, function and options. Skipping some testing or taking shortcuts is always risky. Combinatorial testing is a quick way to find defects earlier in the game while keeping the test sets small, covering as much area as possible."

Jaguar Land Rover - 

Automotive

Dhadyalla, Gunwant, Neelu Kumari, and Timothy Snell. "Combinatorial testing for an automotive hybrid electric vehicle control system: a case study." 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops. IEEE, 2014.

Bozic, J., Garn, B., Kapsalis, I., Simos, D., Winkler, S. and Wotawa, F., 2015, August. Attack pattern-based combinatorial testing with constraints for web security testing. 2015 IEEE Intl Conf Software Quality, Reliability and Security 

The evaluated results indicate that both techniques succeed in detecting security leaks in web applications with different results, depending on the background logic of the testing approach. Last but not least, we claim that attack pattern-based combinatorial testing with constraints can be an alternative method for web application security testing, especially when we compare our method to other test generation techniques like fuzz testing.

Rockwell Collins -

Avionics

R. Bartholomew, An Industry Proof-of-Concept Demonstration of Automated Combinatorial Test,25th Annual IEEE Software Technology Conf., April 8-10, 2013, Salt Lake City, Utah.

“Industry proof-of-concept demonstration that used this approach to automate parts of the unit and integration testing of a 196 KSLOC avionics system. The goal was to see if it might cost-effectively reduce rework by reducing the number of software defects escaping into system test – if it was adequately accurate, rigorous, thorough, scalable, mature, easy to learn, easy to use, etc. Overcoming scalability issues required moderate effort, but in general it was effective – e.g., generating 47,040 test cases (input vectors, expected outputs) in 75 seconds, executing and analyzing them in 2.6 hours. It subsequently detected all seeded defects, and achieved nearly 100% structural coverage.”

Dominka S, Mandl M, Dübner M, Ertl D. Using combinatorial testing for distributed automotive features: Applying combinatorial testing for automated feature-interaction-testing. 2018 IEEE 8th Ann. Computing and Communication Wkshp and Conf (CCWC) Jan 8 (pp. 490-495)

Abstract—Modern passenger cars have a comprehensive embedded distributed system with a huge number of bus devices interlinked in several communication networks. The number of (distributed) features and hence the risk of undesired feature interaction within this distributed system rises significantly. Such distributed automotive features pose a huge challenge in terms of efficient testing. Bringing together Combinatorial Testing with Automated Feature-Interaction Testing reduces the testing effort for such features significantly.

Avaya Corp. - 

Telecommunications

Sherif, Anwar. "Combinatorial testing: Implementations in solutions testing." 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2016.

Ericsson S, Enoiu E. Combinatorial Modeling and Test Case Generation for Industrial Control Software using ACTS. 2018 IEEE Intl Conf Software Quality, Reliability and Security (QRS) 2018 Jul 16 (pp. 414-425)

"Our results show that not all combinations of algorithms and interaction strengths could generate a test suite within a realistic cut-off time. The results of the modeling process and the efficiency evaluation of ACTS are useful for practitioners considering to use combinatorial testing for industrial control software as well as for researchers trying to improve the use of such combinatorial testing techniques."

Mercedez-Benz - 

Automotive

Züfle, Siegmar, and Venkataraman Krishnamoorthy. "A process for nonfunctional combinatorial testing: Selection of parameter values from a nondiscrete domain space." 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops 

Ahmed BS, Pahim A, Junior CR, Kuhn DR, Bures M. Towards an Automated Unified Framework to Run Applications for Combinatorial Interaction Testing. arXiv preprint arXiv:1903.05387. 2019 Mar 13.

Sulake - video games/social network

Puoskari, E., Vos, T.E., Condori-Fernandez, N. and Kruse, P.M., 2013. Evaluating applicability of combinatorial testing in an industrial environment: A case study. 2013 Intl Wkshp Joining Academia and Industry Contributions to testing Automation ACM.

Chunduri, Annapurna, Robert Feldt, and Mikael Adenmark. "An effective verification strategy for testing distributed automotive embedded software functions: A case study." Intl Conf Product-Focused Software Process Improvement, pp. 233-248. Springer, 2016.

The proposed verification strategy to test distributed automotive embedded software functions has given promising results by providing means to identify test gaps and test redundancies. It helps establish an effective and feasible approach to capture function test coverage information that helps enhance the effectiveness of integration testing of the distributed software functions.

Vilkomir, S.A., Swain, W.T., Poore, J.H. and Clarno, K.T., 2008, June. Modeling input space for testing scientific computational software: a case study. Intl Conf Computational Science Springer, Berlin

Abstract. An application of a method of test case generation for scientific computational software is presented. NEWTRNX, neutron transport software being developed at Oak Ridge National Laboratory, is treated as a case study. A model of dependencies between input parameters of NEWTRNX is created. Results of NEWTRNX model analysis and test case generation are evaluated.

Lockheed Martin -

Failure analysis

Cunningham, A. M., Hagar, J., & Holman, R. J.  A System Analysis Study Comparing Reverse Engineered Combinatorial Testing to Expert Judgment. InSoftware Testing, Verification and Validation (ICST), 2012 IEEE Fifth Intl Conference on(pp. 630-635) IEEE.

Lockheed Martin F-16 ventral fin redesign “The historic analysis was able to determine a set of combinations, which isolated the problem and tested a solution. However, the original effort was expensive, time consuming, and required highly specialized knowledge from the expert to be effective. In the study, a series of iterations created combinatorial test cases which could have 'replicated' the original highly optimized and successful test program, without the expert.”

Tsumura, K., Washizaki, H., Fukazawa, Y., Oshima, K. and Mibe, R., 2016, April. Pairwise coverage-based testing with selected elements in a query for database applications. In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 92-101). IEEE.

Develops a method of applying combinatorial testing for use with SQL database query programs. Results showed that the pairwise tests detected “many bugs which are not detected by existing test methods based on predicates in the query”. 

Wu, H., Petke, J., Jia, Y., & Harman, M. (2018). An empirical comparison of combinatorial testing, random testing and adaptive random testing. IEEE Transactions on Software Engineering.

Our study was conducted on nine real- world programs under a total of 1683 test scenarios (combinations of available parameter and constraint information and failure rate). The results show significant differences in the techniques’ fault detection ability when faults are hard to detect (failure rates are relatively low). CT performs best overall; no worse than any other in 98 percent of scenarios studied.

Kuhn DR, Higdon JM, Lawrence JF, Kacker RN, Lei Y. Combinatorial methods for event sequence testing. In2012 IEEE Fifth International Conference on Software Testing, Verification and Validation 2012 Apr 17 (pp. 601-609). IEEE.

"The methods described in this paper were motivated by testing needs for systems that may accept multiple communication or sensor connections and generate output to several communication links and other interfaces, where it is important to test the order in which connections occur. Although pairwise event order testing (both A followed by B and B followed by A) has been described, our algorithm ensures that any t events will be tested in every possible t-way order."

Video coding

Hong, D., & Chae, S. I. (2014, June). Efficient test bitstream generation method for verification of HEVC decoders. In Consumer Electronics (ISCE 2014), The 18th IEEE International Symposium on (pp. 1-2).

Combinatorial methods provided an 84X efficiency improvement for testing conformance to a new video coding standard. The original conformance testing spec included 1,000,182 coding tree units, but using a 3-way covering array plus two more tests, they provided better coverage with 13,712 units. Coverage was measured in 'syntax elements', requiring > 90% coverage of the syntax elements. The authors say, "In the proposed method the SE coverage normalized by the number of CTUs is 84 times higher compared to that in the HEVC conformance test suite. This means that we can verify the HEVC decoders 84 times faster with the test bitstream set obtained by the proposed method, compared to the HEVC conformance test suite."

Fögen, K. and Lichter, H., 2018. A Case Study on Robustness Fault Characteristics for Combinatorial Testing-Results and Challenges. QuASoQ 2018, p.18.

Abstract.  Combinatorial strategies are extended to generate invalid test inputs but the effectiveness of negative test scenarios is yet unclear. Therefore, we conduct a case study and analyze 434 failures reported as bugs of a financial enterprise application. As a result, 51 robustness failures are identified including failures triggered by invalid value combinations and failures triggered by interactions of valid and invalid values.

Bergström, Henning, and Eduard Paul Enoiu. "Using timed base-choice coverage criterion for testing industrial control software." 2017 IEEE Intl Conf Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2017.

Applies combinatorial test methods to industrial control software.  “We found that tests generated for timed base-choice criterion show better code coverage (7% improvement) and fault detection (27% improvement) in terms of mutation score than tests satisfying base-choice coverage criterion. The results demonstrate the feasibility of applying timed base-choice criterion for testing industrial control software.”

Sánchez, A.B., Segura, S., Parejo, J.A. and Ruiz-Cortés, A., 2017. Variability testing in the wild: the Drupal case study. Software & Systems Modeling, 16(1)"

"Among other results, we identified 3392 faults in single features and 160 faults triggered by the interaction of up to four features in Drupal v7.23. We also found positive correlations relating the number of bugs in Drupal features to their size, cyclomatic complexity, number of changes and fault history. To show the feasibility of our work, we evaluated the effectiveness of non-functional data for test case prioritization in Drupal. Results show that non-functional attributes are effective at accelerating the detection of faults, outperforming related prioritization criteria as test case similarity."

Srikanth, H. and Cohen, M.B., 2011, September. Regression testing in software as a service: An industrial case study. In 2011 27th IEEE Intl Conf Software Maintenance (ICSM) IEEE.

Web administration

N. Condori-Fernandez, T. Vos, P.M. Kruse, E. Brosse, A. Bagnato. Analyzing the Applicability of a Combinatorial Testing Tool in an Industrial Environment, Tech. Rpt. UU-CS-2014-008, May 2014, Univ. of Utrecht.

"The main outcomes of the presented study are: (1) with the test suite designed with the CTE, the testers were able to find faults that the traditional test suites did not find, one of them a severe fault; (2) the company realized that the current coverage metrics used for evaluating the quality of test suites needs to be changed to a more sophisticated one; (3) SOFTEAM’s motivation to do more case studies with the CTE is high"

Raunak MS, Kuhn DR, Kacker R. Combinatorial testing of full text search in Web applications. 2017 IEEE Intl Conf on Software Quality, Reliability and Security Companion (QRS-C)

Testing full-text search in a database web application. We develop test-case selection techniques, where test strings are synthesized using characters or string fragments that may lead to system failure. Demonstrated discovery of a number of "corner-cases" that had not been identified previously. We also present simple heuristics for isolating the fault causing factors that can lead to such system failures. The test method and input model described in this paper have immediate application to other systems that provide complex full text search.

Manchester S, Bryce R, Sampath S, Samant N, Kuhn DR, Kacker R. Applying higher strength combinatorial criteria to test case prioritization: a case study.

Using the CPUT tool, we "conduct an empirical study where we compare 2-way and 3-way combinatorial coverage of inter-window parameter interactions in terms of the rate of fault detection for a web application called Schoolmate and a user-session-based test suite. Our results show that the rate of fault detection for 2-way and 3-way prioritization are within 1% of each other, but 2-way provides a slightly better result."

Web applications

Maughan, C.Test Case Generation Using Combinatorial Based Coverage for Rich Web Applications. Logan, UT: Utah State Univ (2012).

Compared exhaustive (discretized values) w/ CT. 2-way tests found all but one fault found by exhaustive using < 13% of tests required for exhaustive.

MP3 audio

Zhang, Z., Liu, X., & Zhang, J. (2012, April). Combinatorial Testing on ID3v2 Tags of MP3 Files. InSoftware Testing, Verification and Validation (ICST), 2012 IEEE Fifth International Conf

Most faults detected by 1-way and 2-way tests, with one caused by 4-way interaction.

iGate Corp. -

Product engineering; banking & financial services; insurance

M. Mehta, R. Philip, Applications of Combinatorial Testing methods for Breakthrough Results in Software Testing, 2nd Intl. Wkshp on Combinatorial Testing, Luxembourg, IEEE, Mar. 2013.

"Combinatorial Testing (CT) approach has greatly helped our projects from different domains to optimize testing effort without compromising on testing quality. We were able to achieve breakthrough business results. CT based freeware tools such as All Pairs & ACTS are of great help for testing professionals to optimize effort and reduce learning curve."

Various

L. Shikh Gholamhossein Ghandehari, M. N. Bourazjany, Yu Lei, R.N. Kacker and D.R. Kuhn, "Applying Combinatorial Testing to the Siemens Suite", 2nd Intl. Wkshp Combinatorial Testing, Luxembourg, IEEE, Mar. 2013.

"Modeled the seven programs in the Siemens suite and applied combinatorial testing to these programs. The results show that combinatorial testing can detect most faulty versions of the Siemens programs, and is more effective than random testing."

Borazjany MN, Ghandehari LS, Lei Y, Kacker R, Kuhn R. An input space modeling methodology for combinatorial testing. 2013 IEEE Sixth Intl Conf on Software Testing, Verification and Validation Wkshp

This paper describes a method for modeling input structures and parameters, and compares combinatorial with random testing using the same input models.  It is shown that combinatorial testing provides better structural coverage and detects more errors than random. 

Covering array tool

Borazjany, M. N., Yu, L., Lei, Y., Kacker, R., & Kuhn, R. (2012, April). Combinatorial Testing of ACTS: A Case Study. Software Testing, Verification and Validation (ICST), 2012 IEEE Fifth Intl Conf(pp. 591-600)

Applied 2-way and 3-way tests; approx. 80% module and branch coverage, 88% statement coverage 15 faults found; 2-way tests found as many as 3-way.

NIST -

Web browser
DOM modules

C. Montanez, D.R. Kuhn, M. Brady, R. Rivello, J. Reyes, M.K. Powers,Evaluation of Fault Detection Effectiveness for Combinatorial and Exhaustive Selection of Discretized Test Inputs Software Quality Professional - June, 2012.

Compared exhaustive (discretized values) w/ CT, 2-way to 6-way. 4-way tests found all faults using < 5% of tests required for exhaustive, for a 20X reduction in test set size.