U.S. flag   An unofficial archive of your favorite United States government website
Dot gov

Official websites do not use .rip
We are an unofficial archive, replace .rip by .gov in the URL to access the official website. Access our document index here.


We are building a provable archive!
A lock (Dot gov) or https:// don't prove our archive is authentic, only that you securely accessed it. Note that we are working to fix that :)

This is an archive
(replace .gov by .rip)

Combinatorial Testing

Workshop Papers

In 2012, we co-founded the International Workshop on Combinatorial Testing, focused on theory and application of CT.  Papers from previous workshops are listed below. 


IWCT 2021

  • A Combinatorial Approach to Explaining Image Classifiers
    Jaganmohan Chandrasekaran, Yu Lei, Raghu Kacker and Richard Kuhn.
  • A Practical Method for API Testing in the Context of Continuous Delivery and Behavior Driven Development  
    Brian Elgaard Bennett.
  • An Environment for Benchmarking Combinatorial Test Suite Generators
    Andrea Bombarda, Edoardo Crippa and Angelo Gargantini.
  • A Combinatorial Approach to Testing Deep Neural Network-based Autonomous Driving Systems
    Jaganmohan Chandrasekaran, Yu Lei, Raghu Kacker and Richard Kuhn.
  • SYSMODIS: A Systematic Model Discovery Approach
    Omer Korkmaz and Cemal Yilmaz.
  • Computing Sequence Covering Arrays using Unified Combinatorial Interaction Testing
    Hanefi Mercan and Cemal Yilmaz.
  • Combinatorial Testing Metrics for Machine Learning
    Erin Lanus, Laura J. Freeman, D. Richard Kuhn and Raghu N. Kacker.
  • Combinatorially XSSing Web Application Firewalls
    Bernhard Garn, Daniel Sebastian Lang, Manuel Leithner, Rick Kuhn, Raghu Kacker and Dimitris E. Simos.

IWCT 2020

  • Toward an Encoding Approach to Interaction-based Test Suite Minimization
    Kohei Nishino, Takashi Kitamura, Tomoji Kishi, Cyrille Artho
  • An Automata-Based Generation Method for Combinatorial Sequence Testing of Finite State Machines
    Andrea Bombarda, Angelo Gargantini
  • Combinatorial Methods for Explainable AI
    Rick Kuhn, Raghu Kacker, Jeff Lei, Dimitris Simos
  • Generation of Invalid Test Inputs from Over-Constrained Test Models for Combinatorial Robustness Testing
    Konrad Fögen, Horst Lichter
  • On Using Ontologies for Testing Compilers
    Yihao Li, Franz Wotawa
  • A Comparison Infrastructure for Fault Characterization Algorithms
    Torben Friedrichs, Konrad Fögen, Horst Lichter
  • Avocado: Open-Source Flexible Constrained Interaction Testing for Practical Application
    Jan Richter, Bestoun S. Ahmed, Miroslav Bures, Cleber R. Rosa Junior
  • CAGEN: A fast combinatorial test generation tool with support for constraints and higher-index arrays
    Michael Wagner, Kristoffer Kleine, Dimitris Simos, Rick Kuhn, Raghu Kacker


IWCT 2019

  • A Framework for Automated Combinatorial Test Generation
    Joshua Bonn, Konrad Fögen and Horst Lichter
  • On the Industrial Application of Combinatorial Testing for Autonomous Driving Functions
    Jianbo Tao, Yihao Li, Franz Wotawa, Hermann Felbinger and Mihai Nica
  • An Approach to T-way Test Sequence Generation With Constraints
    Feng Duan, Yu Lei, Raghu Kacker and Rick Kuhn
  • Efficient and Guaranteed Detection of t-way Failure-inducing Combinations
    Paolo Arcaini, Angelo Gargantini and Marco Radavelli
  • Using Parameter Mapping to Avoid Forbidden Tuples in a Covering Array
    Chang Rao, Nan Li, Yu Lei, Raghu Kacker, Richard Kuhn and Jin Guo
  • Partitioned Search with Column Resampling for Locating Array Construction
    Erin Lanus, Charles Colbourn and Douglas Montgomery
  • Measuring Combinatorial Coverage at Adobe
    Riley Smith, Darryl Jarman, Richard Kuhn, Raghu Kacker, Dimitris Simos and Jared Bellows
  • On Measuring Combinatorial Coverage of Manually Created Test Cases for Industrial Software  
    Miraldi Fifo, Eduard Paul Enoiu and Wasif Afzal
  • Reduce Test Cost by Reusing Test Oracles through Combinatorial Join 
    Hiroshi Ukai, Xiao Qu, Hironori Washizaki and Yoshiaki Fukazawa
  • Applying Combinatorial Testing to Large-scale Data Processing at Adobe
    Riley Smith, Darryl Jarman, Richard Kuhn, Raghu Kacker, Dimitris Simos, Ludwig Kampel, Manuel Leithner and Gabe Gosney
  • Practical Fault Localization with Combinatorial Test Design
    Dale Blue, Andrew Hicks, Rachel Tzoref-Brill and Ryan Rawlins
  • An Industrial Study on Applications of Combinatorial Testing in Modern Web Development and Cloud Computing   
    Murat Ozcan
  • Crucial Tool Features for Successful Combinatorial Input Parameter Testing in an Industrial Application   
    Christian Eitner and Franz Wotawa
  • Scenario Sampling for Cyber Physical Systems using Combinatorial Testing 
    Akihisa Yamada, Clovis Eberhart, Fuyuki Ishikawa and Nian-Ze Lee


IWCT 2018 

  • A Satisfiability-Based Approach to Generation of Constrained Locating Arrays
    Hao Jin, Takashi Kitamura, Eun-Hye Choi and Tatsuhiro Tsuchiya
  • Optimal Scheduling for Combinatorial Software Testing and Design of Experiments
    Robert V. Binder
  • Enumerator: An Efficient Approach for Enumerating all Valid t-tuples (short 20 mins)
    Hanefi Mercan, Kamer Kaya and Cemal Yilmaz
  • Combinatorial Test Design – smarter way to connect with the business for quality coverage
    Saritha Route and Sudheer Pendela
  • Bringing combinatorial interaction testing modeling and generation to the web
    Angelo Gargantini and Marco Radavelli
  • CAmetrics: A Tool for Advanced Combinatorial Analysis and Measurement of Test Sets
    Manuel Leithner, Kristoffer Kleine and Dimitris E. Simos
  • Combinatorial Testing with Constraints for Negative Test Cases (short 20 mins)
    Konrad Fögen and Horst Lichter
  • Practical Amplification of Condition/Decision Test Coverage by Combinatorial Testing
    Artur Andrzejak and Thomas Bach
  • The effect of time-between-events for sequence interaction testing of a real-time system
    Chek Pin Yang, Gunwant Dhadyalla, James Marco and Paul Jennings
  • Combining Combinatorial Testing and Metamorphic Testing for Testing a Logic-based Non-monotonic Reasoning System (short 20 mins)
    Franz Wotawa
  • Adapting Unit Tests by Generating Combinatorial Test (short 20 mins)
    Hermann Felbinger, Franz Wotawa and Mihai Nica

IWCT 2017 

  • Estimating t-way Fault Profile Evolution During Testing
    D. Richard Kuhn, Raghu N. Kacker and Jeff Lei[slides]
  • Mutation Score, Coverage, Model Inference: Quality Assessment For t-way Combinatorial Test-Suites
    Hermann Felbinger, Franz Wotawa and Mihai Nica[slides]
  • Optimizing IPOG’s Vertical Growth with Constraints Based on Hypergraph Coloring
    Feng Duan, Yu Lei, Linbin Yu, Raghu N. Kacker and D. Richard Kuhn  [slides]
  • Test Case Generation with Regular Expressions and Combinatorial Techniques
    Macario Polo, Francisco Ruiz Romero, Rosana Rodríguez-Bobada Aranda and Ignacio García-Rodríguez
  • Applying Combinatorial Testing to High-Speed Railway Track Circuit Receiver
    Chang Rao, Jin Guo, Nan Li, Jeff Lei, Yadong Zhang, Yao Li and Yaxin Cao  [slides]
  • Applications of Practical Combinatorial Testing Methods at Siemens Industry Inc.
    Murat Ozcan  [slides]
  • Using Timed Base-Choice Coverage Criterion for Testing Industrial Control Software Bergstrom and Eduard Enoiu  [slides]
  • Finding Minimum Locating Arrays Using a SAT Solver
    Tatsuya Konishi, Hideharu Kojima, Hiroyuki Nakagawa and Tatsuhiro Tsuchiya  [slides]
  • Test Optimisation using Combinatorial Test Design
    Saritha Route  [slides]
  • Building Combinatorial Test Input Model from Use Case Artefacts
    Preeti S, Milind B, Medhini S. Narayan and Krishnan Rangarajan  [slides]
  • Combinatorial Methods for Modelling Composed Software Systems
    Ludwig Kampel, Bernhard Garn and Dimitris E. Simos  [slides]
  • Combinatorial Interaction Testing for Automated Constraint Repair
    Angelo Gargantini, Justyna Petke and Marco Radavelli  [slides]
  • A composition-based method for combinatorial test design (short 15 mins)
    Anna Zamansky, Eitan Farchi, Sery Khoury and Amir Schwartz  [slides]
  • Applying Combinatorial Testing to Data Mining Algorithms
    Jaganmohan Chandrasekaran, Huadong Feng, Yu Lei, D. Richard Kuhn and Raghu N. Kacker  [slides]
  • Combinatorial Testing on Implementations of HTML5 Support
    Xi Deng, Tianyong Wu, Jun Yan and Jian Zhang  [slides]
  • Combinatorial Testing on MP3 for Audio Players (short 15 mins)
    Shaojiang Wang, Tianyong Wu, Yuan Yao, Beihong Jin and Liping Ding   [slides]


IWCT 2016 

  • Evaluating the Effectiveness of BEN in Locating Different Types of Software Fault
    Jaganmohan Chandrasekaran, Laleh Sh. Ghandehari, Yu Lei, Raghu Kacker and Richard Kuhn [slides]
  • Analyzing Minimal Failure-Causing Schemas in Siemens Suite 
    Chiya Xu, Yuanchao Qi, Ziyuan Wang and Weifeng Zhang
  • Should We Care about “Don’t Care” Testing Inputs? Empirical Investigation of Pair-Wise TestingSergiy Vilkomir and Galen Pennell  [slides]
  • Towards Automatic Cost Model Discovery for Combinatorial Interaction Testing
    Gulsen Demiroz and Cemal Yilmaz  [slides]
  • Combinatorial Testing Implementations in Solutions Testing
    Anwar Sherif
  • Embedded Functions for Constraints and Variable Strength in Combinatorial Testing
    George Sherwood  [slides]
  • Test Oracles and Test Script Generation in Combinatorial Testing
    Peter M. Kruse  [slides]
  • Pseudo-exhaustive Testing of Attribute Based Access Control Rules
    Rick Kuhn, Vincent Hu, David Ferraiolo, Raghu Kacker and Yu Lei
  • Applying a pairwise coverage criteria to scenario-based testing 
    Lydie Du Bousquet, Mickaël Delahaye and Catherine Oriat [slides]
  • Pairwise Coverage-based Testing with Selected Elements in a Query for Database Applications
    Koji Tsumura, Hironori Washizaki, Yoshiaki Fukazawa, Keishi Oshima and Ryota Mibe


IWCT 2015

  • Embedded Functions in Combinatorial Test Designs 
    George Sherwood. 
  • Introducing Combinatorial Testing in the Organization a Report on a First Attempt
    Douglas Hillmer. 
  • Combining Minimization and Generation for Combinatorial Testing
    Itai Segall, Rachel Tzoref-Brill and Aviad Zlotnick. 
  • A First Systematic Mapping Study on Combinatorial Interaction Testing for Software Product Lines
    Roberto Erick Lopez-Herrejon, Stefan Fischer, Rudolf Ramler and Alexander Egyed. 
  • Evaluation of the IPO-Family Algorithms for Test Case Generation in Web Security Testing
    Josip Bozic, Bernhard Garn, Dimitris. E. Simos and Franz Wotawa. 
  • A process for non-functional combinatorial testing (short)
    Siegmar Züfle and Venkataraman Krishnamoorthy. 
  • Combinatorial Interaction Testing of Tangled Configuration Options (short)
    Arsalan Javeed and Cemal Yilmaz. 
  • Relationship between Pair-wise and MC/DC Testing: Initial Experimental Results (short)
    Sergiy Vilkomir and David Anderson. 
  • Why Combinatorial Testing Works: Analyzing Minimal Failure-Causing Schemas in Logic Expressions (short)
    Ziyuan Wang and Yuanchao Qi
  • Constraint Handling In Combinatorial Test Generation Using Forbidden Tuples
    Linbin Yu, Feng Duan, Yu Lei, Raghu N. Kacker and D. Richard Kuhn. 
  • Using Decision Trees to aid Algorithm Selection in Combinatorial Interaction Test Generation
    Angelo Gargantini and Paolo Vavassori. 
  • Flattening or Not of the Combinatorial Interaction Testing Models?(short)
    Christopher Henard, Mike Papadakis and Yves Le Traon. 
  • Improving IPOG’s Vertical Growth Based on a Graph Coloring Scheme
    Feng Duan, Linbin Yu, Yu Lei, Raghu N. Kacker and D. Richard Kuhn. 
  • Equivalence Class Verification and Oracle-free Testing Using Two-layer Covering Arrays (short)
    Rick Kuhn, Raghu Kacker, Yu Lei and Jose Torres-Jimenez. 
  • BEN: A Combinatorial Testing-Based Fault Localization Tool (short) 
    Laleh Shikh Gholamhosseinghandehari, Jaganmohan Chandrasekaran, Yu Lei, Raghu Kacker and Richard Kuhn. 


IWCT 2014 

  • Combinatorial Testing for an Automotive Hybrid Electric Vehicle Control System - A Case Study, Gunwant Dhadyalla, Neelu Kumari and Timothy Snell
  • Eris: A Tool for Combinatorial Testing of the Linux System Call Interface,
    Bernhard Garn and Dimitris. E. Simos
  • An Empirical Comparison of Combinatorial and Random Testing,
    Laleh Shikh Gholamhosseingghandehari, Jacek Czerwonka, Yu Lei, Soheil Shafiee, Raghu Kacher and Richard Kuhn
  • Using Combinatorial Approaches for Testing Mobile Applications,
    Sergiy Vilkomir and Brandi Amstutz
  • Combinatorial interaction testing of a Java Card static verifier (short paper),
    Andrea Calvagna, Andrea Fornaia and Emiliano Tramontana
  • Extracting the Combinatorial Test Parameters and Values from UML Sequence Diagrams,
    Preeti Satish, Arinjita Paul and Krishnan Rangarajan
  • Validation of Models and Tests for Constrained Combinatorial Interaction Testing,
    Paolo Arcaini, Angelo Gargantini and Paolo Vavassori
  • Functional Dependence and Equivalence Class Factors in Combinatorial Test Designs,
    George Sherwood
  • Combinatorial Testing with Order Requirements,
    Eitan Farchi, Itai Segall, Rachel Tzoref-Brill and Aviad Zlotnick
  • Test Suite Reduction by Combinatorial-based Coverage of Event Sequences (short paper),
    Quentin Mayo, Renee Bryce and Ryan Michaels
  • Test Suite Prioritization by Switching Cost,
    Huayao Wu, Changhai Nie and Fei-Ching Kuo
  • Refining the Randomized Post-optimization Strategy for Covering Arrays,
    Xiaohua Li, Zhao Dong, Huayao Wu, Changhai Nie and Kai-Yuan Cai
  • Introducing Combinatorial Testing in a Large Organization: Pilot Project Experience Report,
    Jon Hagar, Rick Kuhn, Raghu Kacker and Tom Wissink
  • Estimating Fault Detection Effectiveness,
    Rick Kuhn, Raghu Kacker and Yu Lei


IWCT 2013 

  • Empirically Identifying the Best Greedy Algorithm for Covering Array Generation,
    Changhai Nie, Jing Jiang, Huayao Wu, Hareton Leung and Charles J. Colbourn
  • Evaluation of t-wise Approach for Testing Logical Expressions in Software,
    Sergiy Vilkomir, Oleksii Starov and Ranjan Bhambroo
  • On Use of Coverage Metrics in Assessing Effectiveness of Combinatorial Test Designs,
    Jacek Czerwonka
  • Cascade: A Test Generation Tool For Combinatorial Testing,
    Yong Zhao, Zhiqiang Zhang, Jun Yan and Jian Zhang
  • Identifying Failure-Inducing Combinations Using Tuple Relationship,
    Xintao Niu, Changhai Nie, Lei Yu and Alvin Chan
  • Better Bounds for Event Sequencing Testing,
    Oded Margalit
  • A Study in Prioritization for Higher Strength Combinatorial Testing,
    Xiao Qu and Myra Cohen
  • Combinatorial Test Architecture Design Using Viewpoint diagram,
    Yasuharu Nishi, Tetsuro Katayama and Satomi Yoshizawa
  • Experimenting with Category Partition's 1-way and 2-way test selection criteria,
    Yvan Labiche and Faezeh Sadeghi
  • Using Projections to Debug Large Combinatorial Models,
    Eitan Farchi, Itai Segall and Rachel Tzoref-Brill
  • Combinatorial Interaction Testing with Multi-Perspective Feature Models,
    Sachin Patel, Priya Gupta and Vipul Shah
  • Deriving Combinatorial Test Design Model from UML Activity Diagram,
    Preeti Satish, Sheeba K and Krishnan Rangarajan
  • Combinatorial Testing for Feature Models Using CitLab,
    Andrea Calvagna, Angelo Gargantini and Paolo Vavassori
  • Applications of Combinatorial Testing methods for Breakthrough Results in Software Testing,
    Manish Mehta and Roji Philip
  • Combinatorial Coverage Measurement Concepts and Applications,
    Rick Kuhn, Itzel Dominguez, Raghu Kacker and Yu Lei
  • Applying Combinatorial Testing to the Siemens Suite,
    Laleh Shikh Gholamhossein Ghandehari, Mehra N. Bourazjany, Yu Lei, Raghu Kacker and Richard Kuhn
  • An Input Space Modeling Methodology for Combinatorial Testing,
    Mehra Nouroz Borazjany, Laleh Shikh Gholamhossein Ghandehari, Yu Lei, Raghu Kacker and Rick Kuhn


IWCT 2012

  • Effectiveness of Pair-wise Testing for Software with Boolean Inputs  
    William Alton Ballance, Sergiy Vilkomir, William Jenkins
  • Combinatorial Testing of ACTS: A Case Study  
    Mehra N.Borazjany, Linbin Yu, Yu Lei, Raghu Kacker, Rick Kuhn
  • A System Analysis Study Comparing Reverse Engineered Combinatorial Testing to Expert Judgment 
    Atlee M. Cunningham, Jr., Jon Hagar, Ryan J. Holman
  • CITLAB: a Laboratory for Combinatorial Interaction Testing 
    Angelo Gargantini, Paolo Vavassori
  • Identifying Failure-Inducing Combinations in a Combinatorial Test Set  
    Laleh Shikh Gholamhossein Ghandehari, Yu Lei, Tao Xie, Richard Kuhn, Raghu Kacker
  • Combinatorial Methods for Event Sequence Testing  
    D. Richard Kuhn, James M. Higdon, James F. Lawrence, Raghu N. Kacker, Yu Lei
  • Combinatorial Test Design in the TOSCA Testsuite: Lessons Learned and Practical Implications 
    Rudolf Ramler, Theodorich Kopetzky
  • Combinatorial Interaction Testing for Test Selection in Grammar-Based Testing 
    Elke Salecker, Sabine Glesner
  • Common Patterns in Combinatorial Models 
    Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick
  • Simplified Modeling of Combinatorial Test Spaces 
    Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick
  • Isolating Failure-Inducing Combinations in Combinatorial Testing using Test Augmentation and Classification 
    Kiran Shakya, Tao Xie, Nuo Li, Yu Lei, Raghu Kacker, Richard Kuhn
  • Combinatorial Testing on ID3v2 Tags of MP3 Files 
    Zhiqiang Zhang, Xiaojian Liu, Jian Zhang

Created May 24, 2016, Updated October 12, 2021