ct-competition

Welcome to the combinatorial interaction testing competition

View project on GitHub

First edition of the CT Competition

In this page you can find the information and the results obtained by each participant to the first edition of the CT competition.

Participants

  • APPTS: a tabu search-based optimization of Cas, uses ACTS for initial generation
  • CAgen: a multithreaded optimized IPO-based tool
  • IPO Solver: a C++ implementation of the IPO algorithm
  • pMEDICI: a multi-thread implementation of the MEDICI tool, based on Multi-valued Decision Diagrams

Benchmarks and execution rules

For this edition of the CT competition:

  • 300 benchmark models have been generated
  • Strengths from 2 to 5 have been used
  • Each benchmark has been executed 3 times for each tool and strength (you can find the list of all the execution results here), and the best run has been selected for the attribution of the score (see here for the best executions list)

Results

All the results reported into this section derives from the list of best executions.

The file containing all the CAs which have been analyzed for giving the score to each tool can be found here.

Validity and timeouts

All the tools have reported at least one timeout or produced an invalid test suite (even if not in all the categories and all strenghts). The list of timed out instances for each category and each strength can be found here, while the one of invalid instances can be found here.

Score - Time and Size

The competition score has been given considering in an equal way the generation time and the test suite size. However, for completeness, we here report in a separate way the score given either only considering the time or the test suite size, by distinguishing between different strengths.

Time score Size score

OVERALL ranking

This section reports the overall ranking, considering the aggregated score of each category. The detailed data can be found here for all the strenghts (or, if interested in a specific strength, you can look at the specific file - 2, 3, 4, 5)

  1. CAgen (1795.5 pts)
  2. APPTS (1384 pts)
  3. IPO Solver (967.5 pts)
  4. pMEDICI (566 pts)

For completeness, we here report the score given for each strength.

UNIFORM_BOOLEAN ranking

This section reports the ranking for the UNIFORM_BOOLEAN category. The detailed data can be found here for all the strenghts (or, if interested in a specific strength, you can look at the specific file - 2, 3, 4, 5)

  1. CAgen (614.5 pts)
  2. IPO Solver (468.5 pts)
  3. APPTS (452.5 pts)
  4. pMEDICI (274.5 pts)

For completeness, we here report the score given for each strength for this category.

UNIFORM_ALL ranking

This section reports the ranking for the UNIFORM_ALL category. The detailed data can be found here for all the strenghts (or, if interested in a specific strength, you can look at the specific file - 2, 3, 4, 5)

  1. CAgen (482 pts)
  2. APPTS (326 pts)
  3. IPO Solver (319 pts)
  4. pMEDICI (148.5 pts)

For completeness, we here report the score given for each strength for this category.

MCA ranking

This section reports the ranking for the MCA category. The detailed data can be found here for all the strenghts (or, if interested in a specific strength, you can look at the specific file - 2, 3, 4, 5)

  1. CAgen (423.5 pts)
  2. APPTS (298 pts)
  3. IPO Solver (273 pts)
  4. pMEDICI (107.5 pts)

For completeness, we here report the score given for each strength for this category.

BOOLC ranking

This section reports the ranking for the BOOLC category. The detailed data can be found here for all the strenghts (or, if interested in a specific strength, you can look at the specific file - 2, 3, 4, 5)

  1. CAgen (280.5 pts)
  2. APPTS (216.5 pts)
  3. pMEDICI (40 pts)

For completeness, we here report the score given for each strength for this category.

MCAC ranking

This section reports the ranking for the MCAC category. The detailed data can be found here for all the strenghts (or, if interested in a specific strength, you can look at the specific file - 2, 3, 4, 5)

  1. CAgen (140 pts)
  2. APPTS (192 pts)
  3. pMEDICI (62 pts)

For completeness, we here report the score given for each strength for this category.

NUMC ranking

None of the participants was able to deal with NUMC instances.