Fourth Round of the SBST Java Unit Testing Tool Contest (with SBST 2016 @ ICSE)

After three successful competitions we, again this year, invite developers of tools for Java unit testing at the class level—both SBST and non-SBST—to participate in the 4th round of our tools competition!

Competition entries are in the form of short papers (maximum of 4 pages) describing an evaluation of your tool against a benchmark supplied by the workshop organizers. In addition to comparing your tool to other popular and successful tools such as Randoop, we will manually create unit tests for the classes under test, to be able to obtain and compare benchmark scores for manual and automated test generation!

The competition paper should be uplaoded to Easychair ( ).

The results of the tools competition will be presented at the SBST 2016 workshop.

Importante dates

  • Deadline for installing a copy of your tool on the server ready to be run and collect the results: 22th of January
  • Test will be run and results will be communicated after: 5th of February
  • Deadline for uploading the 4 page competition paper: 18th of February
  • Camera ready deadline is: 26th of February

The Contest

The contest is targeted at developers/vendors of testing tools that generate test input data for unit testing java programs at the class level. Each tool will be applied on a set of java classes taken from open-source projects, and selected by the contest organization. The participating tools will have a fixed time budget to generate tests for Java classes. The score will be calculated based on:

  • statement and branch coverage ratios
  • fault detection and mutation scores
  • preparation, generation and execution times

Each participant should install a copy of their tool on the server where the contest will be run. To this end each participant will have SSH-access to the contest-server. The benchmark infrastructure will run the tools and measure their outputs fully automatically, therefore tools must be capable of running without human intervention.


If you are interested in participating, please send a mail to Tanja Vos describing the following characteristics of your tool: 1) name, 2) testing techniques implemented (SBST or other), 3) compatible operating systems, 4) tool inputs and outputs, and 5) optionally any evaluative studies already published.

You will be sent credentials to log-in to the sbst-contest-server.


To allow automatic execution of the participating tools, these need to be configured and installed on the contest-server:

  • Host:
  • OS: Ubuntu 12.04 LTS

You should install and configure your testing tool in the home directory of your account. You can basically do that in any way you want with the following exceptions.

  • You must have an executable (or shell script) named $HOME/runtool that implements the protocol described below
  • Your tool must store intermediate data in $HOME/temp/data
  • Your tool must output the generated test cases in JUnit4 format in $HOME/temp/testcases

The Benchmark Automation Protocol

The runtool script/binary is the interface between the benchmark framework and the participating tools. The communication between runtool and the benchmark framework is done through a very simple line based protocol over the standard input and output channels. The following table describes the protocol, every step consists of a line of text received by the runtool program on STDIN or sent to STDOUT.

1 BENCHMARK Signals the start of a benchmark run; directory $HOME/temp is cleared
2 directory Directory with the source code SUT
3 directory Directory with compiled class files of the SUT
4 number Number of entries in the class path (N)
5 directory/jar file Class path entry (repeated N times)
6 number Number of classes to be covered (M)
7 CLASSPATH Signals that the testing tool required additional classpath entries
8 Number Number of additional class path entries (K)
9 directory/jar file Repeated K times
10 READY Signals that the testing tool is ready to receive challenges
11 time budget Scoring for each class under test will take place after a fixed amount of time, in seconds. Any test artefact generated after this time will be ignored.
12 class name The name of the class for which unit tests must be generated.
13 READY Signals that the testing tool is ready to receive more challenges; test cases in $HOME/temp/testcases are analyzed; subsequently $HOME/temp/testcases is cleared; goto step 11 until M class names have been processed

To ease the implementation of a runtool program according to the protocol above, we provide a skeleton implementation in Java.

Test the Protocol

In order to test whether your runtool implemented the protocol correctly, we installed a utility called sbstcontest on the machine. If you run it, it will output:

sbstcontest <toolname> <benchmark> <tooldir> <run_number> <timebudget>
Available benchmarks: [Closure-9, Math-9, Chart-5, Time-6, Lang-61]

The first line shows how the tool is used. <toolname> is a short string identifier for your testing tool, <benchmark> is one of the installed benchmarks as shown in the second line. <tooldirectory> is the directory where your runtool resides and <run_number> is a positive number greater than 0. The benchmarks are collections of classes from different open source projects. An example invocation would be (120 seconds of time budget):

sbstcontest MyToolName Math-9 . 1 120

If you implemented the protocol correctly and generated all the test cases, the tool will create a transcript file in the runtool’s directory. This file will show you several statistics, such as achieved branch coverage, mutation score, etc.

You will be able to run the full set of benchmarks through the script: MyToolName runs-number <timebudget>

It will run each available benchmark “runs-number” times. Output is stored in ./results folder. You may also want to obtain a single transcript.csv file from all runs: ./results MyToolName

Test Case Format

The tests generated by the participating tools must be stored as one or more java files containing JUnit4 tests.

  • declare classes public
  • add zero-argument public constructor
  • annotate test methods with @Test
  • declare test methods public

The generated test cases will be compiled against

  • JUnit 4.10
  • The benchmark SUT
  • Any dependencies of the benchmark SUT