Tutorial Part 1: EvoSuite on the Command Line

(Help us improve this document on our wiki)

What you will learn in this tutorial

  • Obtaining EvoSuite
  • Generating tests with EvoSuite on the command line
  • Running EvoSuite tests on the command line
  • Changing EvoSuite properties
  • Using EvoSuite together with existing tests

Prerequisites

NOTE: This tutorial is mainly written for unix-like operating systems (e.g., Linux, MacOS). If you are a Windows user, please look for windows-specific commands/tips.

The first prerequisite for this tutorial is a working Java 8 JDK installation and a command-line prompt, with java and javac on the classpath. To check if you are set up, fire up a terminal and try the following:

javac -version

If you have JDK installed, you will see an output similar to the following:

javac 1.8.0_51

The specific build version (51 in this case) does not matter. This tutorial should work with Oracle JDK or OpenJDK. If you do not have javac, then you will need to get a JDK (Oracle JDK or [Open JDK](openjdk download)).

The second prerequisite is Apache Maven. In the first part of the tutorial we will just use Maven to compile our example project, but in the second part we will be integrating EvoSuite into the Maven setup, so please make sure you have version 3.1 or newer installed. To determine which version of Maven you are using, type the following command:

mvn -version

This should give you detailed information about the version, e.g.

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00)

Anything greater or equal to 3.1 is fine. If you don’t have Maven, or have an older version, please install it. You can find instructions online at https://maven.apache.org.

Preparing the class under test

For the first part of the tutorial, we will be testing a simple Stack datastructure implementation. Please download and extract the archive containing this example.

wget http://evosuite.org/files/tutorial/Tutorial_Stack.zip
unzip Tutorial_Stack.zip

WINDOWS: Download the tutorial http://evosuite.org/files/tutorial/Tutorial_Stack.zip and extract it to your desired folder. Then open Command Prompt (cmd) and cd to the extracted tutorial folder.

Now let’s have a closer look at this example. Change into the main directory, which is also where we will be running EvoSuite:

cd Tutorial_Stack

You will find three files in here. First, there is a pom.xml, which is the Maven build file. Second, there is the class under test, which you can find in src/main/java/tutorial/Stack.java:

package tutorial;

import java.util.EmptyStackException;

public class Stack<T> {
    private int capacity = 10;
    private int pointer  = 0;
    private T[] objects = (T[]) new Object[capacity];

    public void push(T o) {
        if(pointer >= capacity)
            throw new RuntimeException("Stack exceeded capacity!");
        objects[pointer++] = o;
    }

    public T pop() {
        if(pointer <= 0)
            throw new EmptyStackException();
        return objects[--pointer];
    }

    public boolean isEmpty() {
        return pointer <= 0;
    }
}

The third file is an existing test suite in src/test/java/tutorial/StackTest.java which we will have a look at later.

To get started, please invoke maven to compile the project:

mvn compile

Maven will produce some output to keep you informed about what it is doing, and at the end you should see a success message, similar to the following:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.577 s
[INFO] Finished at: 2016-04-03T10:38:41+01:00
[INFO] Final Memory: 13M/114M
[INFO] ------------------------------------------------------------------------

If you see a build failure, please have a look at the error messages and fix any errors in your setup.

As a result of the compilation, we now have a file target/classes/tutorial/Stack.class which contains the bytecode of the Stack class.

Obtaining EvoSuite

The latest release of EvoSuite is always available at http://www.evosuite.org/downloads/.

EvoSuite consists of two jar files. At the time of this writing, the current release is 1.0.6:

The executable jar file contains EvoSuite together with all its dependency libraries, so this is all you need to generate tests. Tests generated by EvoSuite use several EvoSuite-specific dependencies and types of instrumentation, mainly to avoid flaky tests. These dependencies need to be available on the classpath when running the generated tests. One option is to simply use the executable jar file as well for that, but if you don’t want all the test generation dependencies on the classpath you can simply use the runtime jar, which is smaller and contains fewer dependencies.

Download evosuite-1.0.6.jar and evosuite-standalone-runtime-1.0.6.jar:

wget https://github.com/EvoSuite/evosuite/releases/download/v1.0.6/evosuite-1.0.6.jar
wget https://github.com/EvoSuite/evosuite/releases/download/v1.0.6/evosuite-standalone-runtime-1.0.6.jar

As the name suggests, the executable jar file can be executed. Try the following

java -jar evosuite-1.0.6.jar

You will see the following output:

* EvoSuite 1.0.6
usage: EvoSuite
...

This output is EvoSuite listing all the possible command-line options, as we haven’t told EvoSuite what to do yet.

To make the rest of this tutorial easier to read, we will create an environment variable to point to EvoSuite.

export EVOSUITE="java -jar $(pwd)/evosuite-1.0.6.jar"

WINDOWS: You can achieve the same thing using: set EVOSUITE=java -jar "%CD%"\evosuite-1.0.6.jar

Now we can simply invoke EvoSuite by typing

$EVOSUITE

WINDOWS: The windows equivalent is: %EVOSUITE%

Generating tests

Now let’s invoke EvoSuite on our example class. For this, we need to tell EvoSuite two things:

  1. What is the class under test?
  2. What is the classpath where it can find the bytecode of the class under test and its dependencies?

The class under test is specified using the -class argument (assuming we are targeting a single class). Note that we need to use the fully qualified class name; that is, we need to include the package name. Thus, in our example, we need to use -class tutorial.Stack.

The classpath is specified using the -projectCP argument. This takes a regular classpath entry, like you would specify when using java -cp or by setting export CLASSPATH=<...>. In our case, we compiled the Stack.java class in place (tutorial/Stack.class), so the classpath is the current directory, which we specify using -projectCP ..

Thus, we can now run EvoSuite as follows:

$EVOSUITE -class tutorial.Stack -projectCP target/classes

If everything worked correctly, then EvoSuite has now produced two files:

evosuite-tests/tutorial/Stack_ESTest.java
evosuite-tests/tutorial/Stack_ESTest_scaffolding.java

Let’s take a closer look at these two files.

If we look into the scaffolding file, we’ll see lots of things happening in methods annotated with @Before and @After. These are JUnit annotations which ensure that these methods are executed before/after execution of each individual test. The reason for all this is that EvoSuite avoids flaky tests by controlling everything that might be non-deterministic. The scaffolding ensures that tests are always executed in the same consistent state, so they should really only fail if they reveal a bug, not because they are flaky. The scaffolding may look a bit scary, but the good news is that you’ll probably never need to look at it.

The tests are in the main Stack_ESTest.java file. The test class inherits from the scaffolding, such that all the setup/pulldown happens without showing all the overhead to ensure tests are not flaky:

@RunWith(EvoRunner.class) @EvoRunnerParameters(mockJVMNonDeterminism = true, useVFS = true, useVNET = true, resetStaticState = true, separateClassLoader = true)
public class Stack_ESTest extends Stack_ESTest_scaffolding {

Besides inheriting from the scaffolding, we also see some annotation that is specific to EvoSuite. The test class declares that it will be executed with the EvoRunner, rather than a default JUnit runner. The test runner takes a couple of parameters that tell it which parts of the execution environment are controlled. You can safely ignore these for now – the values for these parameters are set automatically by EvoSuite.

The rest of the file consists of the actual tests. The tests use JUnit 4 and are annotated with @Test. Because automatically generated tests sometimes do silly things causing infinite loops, all tests have a specified timeout, with a default value of 4 seconds.

Running EvoSuite tests

Let’s compile the tests. The compiler will need several things on the classpath:

  • target/classes: This is the root directory which we need for the tutorial.Stack class
  • evosuite-standalone-runtime-1.0.6.jar: This is the EvoSuite runtime library.
  • evosuite-tests: This is the root directory where we put the test class files
  • junit-4.12.jar and hamcrest-core-1.3.jar: We need JUnit to execute JUnit tests.

In order to get the junit and hamcrest dependencies, we can use Maven again. Type the following command:

mvn dependency:copy-dependencies

This will download the two jar files and put them into the directory target/dependency.

Now we need to tell the Java compiler where to find all these things, for which we set the CLASSPATH environment variable:

export CLASSPATH=target/classes:evosuite-standalone-runtime-1.0.6.jar:evosuite-tests:target/dependency/junit-4.12.jar:target/dependency/hamcrest-core-1.3.jar

WINDOWS – NOTE 1: The path separator on Windows is the semicolon ; while on Unix-like systems it’s the colon :.

WINDOWS – NOTE 2: Although not usually necessary, you may need to use the backslash \ for directory separator on Windows.

WINDOWS – NOTE 3: The Windows equivalent of the command above is: set CLASSPATH=target/classes:evosuite-standalone-runtime-1.0.6.jar;evosuite-tests;target/dependency/junit-4.12.jar;target/dependency/hamcrest-core-1.3.jar

For now, we will simply compile the tests in place. In the second part of the tutorial we will integrate EvoSuite into the Maven project properly, such that Maven takes care of compiling the tests. Type the following command:

javac evosuite-tests/tutorial/*.java

Check that there are the two .class files in evosuite-tests/tutorial. If they are not there, then check what error messages the Java compiler gave you – most likely some part of the classpath is not set correctly.

If they were compiled correctly, we can now run the tests on the commandline:

java org.junit.runner.JUnitCore tutorial.Stack_ESTest

If you followed all the steps so far correctly, you should see the following output:

JUnit version 4.12
.....
Time: 2.021

OK (5 tests)

Congratulations! You just generated and executed your first EvoSuite test suite!

Playing with EvoSuite settings

Now let’s have a closer look at how we can influence what EvoSuite does. First, we had to wait quite a while until test generation completed – even though this is such a simple class. A simple way to tell EvoSuite that we’ve waited long enough for test generation is to simply hit Ctrl+C while it is generating tests. EvoSuite will stop the search, and write the test cases generated up to that point. If you hit Ctrl+C a second time, this will kill EvoSuite completely.

To try this out, generate some more tests:

$EVOSUITE -class tutorial.Stack -projectCP target/classes

After a couple of seconds, when you think coverage is sufficient, hit Ctrl+C and wait for the tests to be written. If you wait 10-20 seconds, you will notice that the tests we got still cover all the lines in the Stack class. So why does EvoSuite take so long? The reason is that EvoSuite by default targets not only lines of code, but attempts to satisfy a range of different testing criteria, including things like mutation testing. Some of the testing goals described by these criteria are infeasible, which means that there exist no tests that satisfy; some other goals are just so difficult to cover that EvoSuite cannot easily produce the tests. This is a well-known aspect of test generation, and to deal with it, EvoSuite uses a fixed amount of time for test generation, and stops generating tests once this time has been used up. By default, this is 60 seconds. If we want to change this, then besides manually stopping EvoSuite, we have two options: Either we change the testing criteria to avoid the stronger criteria that may not be satisfiable, or we set the timeout explicitly.

Let’s start by generating tests for a weaker criterion. We’ll use branch coverage, which requires that all if-conditions evaluate to true and false, and all lines of code are covered. We can set the criterion using the -criterion argument. To generate branch coverage tests, type:

$EVOSUITE -class tutorial.Stack -projectCP target/classes -criterion branch

EvoSuite will work for a couple of seconds, but once it has reached 100% branch coverage it will terminate and give us a branch coverage test suite.

Alternatively, we can tell EvoSuite how much time to spend on test generation. EvoSuite uses search-based techniques, so the time it spends on test generation is called the search budget. Unlike the target criterion, the search budget is not a command line argument, but one of many properties that configure how EvoSuite behaves. To set properties, we can use the -Dproperty=value command line argument. For example, to specify the search budget to 20 seconds, we would use the following command:

$EVOSUITE -class tutorial.Stack -projectCP target/classes -Dsearch_budget=20

EvoSuite has a large number of parameters that can all be set using the -Dproperty=value syntax. To get an overview of the available properties, type the following command:

$EVOSUITE -listParameters

For example, by default EvoSuite will apply minimization to test cases, which means that it removes all statements that are not strictly needed to satisfy the coverage goals; this can be deactivated using -Dminimize=false. EvoSuite also minimizes the assertions it adds, and this can be changed by switching the assertion generation strategy, e.g. to -Dassertion_strategy=all. Thus, to generate long tests with loads of assertions we could use the following command:

$EVOSUITE -class tutorial.Stack -projectCP target/classes -Dsearch_budget=20 -Dminimize=false -Dassertion_strategy=all

Remembering EvoSuite settings

It quickly becomes inconvenient to provide the project classpath to EvoSuite every time when calling it on the same project, in particular when projects become more complex and the number of classpath entries is large. Similarly, one might want to set certain properties for all runs of EvoSuite. We can set these things in a properties file, which EvoSuite by default tries to find in evosuite-files/evosuite.properties. We can generate this file automatically using the -setup command. Let’s tell EvoSuite that our classes are found in the classpath entry src:

$EVOSUITE -setup target/classes

EvoSuite creates a new directory evosuite-files, and puts evosuite.properties in it. In this properties file, all properties are listed with their default values. If you want to change any of the properties, just comment them out and change the value. From now on, EvoSuite will pick up this configuration file. Thus, we don’t need to provide the project classpath and can simply call:

$EVOSUITE -class tutorial.Stack

Working with existing tests

We mentioned earlier that there is a third file in our little Maven project, but so far we ignored it. The third file src/test/java/tutorial/StackTest.java is a test suite containing a single test:

package tutorial;

import org.junit.Test;
import org.junit.Assert;

public class StackTest {

    @Test
    public void test() {
        Stack<Object> stack = new Stack<Object>();
        stack.push(new Object());
        Assert.assertFalse(stack.isEmpty());
    }
}

Not a very exciting test, and also one that EvoSuite can easily generate. However, in practice you might have already written some tests at the point you invoke EvoSuite, and so maybe you don’t want to see some generated tests for code you have already covered.

We can tell EvoSuite to only output tests that ar enot already covered using the junit property. For example, to tell EvoSuite to only give us tests that are not already covered by tutorial.StackTest, we would set the property using -Djunit=tutorial.StackTest. If we have multiple test classes, we can use a colon-separated list for the property.

We also need to tell EvoSuite where to find this test, as it needs to execute the test. So let’s first make sure that the test is compiled and passes. To do this, let’s use Maven. Type the following command:

mvn test

This should give you the following output (among some other messages):

-------------------------------------------------------
 T E S T S
 -------------------------------------------------------
 Running tutorial.StackTest
 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.091 sec

 Results :

 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

If the test doesn’t pass then most likely you have edited (and broken?) the Stack class and should fix it.

For EvoSuite, the interesting part is that Maven placed the bytecode of this test into the directory target/test-classes/tutorial/StackTest.class. If we want to know how great this test suite is, we can ask EvoSuite to measure the coverage for us. EvoSuite supports the command -measureCoverage, and we need to specify the class under test (-class tutorial.Stack), the tests we are interested in (-Djunit=tutorial.StackTest), the classpath containing the class under test and the tests (-projectCP target/classes:target/test-classes), and optionally, which criteria we are interested (e.g., -criterion branch):

$EVOSUITE -measureCoverage -class tutorial.Stack -Djunit=tutorial.StackTest -criterion branch -projectCP target/classes:target/test-classes

This should give you the following output (among other messages):

* Total number of covered goals: 3 / 7
* Total coverage: 43%

If we now only want to have tests that cover the remaining 4 branch coverage goals, we would invoke EvoSuite as follows:

$EVOSUITE -class tutorial.Stack -Djunit=tutorial.StackTest -projectCP target/classes:target/test-classes -criterion branch

WINDOWS: Remember to use the semicolon path separator (;) instead of colon (:) when running this command on Windows.

Take a look at evosuite-tests/tutorial/Stack_ESTest.java to check that it worked.

Running EvoSuite on multiple classes

Our example project only has a single class, so all calls to EvoSuite so far used the argument -class. However, sometimes we might want to target more than just a single class, for example when generating a regression test suite. In this case, we can replace the -class argument with either -prefix or -target.

The -target argument specifies a classpath entry (e.g., directory or jar file). EvoSuite will then be invoked sequentially on every testable class it can find in that classpath entry. If you want to know which classes EvoSuite thinks are testable (e.g., public), then type the following command:

$EVOSUITE -listClasses -target target/classes

The output should be just our example class:

tutorial.Stack

To invoke EvoSuite on all the classes in a classpath entry, type the following:

$EVOSUITE -target target/classes

EvoSuite will now go and test each class it finds, one at a time. Alternatively, we might want to test all classes in a certain package. To test all classes in the tutorial package, type the following command:

$EVOSUITE -prefix tutorial

As our project has only one class this will again just test the Stack.

The arguments -target and -prefix will run EvoSuite sequentially on each class they find. If your project is large, this might not be the ideal strategy. In fact, if you project is large and you want to use EvoSuite repeatedly, you will probably not want to run things manually on the command line. Therefore, part 2 of our tutorial will show you how to integrate EvoSuite properly into a Maven project.