Tests generated by EvoSuite can be run like any other JUnit test, i.e., directly from an IDE, or as part of a build process (e.g., mvn test).
EvoSuite tests require a runtime dependency
The generated tests rely on the EvoSuite framework, and so the evosuite-standalone-runtime-<version>.jar has to be on the classpath. This is, for example, needed because EvoSuite isolates tests from environmental dependencies like the file system and the network. For this, the class under test has to be instrumented, which is done automatically by EvoSuite, and you do not need to worry (too much) about them.
In a Maven project, for example, one would add the following dependency in scope test:
<dependency>
<groupId>org.evosuite</groupId>
<artifactId>evosuite-standalone-runtime</artifactId>
<version>1.0.6</version>
<scope>test</scope>
</dependency>
Unfortunately, the bytecode instrumentation might conflict with the bytecode instrumentations done by tools used to measure code coverage (e.g., JaCoCo, EclEmma, Cobertura, Clover and JMockit). If this happens, then you would see 0% code coverage, even for reasonable tests.
How EvoSuite instruments bytecode
There are two options:
1) In the @RunWith(EvoRunner.class), EvoSuite uses its own classloader that applies the bytecode instrumentation. This unfortunately has the side-effect of throwing away any previously done bytecode instrumentation done by a coverage measure tool (unless this was done offline and/or with a Java Agent).
2) EvoSuite activates a Java Agent that intercepts any class loading, and add its own instrumentation. This avoids losing previous instrumentations (i.e., code coverage tools), but has the side effect that it will not work if the class under test was loaded before the agent is activated in the tests (and this depends on how the tests are started/run). Even worse, if for any reason the EvoSuite instrumentation is not applied, tests relying on environment mocks will most likely fail because of the lack of instrumentation.
Using (2) would be the best option, as long as we can guarantee that the Java classes get properly instrumented. But there are many ways to run tests (Maven, Ant, IDEs, etc) and, therefore, for the the moment as default we use (1). Once we can cover most of the cases for (2), we will rather use (2) and not (1).
In general, if you get 0% coverage, you can try to change the setting from (1) to (2), eg in :
@RunWith(EvoRunner.class) @EvoRunnerParameters(mockJVMNonDeterminism = true, useVFS = true, useVNET = true, resetStaticState = true, separateClassLoader = true, useJEE = true)
you can try to change separateClassLoader = true into separateClassLoader = false and see what happens. If that works, then you can generate tests directly with (2) using the command line option -Duse_separate_classloader=false. If you generate tests from Maven plugin, you can try to configure it with:
<configuration>
<extraArgs> -Duse_separate_classloader=false </extraArgs>
</configuration>
Coverage tool support summary
- Jacoco with runtime instrumentation: Does not work with (1), requires (2).
- Jacoco with offline instrumentation: Works with (1) and (2).
- Cobertura: Works with (1) and (2).
- Clover: Works with (1) and (2).
- PIT: Does not work with (1), requires (2).
We have created example projects for each of these tools set up to execute EvoSuite tests. In the following, we summarise the main points for each tool.
Setting up Jacoco
Jacoco works out of the box when using EvoSuite’s JavaAgent. The following example shows a Maven/Surefire configuration of Jacoco usable with EvoSuite:
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.0</version>
<executions>
<execution>
<id>pre-unit-test</id>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</destFile>
<propertyName>surefireArgLine</propertyName>
</configuration>
</execution>
<execution>
<id>post-unit-test</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<dataFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</dataFile>
<outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.15</version>
<configuration>
<argLine>${surefireArgLine}</argLine>
</configuration>
</plugin>
</plugins>
</build>
Setting up Jacoco with offline instrumentation
Jacoco with offline instrumentation works regardless of the classloader setting in the generated tests. It is important, however, to explicitly set the location of the jacoco.exec file (due to EvoSuite’s handling of environmental dependencies).
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.0</version>
<configuration>
<dataFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</dataFile>
<destFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</destFile>
<outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
</configuration>
<executions>
<execution>
<id>default-instrument</id>
<goals>
<goal>instrument</goal>
</goals>
</execution>
<execution>
<id>default-restore-instrumented-classes</id>
<goals>
<goal>restore-instrumented-classes</goal>
</goals>
</execution>
<execution>
<id>default-report</id>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.15</version>
<configuration>
<systemPropertyVariables>
<jacoco-agent.destfile>${project.build.directory}/coverage-reports/jacoco-ut.exec</jacoco-agent.destfile>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
</build>
To create a coverage report, in this configuration Maven is invoked as follows:
mvn clean test jacoco:restore-instrumented-classes jacoco:report
Setting up Cobertura
Cobertura should work regardless of EvoSuite’s classloader settings. To use it in a Maven project, you simply need to activate the plugin:
<reporting>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<version>2.7</version>
</plugin>
</plugins>
</reporting>
Setting up Clover
Clover should work regardless of EvoSuite’s classloader settings. However, it is necessary to avoid that Clover instruments EvoSuite’s tests by configuring the Clover plugin to exclude anything generated by EvoSuite:
<build>
<plugins>
<plugin>
<groupId>com.atlassian.maven.plugins</groupId>
<artifactId>clover-maven-plugin</artifactId>
<version>4.1.2</version>
<configuration>
<excludes>
<exclude>**/*_ESTest.java</exclude>
<exclude>**/*_ESTest_scaffolding.java</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
Setting up PIT
PIT works out of the box when using EvoSuite’s JavaAgent.
<dependency>
<groupId>org.pitest</groupId>
<artifactId>pitest-parent</artifactId>
<version>1.3.2</version>
<type>pom</type>
</dependency>
PIT would then be invoked, for example, as follows:
mvn org.pitest:pitest-maven:mutationCoverage
More information/examples
Check out the example projects on GitHub for full working examples of each of these tools.