Does Automated Test Generation Really Help Software Testers?
Gordon Fraser, Matt Staats, Phil McMinn, Andrea Arcuri, Frank Padberg
Automated test generation techniques can efficiently produce test data that systematically cover structural aspects of a program. In the absence of a specification, a common assumption is that these tests relieve a developer of most of the work, as the act of testing is reduced to checking the results of the tests. Although this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the fact that the approach has only seen a limited uptake in industry suggests the contrary, and calls into question its practical usefulness. To investigate this issue, we performed a controlled experiment comparing a total of 49 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, EvoSuite. We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.
-
![[PDF]](http://www.evosuite.org/wp-content/plugins/papercite/img/pdf.png)
G. Fraser, M. Staats, P. McMinn, A. Arcuri, and F. Padberg, “Does Automated White-Box Test Generation Really Help Software Testers?,” in Proceedings of the 2013 International Symposium on Software Testing and Analysis, New York, NY, USA, 2013.
[Bibtex]@inproceedings{ISSTA13_Study, author = {Gordon Fraser and Matt Staats and Phil McMinn and Andrea Arcuri and Frank Padberg}, title = {Does Automated White-Box Test Generation Really Help Software Testers?}, booktitle = {Proceedings of the 2013 International Symposium on Software Testing and Analysis}, series = {ISSTA '13}, year = {2013}, publisher = {ACM}, address = {New York, NY, USA}, }
Slides from the ISSTA 2013 presentation can be viewed here.
This page provides the experimental material used in this experiment.
Case study objects are provided as Eclipse workspaces. Each target class contains five seeded faults.
Option (Apache Commons CLI)
- EvoSuite Eclipse Workspace [tar.gz]
- Manual Eclipse Workspace [tar.gz]
- Individual faults [tar.gz]
- Note: As Commons CLI is a dependency of EvoSuite, we use a changed package name in these archives
Rational (Math4J)
DocType (XOM)
Tutorial Example
Environment
- We used a customized Eclipse package containing only bare-bones Eclipse with JDE [ Linux x86 | MacOS X ]
- Alternatively, see Eclipse Indigo [ www ]
- Sun JDK 6 [www]
- EvoSuite Eclipse+Study plugin [ jar ]