About SF110

Research on software testing produces many innovative automated techniques, but because software testing is by necessity incomplete and approximate, any new technique faces the challenge of an empirical assessment. Scientific advance is typically demonstrated using a set of examples that represent a particular problem addressed by the technique. However, demonstrating scientific advance is not necessarily the same as demonstrating practical value: A technique that works well on small, artificial problems might not scale up to the complexity of real systems. Ideally, one would use large “real-world” case studies to minimize the threats to external validity when evaluating research tools. However, neither choosing such case studies nor applying research prototypes to them are trivial tasks. To address this problem, we originally introduced the SF110 corpus of classes:

• G. Fraser and A. Arcuri, “A Large Scale Evaluation of Automated Unit Test Generation Using EvoSuite,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 24, iss. 2, p. 8, 2014.
  [Bibtex]

  @article{TOSEM_evaluation,
  author = {Gordon Fraser and Andrea Arcuri},
  title = {A Large Scale Evaluation of Automated Unit Test Generation Using EvoSuite},
  journal ={ACM Transactions on Software Engineering and Methodology (TOSEM)},
  year={2014},
  volume={24},
    number={2},
    pages={8},
    year={2014},
    publisher={ACM}
  }

The SF100 corpus of classes is a statistically representative sample of 100 Java projects from SourceForge, which is a popular open source repository (more than 300,000 projects with more than two million registered users). Because SourceForge is home to many old and stale projects, we have extended SF100 with the 10 most popular projects, resulting in a revised corpus of classes, SF110.

We now also make a source version available, where we massaged all projects into a common build infrastructure. Unit tests are included for those projects that come with tests, and for each project we provide a test suite produced with EvoSuite.

Download

All projects selected for the case study are open source projects. To check the individual licenses please check the projects’ websites (see table below). To download SF110, choose one of the following options:

• Binaries only, including dependencies and EvoSuite setup (943MB)
• Source code with dependencies and tests (1014MB)

Project details

Project Name	Short Name	Website	Source
Tullibee	tullibee
A4J (Amazon AWS 4 Java)	a4j
Genetic Algorithms in Java	gaj
Remote Invocation Framework (RIF)	rif
templateIt	templateit
jNFe – Java NFe	jnfe
sfm	sfmis
gfarceGestionFac	gfarcegestionfa
FalseLight	falselight
DAWN	water-simulator
imSMART Migration	imsmart
DSA Chat Program	dsachat
jdbacl	jdbacl
JState	omjstate
BeanBin	beanbin
JoomlaTemplateDetails	templatedetails
Inspirento	inspirento
JSecurity	jsecurity
JMCA – Java Method Cohesion Analyzer	jmca
NekoMUD	nekomud
Geo-Google – Free Address Standardizer	geo-google
byuic	byuic
Java Wiki Bot Framework	jwbf
saxpath	saxpath
JNI-InChI	jni-inchi
JIPA	jipa
Framework for Game Related Research	gangup
greencow	greencow
APBSmem	apbsmem
BluePenguinMail	bpmail
Xisemele	xisemele
HTTP Analyzer	httpanalyzer
JVC – Java View Controller	javaviewcontrol
SBMLReader 2 Plugin for Cytoscape	sbmlreader2
Corina	corina
SchemaSpy	schemaspy
PetSoar – Open Source J2EE PetStore	petsoar
javaBB	javabullboard
Diffi	diffi
Group Layout Engineer	glengineer
Follow	follow
asphodel	asphodel
lilith	lilith
summa	summa
Lotus	lotus
Nutzenportfolioanalyse	nutzenportfolio
DVD Home Video Project	dvd-homevideo
resources4j	resources4j
Die Bierse	diebierse
Pascal interpreter written in Java	biff
Java Interactive Profiler	jiprof
Lagoon	lagoon
Shp2KML	shp2kml
DB-Everywhere	db-everywhere
Lavalamp	lavalamp
jHandballMoves	jhandballmoves
hft-bomberman	hft-bomberman
FPS370 – First Person Shooter Template	fps370
MyGrid – Distributed Grid	mygrid
Sugar : making Java sweeter	sugar
framework for dynamic analysis and test	noen
dom4j: flexible XML framework for Java	dom4j
ObjectExplorer4J	objectexplorer
JTailPlus	jtailgui
GoToServers Simple FTP Client	gsftp
openjms	openjms
Google App Engine Application Manager	gae-app-manager
Bible Study Tool	biblestudytool
macaw	lhamacaw
EchoDep Hub and Spoke	echodep
Ext4J	ext4j
Battlecry	battlecry
Free Instant Messenger Server/Client 1.0	fim1
FixSuite	fixsuite
OpenHRE	openhre
dash-framework	dash-framework
Io Project	io-project
CalorieCount	caloriecount
The West Fortbattle Player	twfbplayer
Wheel	wheelwebtool
JavAthena	javathena
Internet Protocol Address Calculator	ipcalculator
xBus	xbus
IFX Framework	ifx-framework
SHOP	shop
Advanced T-Robots Version 2J	at-robots2-j
JAW-BR	jaw-br
JOpenChart Toolkit	jopenchart
JIGL (Java Image and Graphics Library)	jiggler
dcParseArgs	dcparseargs
Class Viewer	classviewer
JCVI Java Common	jcvi-javacommon
QuickServer – TCP server framework	quickserver
JCLO: A Java Command Line Option package	jclo
CellWars2009	celwars2009
HEAL: Health Education Assets Library	heal
Feudalism Game	feudalismgame
Transmitter Locator	trans-locator
Newzgrabber	newzgrabber
JGAAP	jgaap
SAP Netweaver Server Adapter for Eclipse	sapnweclipse
SQuirreL SQL Client	squirrel-sql
Sweet Home 3D	sweethome3d
Vuze	azureus
FreeMind	freemind
Eclipse Checkstyle Plugin	eclipse-cs
Weka Machine Learning Software	Weka
Liferay Portal	lportal
PDF Split and Merge	pdfsam
Firebird	firebird