SF100 Benchmark

About SF100

Several promising techniques have been proposed to automate different tasks in software testing, such as test data generation for object-oriented software. However, reported studies in the literature only show the feasibility of the proposed techniques, because the choice of the employed artifacts in the case studies (e.g., software applications) is usually done in a non-systematic way. The chosen case study might be biased, and so it might not be a valid representative of the addressed type of software (e.g., internet applications and embedded systems). The common trend seems to be to accept this fact and get over it by simply discussing it in a threats to validity section. To achieve sound empirical results, we randomly selected 100 Java projects from SourceForge, which is the most popular open source repository (more than 300,000 projects with more than two million registered users). The resulting benchmark not only is very large, but more importantly it is statistically sound and representative for open source projects.

For more details about SF100 please consider the following paper, in which it was introduced:

  • [PDF] G. Fraser and A. Arcuri, “Sound Empirical Evidence in Software Testing,” in 34th International Conference on Software Engineering, ICSE 2012, June 2-9, 2012, Zurich, Switzerland, 2012, pp. 178-188.
    [Bibtex]
    @inproceedings{ICSE12,
        author    = {Gordon Fraser and Andrea Arcuri},
        TITLE="Sound Empirical Evidence in Software Testing",
        YEAR={2012},
        pages     = {178--188},
        ee        = {http://dx.doi.org/10.1109/ICSE.2012.6227195},
        booktitle     = {34th International Conference on Software Engineering, ICSE 2012, June 2-9, 2012, Zurich, Switzerland},
        publisher = {IEEE},
      isbn      = {978-1-4673-1067-3},
    }

Update

2012/03/16: We have resolved some dependency issues and updated the SF100 archives.

Download

All projects selected for the case study are open source projects. To check the individual licenses please check the projects’ websites (see table below). A download containing bytecode of all 100 projects together with their dependencies (check webpages for licenses) is available here:

Project details

Project Name Short Name Website Source
Tullibee tullibee Website Source repository
A4J (Amazon AWS 4 Java) a4j Website Source repository
Jigen – Java Installer Generator jigen Website Source repository
Remote Invocation Framework (RIF) rif Website Source repository
templateIt templateit Website Source repository
jNFe – Java NFe jnfe Website Source repository
sfm sfmis Website Source repository
gfarceGestionFac gfarcegestionfa Website Source repository
FalseLight falselight Website Source repository
DAWN water-simulator Website Source repository
imSMART Migration imsmart Website Source repository
DSA Chat Program dsachat Website Source repository
jdbacl jdbacl Website Source repository
JState omjstate Website Source repository
BeanBin beanbin Website Source repository
JoomlaTemplateDetails templatedetails Website Source repository
Inspirento inspirento Website Source repository
JSecurity jsecurity Website Source repository
JMCA – Java Method Cohesion Analyzer jmca Website Source repository
NekoMUD nekomud Website Source repository
Geo-Google – Free Address Standardizer geo-google Website Source repository
byuic byuic Website Source repository
Java Wiki Bot Framework jwbf Website Source repository
saxpath saxpath Website Source repository
JNI-InChI jni-inchi Website Source repository
JIPA jipa Website Source repository
Framework for Game Related Research gangup Website Source repository
greencow greencow Website Source repository
APBSmem apbsmem Website Source repository
BluePenguinMail bpmail Website Source repository
Xisemele xisemele Website Source repository
HTTP Analyzer httpanalyzer Website Source repository
JVC – Java View Controller javaviewcontrol Website Source repository
SBMLReader 2 Plugin for Cytoscape sbmlreader2 Website Source repository
Corina corina Website Source repository
SchemaSpy schemaspy Website Source repository
PetSoar – Open Source J2EE PetStore petsoar Website Source repository
javaBB javabullboard Website Source repository
Diffi diffi Website Source repository
cards24 cards24 Website Source repository
Follow follow Website Source repository
asphodel asphodel Website Source repository
lilith lilith Website Source repository
summa summa Website Source repository
Lotus lotus Website Source repository
Nutzenportfolioanalyse nutzenportfolio Website Source repository
DVD Home Video Project dvd-homevideo Website Source repository
resources4j resources4j Website Source repository
Die Bierse diebierse Website Source repository
Pascal interpreter written in Java biff Website Source repository
Java Interactive Profiler jiprof Website Source repository
Lagoon lagoon Website Source repository
Shp2KML shp2kml Website Source repository
DB-Everywhere db-everywhere Website Source repository
Lavalamp lavalamp Website Source repository
jHandballMoves jhandballmoves Website Source repository
hft-bomberman hft-bomberman Website Source repository
FPS370 – First Person Shooter Template fps370 Website Source repository
MyGrid – Distributed Grid mygrid Website Source repository
Sugar : making Java sweeter sugar Website Source repository
framework for dynamic analysis and test noen Website Source repository
dom4j: flexible XML framework for Java dom4j Website Source repository
ObjectExplorer4J objectexplorer Website Source repository
JTailPlus jtailgui Website Source repository
GoToServers Simple FTP Client gsftp Website Source repository
openjms openjms Website Source repository
Google App Engine Application Manager gae-app-manager Website Source repository
Bible Study Tool biblestudytool Website Source repository
macaw lhamacaw Website Source repository
EchoDep Hub and Spoke echodep Website Source repository
Ext4J ext4j Website Source repository
Battlecry battlecry Website Source repository
Free Instant Messenger Server/Client 1.0 fim1 Website Source repository
FixSuite fixsuite Website Source repository
OpenHRE openhre Website Source repository
dash-framework dash-framework Website Source repository
Io Project io-project Website Source repository
CalorieCount caloriecount Website Source repository
The West Fortbattle Player twfbplayer Website Source repository
Wheel wheelwebtool Website Source repository
JavAthena javathena Website Source repository
Internet Protocol Address Calculator ipcalculator Website Source repository
xBus xbus Website Source repository
IFX Framework ifx-framework Website Source repository
SHOP shop Website Source repository
Advanced T-Robots Version 2J at-robots2-j Website Source repository
JAW-BR jaw-br Website Source repository
JOpenChart Toolkit jopenchart Website Source repository
JIGL (Java Image and Graphics Library) jiggler Website Source repository
dcParseArgs dcparseargs Website Source repository
Class Viewer classviewer Website Source repository
JCVI Java Common jcvi-javacommon Website Source repository
QuickServer – TCP server framework quickserver Website Source repository
JCLO: A Java Command Line Option package jclo Website Source repository
CellWars2009 celwars2009 Website Source repository
HEAL: Health Education Assets Library heal Website Source repository
Feudalism Game feudalismgame Website Source repository
Transmitter Locator trans-locator Website Source repository
Newzgrabber newzgrabber Website Source repository
JGAAP jgaap Website Source repository