logo.png RepairThemAll is a framework that allows the execution of automatic program repair tools on benchmarks of bugs by providing an abstraction around the repair tools and the benchmarks. If you use RepairThemAll, please cite our paper:

  author    = {Thomas Durieux and Fernanda Madeiral and Matias Martinez and Rui Abreu},
  title     = {{Empirical Review of Java Program Repair Tools: A Large-Scale Experiment on 2,141 Bugs and 23,551 Repair Attempts}},
  booktitle = {Proceedings of the 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE '19)},
  year      = {2019},
  url       = {}

1. Features

  • Execution of repair tools on benchmarks of bugs
  • Configuration of the environment to execute repair tools properly on the bugs
  • Multi-process execution
  • Grid5000 cluster support

2. Supported repair tools

# Tool Language Repository Commit id
1 Nopol Java 7ba58a78d
2 DynaMoth Java 7ba58a78d
3 NPEFix Java 403445b9a
4 jGenProg Java 26ee3dfc8
5 jKali Java 26ee3dfc8
6 jMutRepair Java 26ee3dfc8
7 Cardumen Java 26ee3dfc8
8 ARJA Java e60b990f9
9 GenProg-A Java e60b990f9
10 RSRepair-A Java e60b990f9
11 Kali-A Java e60b990f9

3. Supported benchmarks of bugs

# Benchmark Language # Projects # Bugs Link
1 Bears Java 72 251
2 Bugs.jar Java 8 1,158
3 Defects4J Java 6 395
4 IntroClassJava Java 6 297
5 QuixBugs Java 40 40
0 Total 132 2,141

4. Repository structure

This repository is structured as follow:

├── benchmarks: contains a git submodule per benchmark plugged-in RepairThemAll
├── data: 
│ ├── benchmarks: contains additional information/files on benchmarks
│ └── repair_tools: contains information on repair tools, e.g. the launcher class name
├── libs: contains the dependencies
├── repair_tools: contains a jar file per repair tool (e.g. npefix.jar) or per repair framework where several repair tools are implemented (e.g. astor.jar)
└── script: contains the actual scripts of RepairThemAll that use everything mentioned above to run repair tools on benchmarks of bugs
  ├── Main files:
  ├── contains the parameters of RepairThemAll (see file)
  ├── contains the script to run a repair tool on a benchmark
  └── contains the script to print the bugs available from a benchmark

5. Usage

RepairThemAll can be executed directly from the source or via a Docker image where RepairThemAll has been pre-configured and is ready to be used. One can find the usage instructions here .

6. Extending RepairThemAll to add support for different repair tools and benchmarks of bugs

To add a new benchmark or a new repair tool, see the instructions here .

7. Reproduce the experiment reported in the paper using RepairThemAll

The results reported in the paper are at program-repair/RepairThemAll_experiment . In order to reproduce those results, the 11 repair tools need to be executed on the 5 benchmarks. Warning: the experiment took 313 days of combined execution time. That execution requires a command line for each pair of repair tool and benchmark (totalling 55 command lines) as following:

python script/ Nopol --benchmark Defects4J
python script/ Nopol --benchmark Bears
Last Updated: 31/05/2022