The ICPP Reproducibility Initiative
Reproducibility is at the core of solid scientific and technical research. The ability to repeat the research that is produced is a key approach for confirming the validity of a new scientific discovery. Ensuring the robustness and trustworthiness of science that is done using computing and data is critical. The inability to reproduce the results of an experiment can lead to a range of consequences, including the retraction of research and injury to the reputations of the authors and the conference / journal that published it. Furthermore, fostering reproducibility facilitates dissemination, comparison, and adoption of research. Therefore, it can also contribute to increasing the impact and enhancing the visibility of articles.
The International Conference on Parallel Processing (ICPP) recognizes the importance of fostering and enabling reproducible research and it is also committed to raising standards and good practices for reproducibility. For that reason, ICPP is converging with other similar reproducibility initiatives such as the IEEE Transactions on Parallel and Distributed Computing Journal (TPDS) and The International Conference for High Performance (SC’23) reproducibility initiatives, while maintaining the inherent characteristics of the conference.
Fig. 1: Simultaneous review processes of papers (left) and the artifact(s) that enable the reproducibility of the experiments of the papers (right)
In parallel to the submission of a paper, ICPP authors can optionally submit the code and data artifacts that enable to reproduce the experiments of their papers (Fig. 1). Unlike ICPP paper review, which is double-blinded, ICPP is piloting a single-anonymous peer review of artifacts. The reproducibility committee will not share any kind of information with the paper committee that can compromise the double-blinded review process. Eventually, as a result of the reviewing process, reproducibility badges might be granted.
The reproducibility review process consists of two stages:
- The Artifact Description review stage
- The Artifact Evaluation review stage
During the artifact description review stage, reviewers will check that the experiments in the artifact are properly documented, and that they include a complete description of the expected results and an evaluation of them, and most importantly how the expected results from the experiment workflow relate to the results found in the paper.
Once the review stage for papers is complete and the accepted papers are determined, the artifact evaluation stage will commence. Only the accepted papers that submitted their reproducibility artifacts will be considered.
There are four key aspects in reproducibility in practice: