Artifact Metadata and Documentation Structure

Artifact Descriptions for ICPP must follow the following structure and they cannot exceed 2-page long, US letter, ACM format. The Artifact Description (AD) Appendix will be auto generated from author responses to a standard form embedded in the online submission system. It will contain the following aspects:

1. Artifact Identification: Including an abstract describing the main contributions of the article and how the role of the artifact in these contributions. The abstract may include a software architecture or data models and its description to help the reader understand the artifact and a clear description on to what extent the artifact contributes to the reproducibility of the experiments in the article.
2. Artifact Dependencies and Requirements: Including (i) a description of the hardware resources required, (ii) a description of the operating systems required, (iii) the software libraries needed, (iv) the input dataset needed to execute the code or when the input data is generated, and (v) optionally, any other dependencies or requirements. Best practices to facilitate the understanding of the descriptions indicate that unnecessary dependencies and requirements should be suppressed from the artifact.
3. Artifact Installation and Deployment Process: Including (i) the process description to install and compile the libraries and the code, and (ii) the process description to deploy the code in the resources. The description of these processes should include an estimation of the installation, compilation, and deployment times. When any of these times exceed what is reasonable, authors should provide some way to alleviate the effort required by the potential recipients of the artifacts. For instance, capsules with the compiled code can be provided, or a simplified input dataset that reduces the overall experimental execution time. On the other hand, best practices indicate that, whenever it is possible, the actual code of software dependencies (libraries) should not be included in the artifact, but scripts should be provided to download them from a repository and perform the installation.
4. Reproducibility of Experiments: Including (i) a complete description of the experiment workflow that the code can execute, (ii) an estimation of the execution time to execute the experiment workflow, (iii) a complete description of the expected results and an evaluation of them, and most importantly (iv) how the expected results from the experiment workflow relate to the results found in the article. Best practices indicate that, to facilitate the understanding of the scope of the reproducibility, the expected results from the artifact should be in the same format as the ones in the article. For instance, when the results in the article are depicted in a graph figure, ideally, the execution of the code should provide a (similar) figure (there are open-source tools that can be used for that purpose such as gnuplot). It is critical that authors devote their efforts to these aspects of the reproducibility of experiments to minimize the time needed for their understanding and verification.
5. Other notes: Optionally, it might include other related aspects that could be important and were not addressed in the previous points.

Reproducibility Artifact Examples

Some journal articles with reproducibility badges are listed here (https://www.computer.org/csdl/journal/td/misc/268409?title=Articles%20with%20Reproducibility%20Badges&periodical=IEEE%20Transactions%20on%20Parallel%20and%20Distributed%20Systems) and can be explored as guidance.