reproducibilityworkshop24

Artifact Checklists from the Community Workshop on Practical Reproducibility in HPC

To cite this version: Keahey, K., Richardson, M., Tolosana Calasanz, R., Hunold, S., Lofstead, J., Malik, T., & Perez, C. (2025). Report on Challenges of Practical Reproducibility for Systems and HPC Computer Science. Community Workshop on Practical Reproducibility in HPC, Atlanta, Georgia. Zenodo. 10.5281/zenodo.15306610

This repository hosts the artifact checklists developed during the Community Workshop on Practical Reproducibility in HPC, held in November 2024. These checklists are designed to guide researchers in packaging and presenting their experiments to facilitate reproducibility in high-performance computing (HPC) and systems computing environments.

The checklists are divided into two main categories:

  1. Experiment Packaging Checklist: This checklist outlines the essential components and requirements for packaging experiments in a way that ensures all necessary resources, software, data, etc. are included for reproducibility along with considerations for longevity and maintenance.
  2. Experiment Style Checklist: This checklist provides guidelines on how to structure and present experiments to enhance usability for reviewers. It includes recommendations on organization, validation, configuration, reproducibility conditions, and time/resource estimation.

Downloading the Checklists

The checklists are available in both online as HTML and PDF formats for easy access and use.

Contributing to the Checklists

The checklists for experiment packaging and style will continue to evolve as reproducibility practices mature in the HPC community. To encourage feedback, we are hosting these checklists on our GitHub repository and invite community contributions through GitHub Pull Requests.

To submit a pull request with suggested modifications:

  1. Fork this repository. (Go to the main page of our GitHub repo and click the “Fork” option at the top right corner.)
  2. Create a new branch for your changes (optional). 
    git checkout -b my-changes
  3. Make your modifications to the checklist files experiment-style.md or experiment-packaging.md. (You can edit files directly in GitHub or clone the repository locally and edit them using your preferred text editor.)
  4. Submit a pull request to this repository with a clear description of your changes (including the rationale and any relevant context).

Review Process: Our committee will review all contributions within 4 weeks of submission, evaluating them for practical utility and broad applicability across HPC domains.

Updates: Approved contributions will be merged into the main branch, with attribution to contributors.

This collaborative approach ensures the checklists remain current and representative of best practices across the diverse HPC reproducibility landscape.

For additional questions or feedback, please contact Marc Richardson at mtrichardson@uchicago.edu.