Call for Presentations

As in previous Chameleon User Meetings, the organizers will reimburse travel expenses of up to $1,500 for the presenting authors of the top 10 selected presentation abstracts (one author per abstract). Please, take a look at the Call For Presentations below for details.

Presentation Proposal Guidelines

Presentation proposals should be in PDF format, no longer than 2 pages, and include the following:

• Project Description: (One paragraph or less)
  A brief but clear description of the Computer Science experiment you are packaging or reproducing: What research problem does it address? (What is the hypothesis? What is the challenge/trade-off? Why is it significant?) How does the experiment capture this research problem?
• Reproducibility Approach: What is the description of the experiment you are either packaging or reproducing? (One paragraph or less)
  What resources does it need and how many? How do those resources need to be configured? What is the experimental environments in which the experiment should run? How are they configured? What does the experiment body consist of? How long and in what manner is the experiment run? What types of data does it produce and how much? How is the data analyzed? What platforms or tools did you use in your experiment?
• Lessons Learned:
  Good and bad: What were the most important obstacles to either packaging for reproducibility or reproducing the experiment you encountered – and how did you overcome them? Were they on the infrastructure level (hardware availability), configuration level (how the hardware is configured, i.e., “creating an MPI cluster”), executing the experimental workflow, managing or analyzing data? What strategies were effective? How did you decide (or what guidelines did you give to reviewers) as to when the experiment is reproduced? What suggestions or wishes do you have on how support for reproducibility should improve? Specifically, what should improve in (1) the ability to set up an experimental environment, (2) the ability to manage the experiment body, and (3) data analysis? Which of those desired capabilities are the most important ones? Any procedural insights on how to organize reproducibility initiatives?
• Impact and Future Directions: (One paragraph or less)
  How has prioritizing reproducibility influenced your HPC research or applications? How can the HPC community better integrate reproducibility into mainstream research and education?

Selection Criteria

Presentations will be selected based on their relevance to HPC-specific reproducibility challenges, the potential to foster discussion, and the insights they offer. We especially encourage submissions with detailed lessons learned, discussion of reproducibility challenges, and experience in reproducibility initiatives.

We particularly encourage submissions that:

• Have an insightful, detailed, and well-communicated "lessons learned" section
• Have a good discussion/analysis of specific reproducibility challenges in HPC
• Share experiences from HPC-focused reproducibility initiatives

Travel Support

For the top 10 selected abstracts, we will reimburse travel expenses of up to $1,500 for the presenting authors (one per abstract). Submitting the presentation proposal doubles as travel support application.

Workshop Outcomes

This workshop aims to produce a report capturing the community's collective knowledge and recommendations for advancing practical reproducibility in HPC. Your presentations and participation will directly contribute to this valuable resource.

If you have any questions, please contact us at contact@chameleoncloud.org or via the Chameleon users list.

Workshop Agenda

Join us for a rooftop happy hour at the beautiful Terminus 330

Keynotes

The workshop will feature keynotes on the state of reproducibility in HPC from our distinguished speakers. Additionally, panel discussions with authors and reviewers who have participated in reproducibility initiatives at HPC and systems conferences, such as SC, will share their experiences in creating, evaluating, and ranking HPC artifacts to support reproducibility.

Torsten Hoefler

Keynote: Reproducing Performance - The Good, the Bad, and the Ugly

Containers and Jupyter notebooks are useful tools for reproducing computational results of any packaged application. However, if the execution performance or efficiency is the science result, matters are more complex. It may not be sufficient to package codes in containers. In fact, containers may disturb the performance results and reproducibility. We outline a set of techniques to facilitate performance reproducibility in various settings. Some performance results may be linked to specific computer architectures or even specific system configurations that may not be accessible to other researchers or even the original team after a software update. We outline techniques to help researchers interpret results on the original system even if it is practically impossible to reproduce the original results. We discuss such techniques both in the context of pure performance but also in the context of the emerging field of data science and artificial intelligence that often allows for a performance-accuracy tradeoff. All-in-all, our work provides a set of guidelines to follow to support reproducible science of performance and benchmarking.

Kate Keahey

Keynote: Adaptable Infrastructures for Reproducible Science - The Chameleon 4 Approach

The landscape of computer science research is evolving at an unprecedented pace, with innovations in AI, data science, edge computing, and beyond. These advancements demand a flexible, powerful infrastructure capable of supporting a wide array of experiments while facilitating reproducibility and collaboration. Chameleon 4, the latest iteration of the NSF-funded testbed, rises to meet these challenges. In this keynote, Dr. Keahey will unveil how Chameleon 4 extends its deeply reconfigurable edge-to-cloud architecture to support emerging research needs. She will describe the testbed's enhanced virtualization capabilities, expanded edge computing functionalities, and advanced mechanisms for sharing digital artifacts. Dr. Keahey will illustrate how these features, combined with Chameleon's existing bare-metal reconfigurability and diverse hardware offerings, create an unparalleled platform for reproducible science. The talk will explore Chameleon 4's approach to federation, enabling seamless integration with other research infrastructures, and discuss how the platform's adaptability ensures it can evolve alongside the ever-changing frontiers of computer science research. Through real-world examples and future roadmaps, attendees will gain insight into how Chameleon 4 is poised to accelerate innovation and foster a more open, collaborative scientific community in the realm of HPC and beyond.

See You in Atlanta!

The workshop will be held at Terminus 330, a state-of-the-art venue in the heart of Atlanta, GA, conveniently located right around the corner from the Georgia World Congress Center where the biggest annual conference in supercomputing is being held November 17-22, 2024.

Lodging

Our workshop is conveniently located in the heart of downtown Atlanta. There are many high-quality hotels and lodgings in the area. If you are also in town for the big HPC conference, we recommend checking out their resources for more options.

Travel

Getting to Atlanta

Atlanta is easily accessible by air and ground transportation:

• By Air: Hartsfield-Jackson Atlanta International Airport (ATL) is the primary airport serving Atlanta.
• By Car: Atlanta is intersected by several major interstate highways, making it easily accessible by car from many parts of the United States.
• By Train: Amtrak's Crescent line serves Atlanta, connecting it to major cities in the Northeast and New Orleans.

Getting to the Venue

Terminus 330 is located in downtown Atlanta. Here are some options for getting to the venue:

• From the Airport: Take the MARTA train from the airport to downtown Atlanta. The venue is a short walk or ride-share trip from several MARTA stations.
• By Public Transit: Atlanta's MARTA system provides bus and rail service throughout the city. Check the MARTA website for routes and schedules.
• By Car: If driving, there are several parking options near the venue. We recommend checking online parking apps for the best rates.
• Ride-sharing: Services like Uber and Lyft are widely available in Atlanta and can be a convenient option for getting around the city.

For more information on getting around Atlanta, visit the official Atlanta tourism website.

Contact Us

For any questions or feedback, send us an email at contact@chameleoncloud.org or reach out to one of our event coordinators:

Name: Marc Richardson
Email: mtrichardson@uchicago.edu

Name: Roberto Vale
Email: rvale@uchicago.edu

Planning Committee

Meet the dedicated team assisting the Community Workshop on Practical Reproducibility in HPC.

Brian Kocoloski

USC Information Sciences Institute

Fraida Fund

New York University

Sascha Hunold

Vienna University of Technology

Tanu Malik

DePaul University

Rafa Tolosana Calasanz

University of Zaragoza

Our Projects

About the REPETO Project

The REPETO (pronounce to rhyme with Geppetto) project is an NSF-funded research coordination network (RCN) that promotes the concept of practical reproducibility. This practice aims to package experiments in a way that allows cost-effective repetition, making them as accessible for exploring research as reading papers is today.

Key aspects of the REPETO project include:

• Focus on experiments in the computer science community
• Understanding the cost/benefit equation of reproducibility for these experiments
• Identifying factors that make reproducibility difficult or infeasible
• Fostering community practices to integrate reproducibility into mainstream research and education activities in computer science

About the Chameleon Testbed

Chameleon is an NSF-funded, large-scale, deeply programmable experimental platform for Computer Science systems research. It provides a configurable environment that can support a wide range of experimental needs, from bare metal reconfiguration to support for reproducible software environments. Chameleon allows researchers to explore transformative concepts in cloud computing, distributed computing, networking, and machine learning, enabling them to experiment with novel cloud architectures and pursue new applications of cloud computing. With its unique features and commitment to openness, Chameleon plays a crucial role in advancing computer science research and education.