Speakers
Description
Much software engineering (SE) research results in research software artifacts (RSAs), which can be contributions themselves or means to obtain other contributions. However, as research evolves and researchers move on, RSAs tend to be abandoned and become unusable due to a lack of maintenance. As a consequence, other researchers require a lot of effort to re-implement the RSA based on the descriptions in the corresponding publication. Attempts to enhance transparency and artifact availability distilled best practices for sharing artifacts, such as artifact evaluations, badges, and persistent repositories. Still, these attempts are not perfect and RSAs suffer from problems like a lack of documentation, unclear system requirements, or outdated dependencies.
We want to conduct a hackathon as an enjoyable experience for interested participants to see whether, which, and how many artifacts they can reuse from previous publications. Specifically, we will collect a sample of publications (approx. 30) from recent major SE venues, such as ASE, ICSE, and FSE, in which the authors shared RSAs. We will select these so that technical requirements and re-use efforts allow an in-principle reuse within three hours on personal computers. Moreover, we will select RSAs with different badges (e.g., validated vs. non-validated) and provide RSAs targeting different SE domains (e.g., testing, static analysis, refactoring, repair) to motivate many participants. The hackathon is accompanied by an anonymous online survey comprising a minimal set of demographic and background questions as well as the RSA's ReadMe together with a reflection section on its reuse. Single participants or groups (depending on the number of participants) are asked to reproduce the steps in the ReadMe file and replicate the corresponding results. Within the survey, they should document their process and problems by checking whether they can reproduce the single steps in the ReadMe, shortly describing additional steps to achieve reproduction, or issues hindering them. Moreover, participants can update an RSA themselves to make it work (optional), which we track by using version control systems (e.g., $\texttt{git}$). This way, we are collecting steps and best practices for making artifacts (re-)usable as well as identifying RSAs' properties that facilitate or challenge their reuse. We will summarize and synthesize the findings from the surveys and discuss the results with the participants in a separate session. If enough participants are interested in the hackathon, we may be able to publish the findings to guide future RSA sharing. Independently of this case, we will share the synthesized results with the participants; at least during the discussion session and via a report (ideally a published paper) distributed afterwards. If we obtain improved documentations or implementations, we plan to share these with the original RSA authors to consider for updating their repositories. Overall, we hope that this design makes the hackathon an enjoyable experience, with the potential to contribute to new insights as well as improvements to existing RSAs. Since this hackathon is open for practitioners, we hope to get insights on closing the gap between research artifacts and their practical usage.
| Slot length | other(help with comment) |
|---|
