25 February 2025 to 1 March 2025
Building 30.95
Europe/Berlin timezone

What makes computational communication science (ir)reproducible?

26 Feb 2025, 18:00
2h
Audimax Foyer (Building 30.95)

Audimax Foyer

Building 30.95

Str. am Forum 1, 76131 Karlsruhe
Poster computational reproducibility Poster and Demo Session together with Reception

Speaker

Chung-hong Chan (GESIS)

Description

Computational methods are in full swing in communication science. Part of their promise is to make communication research more reproducible. However, how this plays out in practice has not been systematically studied. We verify the reproducibility of the entire cohort of 30 substantive and methods papers published in the journal Computational Communication Research (CCR), the official journal of the International Communication Association Computational Methods Division with a focus on transparency and hence a high rate of voluntary Open Science participation in the field. Among these CCR papers, we are not able to verify the computational reproducibility of 16 papers as no data and/or code were shared. For the remaining 14 papers, we attempt to execute the code shared by the original authors in a standardized containerized computational environment. We encounter a variety of issues that preclude us from reproducing the original findings, where incomplete sharing of data or code is the most common issue. In the end, we could at least partially reproduce the findings in only 6 papers (20%). Based on our findings, we discuss strategies for researchers to correct for this disheartening state of computational reproducibility. We emphasize the lack of computational reproducibility as a socio-technical challenge and at least for this case the social part was the main culprit. To speak to research software engineers, we warn against the "Technological Solutionism" approach to support computational reproducibility.

I want to participate in the youngRSE prize no

Primary authors

Chung-hong Chan (GESIS) Dr Tim Schatto-Eckrodt (Universität Hamburg) Dr Johannes B. Gruber (Vrije Universiteit Amsterdam)

Presentation materials

doi