Speaker
Description
Software is important research output. Therefore, funding agencies are interested in the value that a software contributes to the overall results of a funded project. The Helmholtz Association is working towards a system to evaluate data and software publications. The "Task Group Helmholtz Quality Indicators for Data and Software Publications" has already published a vision paper about how such Key Perfomance Indicators (KPIs) could look like (https://doi.org/10.48550/arXiv.2401.08804).
The goal of the Software Curation and Reporting Dashboard (Software CaRD; ZT-I-PF-3-080) is to support the publication and evaluation of software outputs of funded projects. We therefore regard KPIs as additional metadata (or a quality of certain metadata).
From the HERMES project (ZT-I-PF-3-006) comes a solution that can automatically extract and collate metadata from source code repositories. However, the HERMES process comes short when it comes to curation of the collected metadata, which is important as we need validated inputs for evaluation of KPIs. We also took a deeper look at the current state of the software KPI definitions by the Task Group and concluded that most of the criteria could be informed but not directly measured by the hermes software.
This gap should be bridged by Software CaRD by offering different views and processing options on the metadata produced by the HERMES workflow. On the one hand, we aim to provide an interactive interface that allows the browsing of the enriched CodeMeta graph provided by hermes. This will also include traces to the original source for each meta date and if possible hints or tools to fix inconsistencies. By checking the graph against constraints, we also intend to identify and highlight problems and do automated validation where possible.
Using constraints alongside additional hermes plugins that harvest relevant metadata, we also want to implement as many of the Helmholtz KPIs as possible.
The results of both aspectes are then presented in an explorative and interactive Dashboard that does also present the trail that led to a certain evaluation result. The poster explains the overall envisioned architecture and used technologies.
The HMC Conference is a good opportunity to collect further input about the required features and also to collect further ideas about automatic assessment of metrics.
In addition, please add 3 to 5 keywords.
automation kpi curation assessment publication
| Please assign yourself (presenting author) to one of the following groups. | Researchers |
|---|---|
| For whom will your contribution be of most interest? | Data professionals and stewards |
