Community Expectations for Research Artifacts and Evaluation Processes
| dc.contributor.author | Hermann, Ben | |
| dc.contributor.author | Winter, Stefan | |
| dc.contributor.author | Siegmund, Janet | |
| dc.contributor.editor | Engels, Gregor | |
| dc.contributor.editor | Hebig, Regina | |
| dc.contributor.editor | Tichy, Matthias | |
| dc.date.accessioned | 2023-01-18T13:38:41Z | |
| dc.date.available | 2023-01-18T13:38:41Z | |
| dc.date.issued | 2023 | |
| dc.description.abstract | Artifact evaluation has been introduced into the software engineering and programming languages research community with a pilot at ESEC/FSE 2011 and has since then enjoyed a healthy adoption throughout the conference landscape. We conducted a survey including all members of artifact evaluation committees of major conferences in the software engineering and programming language field from 2011 to 2019 and compared the answers to expectations set by calls for artifacts and reviewing guidelines. While we find that some expectations exceed the ones expressed in calls and reviewing guidelines, there is no consensus on a quality threshold for artifacts in general. We observe very specific quality expectations for specific artifact types for review and later usage, but also a lack of their communication in calls. We also find problematic inconsistencies in the terminology used to express artifact evaluation’s most important purpose. We derive several actionable suggestions which can help to mature artifact evaluation in the inspected community and also to aid its introduction into other communities in computer science. | en |
| dc.identifier.isbn | 978-3-88579-726-5 | |
| dc.identifier.pissn | 1617-5468 | |
| dc.identifier.uri | https://dl.gi.de/handle/20.500.12116/40084 | |
| dc.language.iso | en | |
| dc.publisher | Gesellschaft für Informatik e.V. | |
| dc.relation.ispartof | Software Engineering 2023 | |
| dc.relation.ispartofseries | Lecture Notes in Informatics (LNI) - Proceedings, Volume P-332 | |
| dc.subject | Research Artifacts | |
| dc.subject | Artifact Evaluation | |
| dc.subject | Replicability | |
| dc.subject | Reproducibility | |
| dc.subject | Study | |
| dc.title | Community Expectations for Research Artifacts and Evaluation Processes | en |
| dc.type | Text/Conference Paper | |
| gi.citation.endPage | 66 | |
| gi.citation.publisherPlace | Bonn | |
| gi.citation.startPage | 65 | |
| gi.conference.date | 20.–24. Februar 2023 | |
| gi.conference.location | Paderborn | |
| gi.conference.sessiontitle | Wissenschaftliches Hauptprogramm |
Dateien
Originalbündel
1 - 1 von 1
Vorschaubild nicht verfügbar
- Name:
- paper19.pdf
- Größe:
- 157.4 KB
- Format:
- Adobe Portable Document Format