Inter-Rater Agreement and Usability: A Comparative Evaluation of Annotation Tools for Sentiment Annotation
Abstract
We present the results of a comparative evaluation study of five annotation tools with 50 participants in the context of sentiment and emotion annotation of literary texts. Ten participants per tool annotated 50 speeches of the play Emilia Galotti by G. E. Lessing. We evaluate the tools via standard usability and user experience questionnaires, by measuring the time needed for the annotation, and via semi-structured interviews. Based on the results we formulate a recommendation. In addition, we discuss and compare the usability metrics and methods to develop best practices for tool selection in similar contexts. Furthermore, we also highlight the relationship between inter-rater agreement and usability metrics as well as the effect of the chosen tool on annotation behavior.
- Citation
- BibTeX
Schmidt, T., Winterl, B., Maul, M., Schark, A., Vlad, A. & Wolff, C.,
(2019).
Inter-Rater Agreement and Usability: A Comparative Evaluation of Annotation Tools for Sentiment Annotation.
In:
Draude, C., Lange, M. & Sick, B.
(Hrsg.),
INFORMATIK 2019: 50 Jahre Gesellschaft für Informatik – Informatik für Gesellschaft (Workshop-Beiträge).
Bonn:
Gesellschaft für Informatik e.V..
(S. 121-133).
DOI: 10.18420/inf2019_ws12
@inproceedings{mci/Schmidt2019,
author = {Schmidt, Thomas AND Winterl, Brigitte AND Maul, Milena AND Schark, Alina AND Vlad, Andrea AND Wolff, Christian},
title = {Inter-Rater Agreement and Usability: A Comparative Evaluation of Annotation Tools for Sentiment Annotation},
booktitle = {INFORMATIK 2019: 50 Jahre Gesellschaft für Informatik – Informatik für Gesellschaft (Workshop-Beiträge)},
year = {2019},
editor = {Draude, Claude AND Lange, Martin AND Sick, Bernhard} ,
pages = { 121-133 } ,
doi = { 10.18420/inf2019_ws12 },
publisher = {Gesellschaft für Informatik e.V.},
address = {Bonn}
}
author = {Schmidt, Thomas AND Winterl, Brigitte AND Maul, Milena AND Schark, Alina AND Vlad, Andrea AND Wolff, Christian},
title = {Inter-Rater Agreement and Usability: A Comparative Evaluation of Annotation Tools for Sentiment Annotation},
booktitle = {INFORMATIK 2019: 50 Jahre Gesellschaft für Informatik – Informatik für Gesellschaft (Workshop-Beiträge)},
year = {2019},
editor = {Draude, Claude AND Lange, Martin AND Sick, Bernhard} ,
pages = { 121-133 } ,
doi = { 10.18420/inf2019_ws12 },
publisher = {Gesellschaft für Informatik e.V.},
address = {Bonn}
}
Dateien | Groesse | Format | Anzeige | |
---|---|---|---|---|
paper03_05.pdf | 227.6Kb | View/ |
Sollte hier kein Volltext (PDF) verlinkt sein, dann kann es sein, dass dieser aus verschiedenen Gruenden (z.B. Lizenzen oder Copyright) nur in einer anderen Digital Library verfuegbar ist. Versuchen Sie in diesem Fall einen Zugriff ueber die verlinkte DOI: 10.18420/inf2019_ws12
Haben Sie fehlerhafte Angaben entdeckt? Sagen Sie uns Bescheid: Send Feedback
More Info
ISBN: 978-3-88579-689-3
ISSN: 1617-5468
xmlui.MetaDataDisplay.field.date: 2019
Language:
(en)

Content Type: Text/Conference Paper
Keywords
Collections
Related items
Showing items related by title, author, creator and subject.
-
Annotator-Centered Design: Towards a Tool for Sentiment and Emotion Annotation
Schmidt, Thomas; Jakob, Marco; Wolff, Christian
77-85 -
3D User Interfaces for Interactive Annotation of Vascular Structures
Saalfeld, Patrick; Glaßer, Sylvia; Preim, Bernhard
255-258 -
Generating Interactive Prototypes from Query Annotated Discourse Models
Kis, Filip; Bogdan, Cristian
205–219