Logo des Repositoriums
 

Assessing Large Language Models for annotating data in Dementia-Related texts: A Comparative Study with Human Annotators

Kurzanzeige
dc.contributor.authorSuravee, Sumaiya
dc.contributor.authorStoev, Teodor
dc.contributor.authorKonow, Sara
dc.contributor.authorYordanova, Kristina
dc.contributor.editorKlein, Maike
dc.contributor.editorKrupka, Daniel
dc.contributor.editorWinter, Cornelia
dc.contributor.editorGergeleit, Martin
dc.contributor.editorMartin, Ludger
dc.date.accessioned2024-10-21T18:24:26Z
dc.date.available2024-10-21T18:24:26Z
dc.date.issued2024
dc.description.abstractAs the aging population grows, the incidence of dementia is rising sharply, necessitating the extraction of domain-specific information from texts to gain valuable insights into the condition. Training Natural Language Processing (NLP) models for this purpose requires substantial amounts of annotated data, which is typically produced by human annotators. While human annotation is precise, it is also labor-intensive and costly. Large Language Models (LLMs) present a promising alternative that could potentially streamline and economize the annotation process. However, LLMs may struggle with complex, domain-specific contexts, potentially leading to inaccuracies. This paper investigates the effectiveness of LLMs in annotating words and phrases in ambiguous dementia-related texts by comparing LLM-generated annotations with those produced by human annotators. We followed a specific annotation scheme and had both the LLM and human raters annotate a corpus of informal texts from forums of family carers of people with dementia. The results indicate a moderate overlap in inter-rater agreement between LLM and expert annotators, with the LLM identifying nearly twice as many instances as the human raters. Although LLMs can partially automate the annotation process, they are not yet fully reliable for complex domains. By refining LLM-generated data through expert review, it is possible to reduce the burden on human raters and accelerate the creation of annotated datasets.en
dc.identifier.doi10.18420/inf2024_36
dc.identifier.eissn2944-7682
dc.identifier.isbn978-3-88579-746-3
dc.identifier.issn2944-7682
dc.identifier.pissn1617-5468
dc.identifier.urihttps://dl.gi.de/handle/20.500.12116/45195
dc.language.isoen
dc.publisherGesellschaft für Informatik e.V.
dc.relation.ispartofINFORMATIK 2024
dc.relation.ispartofseriesLecture Notes in Informatics (LNI) - Proceedings, Volume P-352
dc.subjectData Annotation
dc.subjectLarge Language Model
dc.subjectPeople with Dementia
dc.subjectNamed Entity Recognition
dc.titleAssessing Large Language Models for annotating data in Dementia-Related texts: A Comparative Study with Human Annotatorsen
dc.typeText/Conference Paper
gi.citation.endPage498
gi.citation.publisherPlaceBonn
gi.citation.startPage487
gi.conference.date24.-26. September 2024
gi.conference.locationWiesbaden
gi.conference.sessiontitle8th International Workshop on Annotation of useR Data for UbiquitOUs Systems

Dateien

Originalbündel
1 - 1 von 1
Vorschaubild
Name:
Suravee_et_al_Assessing_Large_Language_Models.pdf
Größe:
481.64 KB
Format:
Adobe Portable Document Format
Herunterladen

Sammlungen

P352 - INFORMATIK 2024 - Lock in or log out? Wie digitale Souveränität gelingt