Show simple item record

dc.contributor.authorSchleith, Johannes
dc.contributor.authorHristozova, Nina
dc.contributor.authorChechmanek, Brian
dc.contributor.authorBussey, Carolyn
dc.contributor.authorMichalak, Leszek
dc.contributor.editorWienrich, Carolin
dc.contributor.editorWintersberger, Philipp
dc.contributor.editorWeyers, Benjamin
dc.date.accessioned2021-09-05T18:56:36Z
dc.date.available2021-09-05T18:56:36Z
dc.date.issued2021
dc.identifier.urihttp://dl.gi.de/handle/20.500.12116/37374
dc.description.abstractNatural language processing (NLP) techniques for information extraction commonly face the challenge to extract either ‘too much’ or ‘too little’ information from text. Extracting ‘too much’ means that a lot of the relevant information is captured, but also a lot of irrelevant information or ‘Noise’ is extracted. This usually results in high ‘Recall’, but lower ‘Precision’. Extracting ‘too little’ means that all of the information that is extracted is relevant, but not everything that is relevant is extracted – it is ‘missing’ information. This usually results in high ‘Precision’ and lower ‘Recall’. In this paper we present an approach combining quantitative and qualitative measures in order to evaluate the end-users’ experience with information extraction systems in addition to standard statistical metrics and interpret a preference for the above challenge. The method is applied in a case study of legal document review. Results from the case study suggest that legal professionals prefer seeing ‘too much’ over ‘too little’ when working on an AI-assisted legal document review tasks. Discussion of these results position the involvement of User Experience (UX) as a fundamental ingredient to NLP system design and evaluation.en
dc.language.isoen
dc.publisherGesellschaft für Informatik e.V.
dc.relation.ispartofMensch und Computer 2021 - Workshopband
dc.relation.ispartofseriesMensch und Computer
dc.subjectUser centric evaluation of AI-based systems
dc.subjectPrecision
dc.subjectRecall
dc.subjectTask Support
dc.titleNoise over Fear of Missing Outen
dc.typeText/Conference Poster
dc.pubPlaceBonn
mci.document.qualitydigidoc
mci.conference.sessiontitleMCI-WS02: UCAI 2021: Workshop on User-Centered Artificial Intelligence
mci.conference.locationIngolstadt
mci.conference.date5.-8. September 2021
dc.identifier.doi10.18420/muc2021-mci-ws02-290


Files in this item

Thumbnail

Show simple item record