Logo des Repositoriums
 

Leveraging Distillation Techniques for Document Understanding: A Case Study with FLAN-T5

Kurzanzeige
dc.contributor.authorLamott, Marcel
dc.contributor.authorShakir, Muhammad Armaghan
dc.contributor.editorKlein, Maike
dc.contributor.editorKrupka, Daniel
dc.contributor.editorWinter, Cornelia
dc.contributor.editorGergeleit, Martin
dc.contributor.editorMartin, Ludger
dc.date.accessioned2024-10-21T18:24:13Z
dc.date.available2024-10-21T18:24:13Z
dc.date.issued2024
dc.description.abstractThe surge of digital documents in various formats, including less standardized documents such as business reports and environmental assessments, underscores the growing importance of Document Understanding. While Large Language Models (LLMs) have showcased prowess across diverse natural language processing tasks, their direct application to Document Understanding remains a challenge. Previous research has demonstrated the utility of LLMs in this domain, yet their significant computational demands make them challenging to deploy effectively. Additionally, proprietary Blackbox LLMs often outperform their open-source counterparts, posing a barrier to widespread accessibility. In this paper, we delve into the realm of document understanding, leveraging distillation methods to harness the power of large LLMs while accommodating computational limitations. Specifically, we present a novel approach wherein we distill document understanding knowledge from the proprietary LLM ChatGPT into FLAN-T5. Our methodology integrates labeling and curriculum-learning mechanisms to facilitate efficient knowledge transfer. This work contributes to the advancement of document understanding methodologies by offering a scalable solution that bridges the gap between resource-intensive LLMs and practical applications. Our findings underscore the potential of distillation techniques in facilitating the deployment of sophisticated language models in real-world scenarios, thereby fostering advancements in natural language processing and document comprehension domains.en
dc.identifier.doi10.18420/inf2024_120
dc.identifier.isbn978-3-88579-746-3
dc.identifier.pissn1617-5468
dc.identifier.urihttps://dl.gi.de/handle/20.500.12116/45093
dc.language.isoen
dc.publisherGesellschaft für Informatik e.V.
dc.relation.ispartofINFORMATIK 2024
dc.relation.ispartofseriesLecture Notes in Informatics (LNI) - Proceedings, Volume P-352
dc.subjectDocument Understanding
dc.subjectLarge Language Models
dc.subjectLayout Understanding
dc.subjectKnowledge Distillation
dc.titleLeveraging Distillation Techniques for Document Understanding: A Case Study with FLAN-T5en
dc.typeText/Conference Paper
gi.citation.endPage1381
gi.citation.publisherPlaceBonn
gi.citation.startPage1371
gi.conference.date24.-26. September 2024
gi.conference.locationWiesbaden
gi.conference.sessiontitleAI@WORK

Dateien

Originalbündel
1 - 1 von 1
Vorschaubild
Name:
Lamott_Shakir_Leveraging_Distillation_Techniques.pdf
Größe:
646.29 KB
Format:
Adobe Portable Document Format
Herunterladen

Sammlungen

P352 - INFORMATIK 2024 - Lock in or log out? Wie digitale Souveränität gelingt