Logo des Repositoriums
 

Stories Complicate Things: A Qualitative Analysis of Coding Problems (Un)solved by GitHub Copilot

Kurzanzeige
dc.contributor.authorOertel, Julian
dc.contributor.authorKlüner, Jil
dc.contributor.authorHebit, Regina
dc.contributor.editorFeichtinger, Kevin
dc.contributor.editorSonnleithner, Lisa
dc.contributor.editorHajiabadi, Hamideh
dc.date.accessioned2025-02-14T10:03:34Z
dc.date.available2025-02-14T10:03:34Z
dc.date.issued2025
dc.description.abstractGenerative AI has found increasing interest in software development, giving rise to coding assistants such as GitHub Copilot. However, the correctness of generated code varies strongly. Objectives. In this study, we explore characteristics of coding problems that could (not) be solved by GitHub Copilot and use our results to point to new research directions. Methods. We use open coding to label 100 LeetCode coding problems and 50 associated solutions. For the coding problems, analyse the impact of the labels on GitHub Copilot’s ability to solve the coding problems. For the solutions, we use the labels to infer general metrics which we subsequently extract for a total of 535 solutions. Results. Our results point to three characteristics leading to coding problems being solved less frequently: (1) Usage of real-world scenarios for explanation, (2) long descriptions and (3) the need for a more complex solution. Conclusion. The results underscore the need for future research to enable LLMs to handle coding problems with a higher complexity. Moreover, further investigation is needed to validate our initial findings regarding a worse performance of LLMs on real-world scenarios in programming.en
dc.identifier.doi10.18420/se2025-ws-14
dc.identifier.eissn2944-7682
dc.identifier.issn2944-7682
dc.identifier.urihttps://dl.gi.de/handle/20.500.12116/45822
dc.language.isoen
dc.publisherGesellschaft für Informatik, Bonn
dc.relation.ispartofSoftware Engineering 2025 – Companion Proceedings
dc.subjectGitHub Copilot
dc.subjectCode Generation
dc.subjectLarge Language Model
dc.subjectGenerative AI
dc.titleStories Complicate Things: A Qualitative Analysis of Coding Problems (Un)solved by GitHub Copiloten
dc.title.subtitleA Qualitative Analysis of Coding Problems (Un)solved by GitHub Copiloten
mci.conference.date22.-28. Februar 2025
mci.conference.locationKarlsruhe
mci.conference.sessiontitle2nd Workshop on Generative and Neurosymbolic AI in Software Engineering (GenSE’25)
mci.reference.pages167-178

Dateien

Originalbündel
1 - 1 von 1
Vorschaubild
Name:
B4-4.pdf
Größe:
159.9 KB
Format:
Adobe Portable Document Format
Herunterladen

Sammlungen

SE 2025 - Companion Proceedings