Textdokument
Stories Complicate Things: A Qualitative Analysis of Coding Problems (Un)solved by GitHub Copilot
Lade...
Volltext URI
Dokumententyp
Dateien
Zusatzinformation
Datum
2025
Autor:innen
Zeitschriftentitel
ISSN der Zeitschrift
Bandtitel
Verlag
Gesellschaft für Informatik, Bonn
Zusammenfassung
Generative AI has found increasing interest in software development, giving rise to coding assistants such as GitHub Copilot. However, the correctness of generated code varies strongly. Objectives. In this study, we explore characteristics of coding problems that could (not) be solved by GitHub Copilot and use our results to point to new research directions. Methods. We use open coding to label 100 LeetCode coding problems and 50 associated solutions. For the coding problems, analyse the impact of the labels on GitHub Copilot’s ability to solve the coding problems. For the solutions, we use the labels to infer general metrics which we subsequently extract for a total of 535 solutions. Results. Our results point to three characteristics leading to coding problems being solved less frequently: (1) Usage of real-world scenarios for explanation, (2) long descriptions and (3) the need for a more complex solution. Conclusion. The results underscore the need for future research to enable LLMs to handle coding problems with a higher complexity. Moreover, further investigation is needed to validate our initial findings regarding a worse performance of LLMs on real-world scenarios in programming.