Rossner, AlexanderPagel, SvenDörner, Ralf2023-08-242023-08-242023https://dl.gi.de/handle/20.500.12116/42145This research paper examines a user-centered evaluation approach of artificial intelligence (AI) systems in the context of identifying emotional highlight scenes in reality TV formats. The study investigates the accuracy and reliability of AI compared to humans in identifying these highlights and explores viewers’ ability to distinct human versus AI-assisted decisions. Internal user tests with media company employees (enterprise users) demonstrate that the AI algorithm developed in the AI4MediaData research project achieves a high level of accuracy, closely aligning with human editors’ assessments. External user tests with viewers (consumers) reveal that participants are unable to distinguish whether highlight clips were identified by humans or by an AI. These findings emphasize the importance of user-centered evaluations that go beyond algorithm-centered evaluations to ensure useful AI-based systems. The research contributes to the advancement of Human-Centered Artificial Intelligence (HCAI) by considering both cognitive and emotional elements in AI-assisted decision-making.Text/Workshop Paper10.18420/muc2023-mci-ws16-393