Zhang, WanyiPasserini, AndreaGiunchiglia, Fausto2021-04-232021-04-2320202020http://dx.doi.org/10.1007/s13218-020-00630-5https://dl.gi.de/handle/20.500.12116/36292We propose an interactive machine learning framework where the machine questions the user feedback when it realizes it is inconsistent with the knowledge previously accumulated. The key idea is that the machine uses its available knowledge to check the correctness of its own and the user labeling. The proposed architecture and algorithms run through a series of modes with progressively higher confidence and features a conflict resolution component. The proposed solution is tested in a project on university student life where the goal is to recognize tasks like user location and transportation mode from sensor data. The results highlight the unexpected extreme pervasiveness of annotation mistakes and the advantages provided by skeptical learning.Interactive learningKnowledge and learningManaging annotator mistakesText/Journal Article10.1007/s13218-020-00630-51610-1987