Towards Explanatory Interactive Image Captioning Using Top-Down and Bottom-Up Features, Beam Search and Re-ranking
| dc.contributor.author | Biswas, Rajarshi | |
| dc.contributor.author | Barz, Michael | |
| dc.contributor.author | Sonntag, Daniel | |
| dc.date.accessioned | 2021-04-23T09:36:45Z | |
| dc.date.available | 2021-04-23T09:36:45Z | |
| dc.date.issued | 2020 | |
| dc.description.abstract | Image captioning is a challenging multimodal task. Significant improvements could be obtained by deep learning. Yet, captions generated by humans are still considered better, which makes it an interesting application for interactive machine learning and explainable artificial intelligence methods. In this work, we aim at improving the performance and explainability of the state-of-the-art method Show, Attend and Tell by augmenting their attention mechanism using additional bottom-up features. We compute visual attention on the joint embedding space formed by the union of high-level features and the low-level features obtained from the object specific salient regions of the input image. We embed the content of bounding boxes from a pre-trained Mask R-CNN model. This delivers state-of-the-art performance, while it provides explanatory features. Further, we discuss how interactive model improvement can be realized through re-ranking caption candidates using beam search decoders and explanatory features. We show that interactive re-ranking of beam search candidates has the potential to outperform the state-of-the-art in image captioning. | de |
| dc.identifier.doi | 10.1007/s13218-020-00679-2 | |
| dc.identifier.pissn | 1610-1987 | |
| dc.identifier.uri | http://dx.doi.org/10.1007/s13218-020-00679-2 | |
| dc.identifier.uri | https://dl.gi.de/handle/20.500.12116/36327 | |
| dc.publisher | Springer | |
| dc.relation.ispartof | KI - Künstliche Intelligenz: Vol. 34, No. 4 | |
| dc.relation.ispartofseries | KI - Künstliche Intelligenz | |
| dc.subject | Beam search | |
| dc.subject | Deep learning | |
| dc.subject | Explainable artificial intelligence (XAI) | |
| dc.subject | Image captioning | |
| dc.subject | Interactive machine learning (IML) | |
| dc.subject | Re-ranking | |
| dc.subject | Visual explanations | |
| dc.title | Towards Explanatory Interactive Image Captioning Using Top-Down and Bottom-Up Features, Beam Search and Re-ranking | de |
| dc.type | Text/Journal Article | |
| gi.citation.endPage | 584 | |
| gi.citation.startPage | 571 |