What If We Encoded Words as Matrices and Used Matrix Multiplication as Composition Function?
| dc.contributor.author | Galke, Lukas | |
| dc.contributor.author | Mai, Florian | |
| dc.contributor.author | Scherp, Ansgar | |
| dc.contributor.editor | David, Klaus | |
| dc.contributor.editor | Geihs, Kurt | |
| dc.contributor.editor | Lange, Martin | |
| dc.contributor.editor | Stumme, Gerd | |
| dc.date.accessioned | 2019-08-27T12:55:26Z | |
| dc.date.available | 2019-08-27T12:55:26Z | |
| dc.date.issued | 2019 | |
| dc.description.abstract | We summarize our contribution to the International Conference on Learning Representations CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model, 2019.We construct a text encoder that learns matrix representations of words from unlabeled text, while using matrix multiplication as composition function. We show that our text encoder outperforms continuous bag-of-word representations on 9 out of 10 linguistic probing tasks and argue that the learned representations are complementary to the ones of vector-based approaches. Hence, we construct a hybrid model that jointly learns a matrix and a vector for each word. This hybrid model yields higher scores than purely vector-based approaches on 10 out of 16 downstream tasks in a controlled experiment with the same capacity and training data. Across all 16 tasks, the hybrid model achieves an average improvement of 1.2%. These results are insofar promising, as they open up new opportunities to efficiently incorporate order awareness into word embedding models. | en |
| dc.identifier.doi | 10.18420/inf2019_47 | |
| dc.identifier.isbn | 978-3-88579-688-6 | |
| dc.identifier.pissn | 1617-5468 | |
| dc.identifier.uri | https://dl.gi.de/handle/20.500.12116/24996 | |
| dc.language.iso | en | |
| dc.publisher | Gesellschaft für Informatik e.V. | |
| dc.relation.ispartof | INFORMATIK 2019: 50 Jahre Gesellschaft für Informatik – Informatik für Gesellschaft | |
| dc.relation.ispartofseries | Lecture Notes in Informatics (LNI) - Proceedings, Volume P-294 | |
| dc.subject | machine learning | |
| dc.subject | natural language processing | |
| dc.subject | representation learning | |
| dc.title | What If We Encoded Words as Matrices and Used Matrix Multiplication as Composition Function? | en |
| dc.type | Text/Conference Paper | |
| gi.citation.endPage | 288 | |
| gi.citation.publisherPlace | Bonn | |
| gi.citation.startPage | 287 | |
| gi.conference.date | 23.-26. September 2019 | |
| gi.conference.location | Kassel | |
| gi.conference.sessiontitle | Data Science |
Dateien
Originalbündel
1 - 1 von 1