Logo des Repositoriums
 
Konferenzbeitrag

What If We Encoded Words as Matrices and Used Matrix Multiplication as Composition Function?

Lade...
Vorschaubild

Volltext URI

Dokumententyp

Text/Conference Paper

Zusatzinformation

Datum

2019

Zeitschriftentitel

ISSN der Zeitschrift

Bandtitel

Verlag

Gesellschaft für Informatik e.V.

Zusammenfassung

We summarize our contribution to the International Conference on Learning Representations CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model, 2019.We construct a text encoder that learns matrix representations of words from unlabeled text, while using matrix multiplication as composition function. We show that our text encoder outperforms continuous bag-of-word representations on 9 out of 10 linguistic probing tasks and argue that the learned representations are complementary to the ones of vector-based approaches. Hence, we construct a hybrid model that jointly learns a matrix and a vector for each word. This hybrid model yields higher scores than purely vector-based approaches on 10 out of 16 downstream tasks in a controlled experiment with the same capacity and training data. Across all 16 tasks, the hybrid model achieves an average improvement of 1.2%. These results are insofar promising, as they open up new opportunities to efficiently incorporate order awareness into word embedding models.

Beschreibung

Galke, Lukas; Mai, Florian; Scherp, Ansgar (2019): What If We Encoded Words as Matrices and Used Matrix Multiplication as Composition Function?. INFORMATIK 2019: 50 Jahre Gesellschaft für Informatik – Informatik für Gesellschaft. DOI: 10.18420/inf2019_47. Bonn: Gesellschaft für Informatik e.V.. PISSN: 1617-5468. ISBN: 978-3-88579-688-6. pp. 287-288. Data Science. Kassel. 23.-26. September 2019

Zitierform

Tags