Logo des Repositoriums

Auflistung nach Schlagwort "representation learning"

1 - 3 von 3
  • Textdokument
    Citcom – Citation Recommendation
    (INFORMATIK 2020, 2021) Meyer, Melina; Frey, Jenny; Laub, Tamino; Wrzalik, Marco; Krechel, Dirk
    Citation recommendation aims to predict references based on a given text. In this paper, we focus on predicting references using small passages instead of a whole document. Besides using a search engine as baseline, we introduce two further more advanced approaches that are based on neural networks. The first one aims to learn an alignment between a passage encoder and reference embeddings while using a feature engineering approach including a simple feed forward network. The second model takes advantage of BERT, a state-of-the-art language representation model, to generate context-sensitive passage embeddings. The predictions of the second model are based on inter-passage similarities between the given text and indexed sentences, each associated with a set of references. For training and evaluation of our models, we prepare a large dataset consisting of English papers from various scientific disciplines.
  • Konferenzbeitrag
    Inductive Learning of Concept Representations from Library-Scale Bibliographic Corpora
    (INFORMATIK 2019: 50 Jahre Gesellschaft für Informatik – Informatik für Gesellschaft, 2019) Galke, Lukas; Melnychuk, Tetyana; Seidlmayer, Eva; Trog, Steffen; Förstner, Konrad U.; Schultz, Carsten; Tochtermann, Klaus
    Automated research analyses are becoming more and more important as the volume of research items grows at an increasing pace. We pursue a new direction for the analysis of research dynamics with graph neural networks. So far, graph neural networks have only been applied to small-scale datasets and primarily supervised tasks such as node classification. We propose to use an unsupervised training objective for concept representation learning that is tailored towards bibliographic data with millions of research papers and thousands of concepts from a controlled vocabulary. We have evaluated the learned representations in clustering and classification downstream tasks. Furthermore, we have conducted nearest concept queries in the representation space. Our results show that the representations learned by graph convolution with our training objective are comparable to the ones learned by the DeepWalk algorithm. Our findings suggest that concept embeddings can be solely derived from the text of associated documents without using a lookup-table embedding. Thus, graph neural networks can operate on arbitrary document collections without re-training. This property makes graph neural networks useful for the analysis of research dynamics, which is often conducted on time-based snapshots of bibliographic data.
  • Konferenzbeitrag
    What If We Encoded Words as Matrices and Used Matrix Multiplication as Composition Function?
    (INFORMATIK 2019: 50 Jahre Gesellschaft für Informatik – Informatik für Gesellschaft, 2019) Galke, Lukas; Mai, Florian; Scherp, Ansgar
    We summarize our contribution to the International Conference on Learning Representations CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model, 2019.We construct a text encoder that learns matrix representations of words from unlabeled text, while using matrix multiplication as composition function. We show that our text encoder outperforms continuous bag-of-word representations on 9 out of 10 linguistic probing tasks and argue that the learned representations are complementary to the ones of vector-based approaches. Hence, we construct a hybrid model that jointly learns a matrix and a vector for each word. This hybrid model yields higher scores than purely vector-based approaches on 10 out of 16 downstream tasks in a controlled experiment with the same capacity and training data. Across all 16 tasks, the hybrid model achieves an average improvement of 1.2%. These results are insofar promising, as they open up new opportunities to efficiently incorporate order awareness into word embedding models.