Design of a Third-Order Word Embedding Model Using Vector Projections

Geometrical representations of words are building blocks in the field of natural language processing, especially for tasks such as word sense disambiguation, sentiment analysis, and language modeling. The efforts for developing static word embeddings go all the way back to the 1980s, and currently there are many models in the literature each approaching the problem with a different methodology. In this study, we propose a novel neural static word embedding model that is learned using orthogonal projections of vectors to represent contextual relationships. Our approach employs a third-order model and leverages the contrastive learning paradigm, in which a positive training sample consists of a target word and two context words, and a negative sample consists of one target word, one context word, and a randomly sampled noise word from the corpus. We developed a unique geometrical loss function that aims to minimize the difference between the orthogonal projections of the selected context words onto the target word and maximize the difference between the orthogonal projections of the context word and the negative samples onto the target word. This approach is distinct from the traditional static embedding models, and results in word embeddings that effectively capture contextual information in a higher-order, projective manner. Preliminary experiments on benchmark datasets demonstrate the promising performance of our model in capturing word semantics and contextual relationships. We provide a strong starting point for further studies to explore in depth the theoretical underpinnings of this approach and evaluate its performance in downstream NLP tasks.