HIERARCHICAL ENCODING AND CONDITIONAL ATTENTION IN NEURAL MACHINE TRANSLATION

Аннотация

INTRODUCTION serve as the basis for sequence-to-sequence tasks, The creation of the Transformer models, which https://www.theamericanjournals.com/index.php/tajssei 25 THE USA JOURNALS THE AMERICAN JOURNAL OF SOCIAL SCIENCE AND EDUCATION INNOVATIONS (ISSN- 2689-100X) VOLUME 06 ISSUE09 has led to notable advances in Neural Machine absolute position of the tokens in the sequence. It Translation (NMT). Introduced by Vaswani et al. could be said that the model operates on a bag-of- [1] in 2017, the vanilla Transformer model has tokens derived from the initial sequence. To found extensive use in natural language processing provide the model with information about the (NLP). The model handles the links between relative and absolute position of the tokens the tokens inside a sequence using self-attention original paper suggests to use positional encoding. instead of recurrence or convolution. The primary That is an additional embedding of tokens based focus of this design is on translating phrases solely on their position in the sequence that is independently, which might present difficulties added to the tokens’ original embedding. when translating longer texts because cross- 1.1.1 Is Positional Encoding Needed at All? sentence context is necessary to uphold coherence, There is no known empirical study of the clear out ambiguities, and preserve the original importance of the positional encoding mechanism meaning. for the performance of the Transformer model. This study primary focus is to address these However, it could be argued that the results for challenges and improve the quality and Convolutional S2S model [2] is relatable to some consistency of NMT by maintaining the structural extent to the Transformer model since it is also a and semantic connections between sentences no recurrence model. The Convolutional S2S inside a document. Instead of processing each model was shown to perform well without any sentence separately, redesigning of the positional positional embedding at all [2]. It was also shown encoding mechanism in processing sequences that a vanilla encoder-decoder RNN model could allows it to span numerous sentences. benefit significantly from a refinement of word In order to capture sentence-level dependencies, embeddings with the source sentence’s bag-of- this paper discusses the shortcomings of current words representation [3]. These results suggest positional encoding approaches in discourse-level that the knowledge of the position of tokens might contexts, suggests a hierarchical encoding be of little importance for NMT systems strategy, and presents a novel conditional processing texts in the sentence-by-sentence attention mechanism that allows relevant context fashion. The order of words in output translation is to be selected from the most relevant sentences preserved by the auto-regressive manner of the within a document. By enhancing NMT systems' decoder and in most cases words’ meaning could ability to manage lengthier sequences with be disambiguated just by the bag-of-words intricate inter-sentential interactions, these representation of the source sentence. Now let’s contributions hope to produce translations that consider the case of processing the whole are more precise and sensitive to context. document with the Transformer at once by simple concatenation of sentences together. It is obvious 1 POSITIONAL ENCODING that examination of the sequence (that consists of 1.1 Motivation for Positional Encoding every word in the document) in a bag-of-words The vanilla Transformer model contains no manner would be of no use in the task of recurrence and no convolution, so when attention disambiguation of a word’s meaning if that word used in an unrestricted manner (attention being used in the document in several meanings. The performed over the whole sequence) the model same logic is applicable to the challenges of does not have information about the relative or cohesion and coherence maintenance (e.g. https://www.theamericanjournals.com/index.php/tajssei 26 THE USA JOURNALS THE AMERICAN JOURNAL OF SOCIAL SCIENCE AND EDUCATION INNOVATIONS (ISSN- 2689-100X) VOLUME 06 ISSUE09 anaphora resolution). Therefore, a way to [4] suggests several context fusion strategies of condition the attention scores by the relative which the best-performing one is the position of the tokens is needed. concatenation of the previous source sentence to the sentence being processed with a special 1.2 Redesigning Positional Encoding sentence-break token between them. Authors of Since there is no known prior work on positional the paper demonstrate that such approach encoding with respect to a discourse-level context, significantly improves the overall translation the implementation could be suggested based on quality of the encoder-decoder RNN model. This the various assumptions regarding the context. approach seems to be inapplicable to the Definition 1.1. Structural unit is a part of a text that Transformer model as i