When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?

[howardyclo]

Metadata

• Authors: Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Janani Padmanabhan and Graham Neubig
• Organization: Language Technologies Institute, Carnegie Mellon University
• Release Date: 2018 on Arxiv
• Link: https://arxiv.org/pdf/1804.06323.pdf

[howardyclo]

Summary

1. Pre-training the word embeddings in the source and/or target languages helps to increase BLEU scores.
2. Pre-training source language embeddings gains much improvement, indicating that better encoding of the source sentence is important.
3. Word embeddings are most effective, where there is very little training data but not so little that the system cannot be trained at all.
4. The gain from pre-training of embeddings may be larger when the source and target languages are more similar.
5. A priori alignment of embeddings may not be necessary in bilingual scenarios, but is helpful in multi-lingual training scenarios.