How to use BERT contextualized embeddings

[spookypineapple]

Is it possible to use BERT based contextualized word embeddings along with the nmt implementation? I want to take advantage of the pretrained BERT language model so the NMT weights can be leveraged more to train for the task im solving instead of being used to account for language model (grammer, structure, etc..)

[hichiaty]

Same question but with ELMo, I'm guessing you'd have to modify the embedding lookup

[nashid]

@hichiaty @spookypineapple wondering you were able to use BERT OR ELMo embedding?