GroundedTranslation
GroundedTranslation copied to clipboard
Text input: replace TimeDistributedDense with Embedding?
It would be great to have an explicit keras.embeddings.Embedding
layer for the text input instead of a TimeDistributedDense
layer. An Embedding
layer for the text input would let us initialise the model with word2vec or GloVe vectors. This should make it easier to train models because we wouldn't need to also learn the word representations. Another side-effect of using the TimeDistributedDense
layer is the size of the word representations is tied to the size of the hidden layer.
Fixing this issue would require a rewrite of how the data_generator
builds the text input and text targets, and then changing the model so it has an Embedding
layer instead of a TimeDistributedDense
layer.
@scfrank if you agree this makes sense, could you take a look into this?