fast-LayerNorm-TF icon indicating copy to clipboard operation
fast-LayerNorm-TF copied to clipboard

Published 20 hours ago • verified publisher icon MycChiu

Language Modelling

Open hoagy-davis-digges opened this issue 7 years ago • 3 comments

Hi, I'm interested in using this for a language model, which will need more than 5120 in the last dimension, how would I go about expanding this limit?

hoagy-davis-digges avatar Feb 19 '18 12:02 hoagy-davis-digges

You can change the hyper parameters of the model. They currently are:

embedding_dimension = 1024 num_layers = 3 rnn_dim = 2048

Just increase the values and that should do it. Note however that the memory of your graphic card may be the bottleneck.

manuelsh avatar Feb 20 '18 08:02 manuelsh

Do you mean in the model that I perform the layer norm on, so I don't need to change any of this code?

hoagy-davis-digges avatar Feb 21 '18 11:02 hoagy-davis-digges

Sorry @hoagy-davis-digges I confused your message, I thought you were asking about one language model that I built.

On the layernorm model, unfortunately, cannot comment.

manuelsh avatar Feb 21 '18 12:02 manuelsh