x-transformers icon indicating copy to clipboard operation
x-transformers copied to clipboard

Question about best combinations of features

Open yzhang-github-pub opened this issue 3 years ago • 0 comments

Dear Author,

Thanks for your excellent work! I want to try your implementation for language translation related task. I have two questions and I'd appreciate your help very much:

  1. You implemented many features to improve performance. Which features can be combined together?
  2. You mentioned that small initialization of embeddings is taken care of if l2norm flag is set. What is your overall recommendation of weight initialization for the whole model?

Thanks!

yzhang-github-pub avatar Jun 22 '22 12:06 yzhang-github-pub