Di Li comments

Repositories
Issues
Comments

Results 5 comments of


                                            Di Li

weight init in embedding_config should depend on embedding_dim

> Hmm, fair point. Put out a PR? Sg. Any unit test scripts that I should be changing at the same time? @PaulZhang12

weight init in embedding_config should depend on embedding_dim

PR here: https://github.com/pytorch/torchrec/pull/1902

weight init in embedding_config should depend on embedding_dim

> for context I think DLRM init it with 1 / num_embeddings > > https://github.com/facebookresearch/dlrm/blob/main/dlrm_s_pytorch.py#L281 Thanks for the cross ref. Does the switch to embedding_dim make sense to you in...

weight init in embedding_config should depend on embedding_dim

> @di-wnd I think @PaulZhang12 brought it up, and it seems the result of that discussion is that its better to change user input than to change the default @henrylhtsang...

embedding vector init norm by embedding dim instead of num_embeddings

cc @PaulZhang12