Weight Initialization

[jonny-d]

Hello,

Thank you very much for sharing this code. I am attempting to re-train a model like this from scratch and was wondering which weight initialization method was used for training the model?

Thanks, Jonny

[raulpuric]

Without any regularization I personally found that uniform sampling gave faster convergence, but was more unstable and blew up (see my issue #39). I also tried xavier initialization and that seemed to be more stable.

[jonny-d]

Hello, sorry for the delayed response.

I have achieved pretty good performance using a normal distribution for the initial weights. Here is a link to my Tensorflow Implementation

[raulpuric]

@jonnykira I found like you that they used weight norm in the paper which I initially glossed over/isn't in the code base. This turned out to be what I needed to use.

[nkooli]

@jonnykira hello, I trained the model on my dataset, this generates me three files : model.data, model.index, model.meta

How can i generates the 15 .npy weight files (0.npy, 1.npy, ..., 14.npy) to test the sentiment analysis code (as in in https://github.com/openai/generating-reviews-discovering-sentiment) ?

thank you !