yaringal
Calculation of Tau is different from that on the blog
Hi @yaringal,
Thanks for the wonderful post and the code. I was going through your blogpost and found that there are some inconsistencies in the way you calculate tau in the blog, the supportive python code as well as in this repo at the following line:
https://github.com/yaringal/DropoutUncertaintyDemos/blob/92a371ecfd00fdb514f1811778d44e9aad0ae1c9/convnetjs/regression_uncertainty.js#L164
As I understand, you directly calculate tau_inv in the js code. However, on the line above you add the tau_inv to the std. dev of y instead of the variance of y as mentioned on the blog. Is there a specific reason for this ?
I have tried to replicate your analysis using pytorch at: https://github.com/napsternxg/pytorch-practice/blob/master/Pytorch%2BUncertainity.ipynb, based on the method proposed on the blogpost. However, I am not getting the same uncertainties. Specifically, the uncertainty, it appears, is constant across all X, when in reality it should be high at the edges.
I would appreciate if you can help me with this.
Well spotted! The tau inverse (ie variance) should be inside the squared root. I'll have a look at your code when I get back to the UK next week (from a look at the video I would say it might be an issue with the prior length scale or the bias initialization) Yarin
Thanks. What is usually the range of length scaling? In your code you directly use l2 = length-scale squared. Or is it like hyperparameter, I need to try different values to see which one works.
it's a hyperparameter which depends on the function and the range of the X (incidentally, make sure your data is normalised)
Oh thanks. Yes, I didn't normalize my data, but made sure that the range of X was between -1, 1 in an undated code and that improved things a bit. Do I need to normalize y as well ?
Ideally yes with neural networks
On Thu, 12 Oct 2017, 12:46 Shubhanshu Mishra, [email protected] wrote:
Oh thanks. Yes, I didn't normalize my data, but made sure that the range of X was between -1, 1 in an undated code and that improved things a bit. Do I need to normalize y as well ?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/yaringal/DropoutUncertaintyDemos/issues/2#issuecomment-336196280, or mute the thread https://github.com/notifications/unsubscribe-auth/ADpAqbCSRVMRdtQYocNTj_EcpUhEFDcrks5srkJrgaJpZM4P2Qur .