Neural-Photo-Editor
Neural-Photo-Editor copied to clipboard
Possibly incorrect implementation of Batch Renorm
I came across your implementation of batch re-normalization in the BatchReNormDNNLayer class, and I think there might be an error that might be affecting the model's performance.
My understanding of batch re-norm is that it applies the standard BN normalization first, then applies the r/d correction, and then finally applies the gamma/beta scaling and bias. Something along the lines of this:
normed_x = (x - batch_mean) / batch_std # standard BN
normed_x = normed_x * r + d # The batch renorm correction
normed_x = normed_x * gamma + beta # final scale and bias
However, this line is applying the r/d correction after the scaling and centering with gamma and beta. https://github.com/ajbrock/Neural-Photo-Editor/blob/master/layers.py#L128
It probably works anyway, based on the good results you seem to have gotten. I just thought I'd bring it to your attention.
Thanks, this is definitely a discrepancy, likely born of the fact that I threw this layer together in a couple minutes right after seeing the paper. Also because cuDNN's batchnorm does the scale+shift automatically. If you want to submit a pull request to fix it (just moving the gamma+beta outside the cuDNN bnorm call would do it) feel free, though I'm not really supporting much of the code in this repo thse days <_<