numpy-ml icon indicating copy to clipboard operation
numpy-ml copied to clipboard

Backpropagation of dropout wrapper does not apply the mask

Open TzuChieh opened this issue 1 year ago • 0 comments

Hi, I recently discovered this excellent repository for learning basic concepts in ML and noticed a potential problem in the implementation of dropout wrapper. In particular, this is the line of code I am confused about:

https://github.com/ddbourgin/numpy-ml/blob/b0359af5285fbf9699d64fd5ec059493228af03e/numpy_ml/neural_nets/wrappers/wrappers.py#L231

Shouldn't the gradient from a later layer also apply the mask from the one used in forward()? Otherwise, dLdy will overestimate the true gradient after dividing by the probability 1.0 - self.p. Not sure this is an issue though as I am a beginner in ML.

TzuChieh avatar May 04 '24 18:05 TzuChieh