mohdumar644
mohdumar644
I want to know the same. Probably the Binarize operation on the layer weights is not recorded (the weights are leaf nodes?). During backward pass, the gradient is calculated w.r.t....
weights are binarized in forward pass using `self.weight.data=Binarize(self.weight.org)`. prior to optimizer step, real weights are [copied](https://github.com/itayhubara/BinaryNet.pytorch/blob/f5c3672dede608f568e073a583cadd7a8a88fa9d/main_binary.py#L255) back.
~~btw, a more understandable approach to constructing binarized/quantized neural networks in PyTorch can be found [here ](https://github.com/ussamazahid96/BNN-PYNQ/tree/master/bnn/src/training/Pytorch) with a custom [Quantizer](https://github.com/ussamazahid96/BNN-PYNQ/blob/166f2fe40d78468edd0ccaa665d2dcc11aa3dc99/bnn/src/training/Pytorch/lenet.py#L44)/Binarizer module having [explicit STE](https://github.com/ussamazahid96/BNN-PYNQ/blob/166f2fe40d78468edd0ccaa665d2dcc11aa3dc99/bnn/src/training/Pytorch/binarized_modules.py#L24) in the backward pass.~~
> @mohdumar644 thanks for the links! In the attached implementation too, the class [BinarizeLinear](https://github.com/ussamazahid96/BNN-PYNQ/blob/166f2fe40d78468edd0ccaa665d2dcc11aa3dc99/bnn/src/training/Pytorch/binarized_modules.py#L53) uses the function [QuantizeWeights](https://github.com/ussamazahid96/BNN-PYNQ/blob/166f2fe40d78468edd0ccaa665d2dcc11aa3dc99/bnn/src/training/Pytorch/binarized_modules.py#L27) which is very similar to this PyTorch implementation. They also use the...
> * One is the hardtanh [here](https://github.com/itayhubara/BinaryNet.pytorch/blob/master/models/resnet_binary.py#L94). This function is called between bn and next conv layer. hardtanh makes output between -1 and +1. This makes backward of binarization inside...
As per [this](https://github.com/itayhubara/BinaryNet.pytorch/blob/master/data.py), - the cifar-x datasets (and even the mnist dataset when needed) will be automatically downloaded and extracted in a certain folder the first time the script is...
The line ``` input.data=Binarize(input.data) ``` in binarized_modules.py is related to the activations. It binarizes the output of the HardTanh
> @mohdumar644 : I am confused now because in the backward pass, the gradients for activations are computed at the binary activations and not the real ones. Suppose **h** is...
> @mohdumar644 Thanks for the clarification. I was under the impression that the gradients with respect to activations are always calculcated w.r.t to real activation values. If they are computed...
Yes the paper's algorithm can be functionally achieved using a variety of ways in different frameworks.