Concerns about the loss function

[HaoKun-Li]

Hello, thanks for your excellent work and code!

In the paper, the authors claim that they use the same knowledge distillation scheme as LSQ to train the quantized models. I show the screenshot as follows:

However, in the paper of LSQ, LSQ uses both the distillation loss function of Hinton et al. (2015) with temperature of 1 and equal weight given to the standard loss during training. I show the screenshot as follows:

When I read the code in this github. I notice that you have defined both the KD_loss and CrossEntropyLabelSmooth loss, but you use only distillation loss to train the quantized models. Is this a mistake, or a trick to improve the accuracy?

[szq0214]

Hi @HaoKun-Li Thanks for your interest in this work, the KD loss here is following MEAL V2 without the standard loss term, rather than LSQ scheme. It can provide better results.