Missing parameter of beta

[Corleone-Huang]

Hi, in the original VQVAE paper, the commit_loss is defined as

(quantize.detach()-x) ** 2 + beta * (quantize - x.detach() ** 2)

where the beta is usually to be 0.25. But the commit_loss is defined as the following in your implementation:

F.mse_loss(quantize.detach(), x)

So I wonder if the parameter beta is set to be 1 by default or if the second term is missing? Thank you very much.

[lucidrains]

@Corleone-Huang Hello Corleone! So the beta would correspond to the commit_weight https://github.com/lucidrains/vector-quantize-pytorch/blob/master/vector_quantize_pytorch/vector_quantize_pytorch.py#L543

[dortenenboim314]

Hi @lucidrains, seems that the commitment loss in your code refers only to the first term of the equation:

(quantize.detach()-x) ** 2 + beta * (quantize - x.detach() ** 2)

Does the second term appears in you code somewhere? If it does, can you please help me find it? I tried and found no references to it.

If it doesn't, what "moves" the codebook towards the encoder's output? The commit loss seems to "move" only the encoder outputs.

I'm asking beacsue I'm trying to train RVQ+Decoder on top of a fixed pre trained Encoder (without fine tune the encoder). And it seems that the commit loss is just getting bigger.

[naitian]

In this repository, the codebook is updated based on exponential moving averages.