About Dual-Codebook for Vector Quantization

[tanbuzheng]

Dear author, Thanks for sharing the code. I am greatly interested in your work. I have a question for you and would like your reply. In P-VQVAE, the masked and unmasked patches are encoded by linear layers independently, without considering any contexts. Does this influence the quality of the semantic codebooks, especially the codebook for the masked regions?