vector-quantize-pytorch icon indicating copy to clipboard operation
vector-quantize-pytorch copied to clipboard

Published 20 hours ago • verified publisher icon lucidrains

SimVQ trains Failed!

Open lovekittynine opened this issue 7 months ago • 0 comments
trafficstars

Hi, i have tried SimVQ recently, but i failed. I used the default configuration, the W matrix is a single Linear layer. But i found the activation of codebook is very low, and also the perplexity is low. I set codebook_size is 32, the activation is about 7%. When the model is initialized, the initial codebook activation is high, nearly close 100%, but quickly fall down to zero. Why?

lovekittynine avatar Apr 18 '25 02:04 lovekittynine