vector-quantize-pytorch icon indicating copy to clipboard operation
vector-quantize-pytorch copied to clipboard

Published 20 hours ago • verified publisher icon lucidrains

Support perplexity to measure index collapse

Open Pachark opened this issue 2 years ago • 4 comments
trafficstars

Hi! Thanks for you work. In the implementation of VQVAE (https://github.com/deepmind/sonnet/blob/v2/sonnet/src/nets/vqvae.py#L89C1-L89C1), perplexity is used as an evaluation measure for VQ codebook. It monitors whether index collapse happens. Does this project has similar parameter?

Pachark avatar Jul 24 '23 02:07 Pachark

oh sure, could add this if you are studying index collapse

maybe could add the last calculated perplexity on a non persisted buffer, turned on with a flag?

lucidrains avatar Jul 24 '23 02:07 lucidrains

oh that's great. Add perplexity may tell us what's going on inside the codebook

Pachark avatar Jul 24 '23 06:07 Pachark

@lucidrains Hi, I've found a new problem when applying Residual VQ. Not sure if it's related to index collapse.

I print the embed of codebook, but the values in the middle layer of Residual VQ are updated to huge values such as 60+. Is this normal?  image

Pachark avatar Jul 24 '23 13:07 Pachark