Phil Wang

@dwromero the other thing that would be helpful (if you have the time), is to run it with only one quantizer and see if it still errors 🙏

@dwromero hey David, realized just now the local sampling won't work, as the codes will no longer be synced could you try again on the latest?

@dwromero hey David again so I think your error may be related to an issue with the quantize dropout in a distributed environment, which would also make the above solution...

another way to avoid this issue is to offer a way to delay the expiration of the codes until all the quantizers have been invoked

@dwromero yes please! 🙏

@dwromero oops! one more time?

almost there :smile:

@dwromero works?

nice! happy training!

someone actually pull requested this in and i'm unfamiliar with it does the reconstruction loss look good?