Help with replicating the results for Hubert Pretraining

[a43992899]

❓ Questions and Help

What is your question?

I am trying to replicate the HuBERT base pretraining iter1 on librispeech 960hr. However, the training curve seems to be weird, as the unmask correct rate degrades fast. It seems like the model is not converging correctly, is this related to my kmeans? Are there any secret ingredients for training HuBERT? What will be the normal curve when the model is correctly trained?

What's your environment?

• fairseq Version (main)
• PyTorch Version (1.12.1)
• OS (ubuntu)
• How you installed fairseq: source
• Build command you used (if compiling from source):
• Python version: 3.8
• CUDA/cuDNN version: 11.3, 8.3
• GPU models and configuration: 4*A100
• Any other relevant information:

[wnhsu]

Can you share what the WER is when you fine-tune the model? It doesn't seem weird to me since the accuracy on masked tokens continues improving and that is what models is optimized for