Finetune hubert

[marcoyang1998]

trafficstars

This recipe implements Hubert transducer model. It supports finetuning a pretrained Hubert model with custom vocabulary using pruned rnnt loss. To use this recipe, you need fairseq as dependency. The finetuning setup (learning rate, optimizer, scheduler, ...) is the same as the original Hubert paper.

[marcoyang1998]

Here are some finetuning results on 960h:

model name	test-clean	test-other
Hubert base	2.82	7.09
Hubert large	1.93	3.93

Models are trained using BPE500, WERs are obtained using modified beam search.

[csukuangfj]

Could you also update README.md and RESULTS.md?

[marcoyang1998]

Could you also update README.md and RESULTS.md?

Just did it.

[desh2608]

Perhaps some of the files in finetune_hubert_transducer can be removed or converted to soft links (such as asr_datamodule.py)?

[ezerhouni]

@marcoyang1998 Looks pretty nice ! @marcoyang1998 and @csukuangfj tell me if you need a hand for closing this PR

[csukuangfj]

@marcoyang1998 Looks pretty nice ! @marcoyang1998 and @csukuangfj tell me if you need a hand for closing this PR

@ezerhouni Thanks! Let's see what @marcoyang1998 would comment on this.

[marcoyang1998]

Sorry, I may be slow to respond during the holiday. Will catch up on all of this after the holiday!

[marcoyang1998]

I updated the huggingface repo. Everything should be ready now.

[csukuangfj]

I updated the huggingface repo. Everything should be ready now.

Thanks! Will look at it later today.

[marcoyang1998]

I need to update pretrained.py as hubert requires waveform as input.