settings

[B3AU]

Hi Jonatas,

I'm trying to replicate your performance on the dutch language but can't get nearly the same WER.

What are the parameters you use to train the final model? Do they come from the sweep?

Thanks! Beau

[jonatasgrosman]

Hi Beau! I've been pretty busy lately. Sorry for the delay in replying...

You're not the first person that asks me for this 😄

I could put here the parameters I've previously used, but I've recently changed the training approach and got better results. I'm running some experiments right now to check the results I've got.

ASAP I'll release some improved models in the HuggingFace, and I'm planning to describe my training approach in detail in some way for everybody (probably on a blog post or maybe on an arXiv paper). I'll ping you here when this happens!

Regards Jonatas Grosman

[B3AU]

Any update on this? I'd be fine with just a code commit. Sorry for the impatience.. running my own experiments and don't want to waste time possibly doing similar things.

[StephennFernandes]

@jonatasgrosman , could you please share or update your improved training approach here, in-case if you already have could you please share the following links please.

[jonatasgrosman]

Hi @B3AU and @StephennFernandes ! Sorry for the very late reply. I had some health issues that forced me to stay offline for a while. My life is going back to normal slowly now...

About my new training approach, I've made some modifications to the code of this repository and I've created a new one with these modifications. But in summary, to achieve better results in comparison to my previous ones, I actually just added a language model for decoding and changed the pre-trained model size (from 300M to 1B parameters). You can check these 1B new models here. And it seems that size really matters, you can see that comparing a 1B fine-tuned model VS a 2B fine-tuned model results.