Fine-tuning whisper.cpp models

[kittechAI]

Is it possible to fine-tune multilingual whisper.cpp models, in a non-english language, using python, just like we can fine-tune any other OpenAI whisper model. If so, please help on how to get started with this.

I want this model to be able to run on a CPU only, thats the reason why I want to fine-tune whisper.cpp model instead of the OpenAI`s whisper models.

[kittechAI]

Please help @ggerganov

[ephemer]

You'll need to fine tune the original whisper model in python and then convert the fine-tuned model to ggml for use in whisper.cpp.

I have never done either before so I can't help you more than that, but that's how you need to approach this problem.

[kittechAI]

Thank you very much, you really made my day

[kittechAI]

@ggerganov . This is not an issue with the ggml models as such. How do I convert an openai/whisper model that I have fine-tuned, to the ggml-format, and if I want the model to work in real-time without internet access, are there any ways of performing such a feet using the ggml format. I want to include the model in my C++ Qt desktop application.

Your help in this regard will be sincerely appreciated.

[ggerganov]

Did you try following the instructions in the README ?

[kittechAI]

Ok thanks. I wanted to know if there is a way I can make the model transcribe in real-time instead, after it has been successfully converted to ggml format. And also, I happened to have trained my model with peft lora, so will the same methods apply. Sorry if I sound so unlearned, indeed I am, but I really need help in this area.

[jiahansu]

@kittechAI You can merge the adapter with the base model to create a new full model, and then convert it to GGML.

peft_model_id = "adapter_model" 
peft_config = PeftConfig.from_pretrained(peft_model_id)
model = WhisperForConditionalGeneration.from_pretrained(
    peft_config.base_model_name_or_path, torch_dtype=torch.float16, device_map="mps"
)
model = PeftModel.from_pretrained(model, peft_model_id)
merged_model = model.merge_and_unload()
merged_model.save_pretrained("model")

[kittechAI]

@kittechAI You can merge the adapter with the base model to create a new full model, and then convert it to GGML.

peft_model_id = "adapter_model" 
peft_config = PeftConfig.from_pretrained(peft_model_id)
model = WhisperForConditionalGeneration.from_pretrained(
    peft_config.base_model_name_or_path, torch_dtype=torch.float16, device_map="mps"
)
model = PeftModel.from_pretrained(model, peft_model_id)
merged_model = model.merge_and_unload()
merged_model.save_pretrained("model")

Thanks very much, will give that a try.