Fine tuning with regression head

[7AtAri]

I adapted your fine tuning notebook to a regression task. Unfortunately the model gets unstable during training and only returns NANs as hidden states to my regression head.

Also I wanted to try qlora, but could not use the peft library because the model does not support gradient checkpointing.

so finetuning currently seems to be possible only for some tasks…could it be possible to use peft for moondream in the future?

[vikhyat]

PEFT is possible -- we use it internally and I know of other folks who have had success too. I have not tried qlora though. I would recommend looking into learning rate/other hyperparmeters if you're getting NaNs as hidden states, also make sure you're training in bfloat16 not float16 (unless you're taking other measures to ensure training stability in float16).