does the model support fine-tuning?

[bolt163]

as title says,
if someone wants fine-tune the model, what's the process ?

[tuke307]

OmniParser uses florence2 (or blip2-opt-2.7b) for captioning details and a custom trained yolo model for the icon detection. you can finetune both models as you like. I e.g. just trained the yolo model on my custom dataset. but be aware there is only one class embedded and thats the icon label.

you can then replace the yolo *.pt model in the weights/icon_detect folder with your own fine tuned one. for the training, just use the train_args.yaml provided and override stuff like device and batch.

same for the florence2 model. finetune and replace the *.safetensors in weights/icon_caption_florence .

i can provide some code if needed.

[bolt163]

OmniParser uses florence2 (or blip2-opt-2.7b) for captioning details and a custom trained yolo model for the icon detection. you can finetune both models as you like. I e.g. just trained the yolo model on my custom dataset. but be aware there is only one class embedded and thats the icon label.

you can then replace the yolo *.pt model in the weights/icon_detect folder with your own fine tuned one. for the training, just use the train_args.yaml provided and override stuff like device and batch.

same for the florence2 model. finetune and replace the *.safetensors in weights/icon_caption_florence .

i can provide some code if needed.

hi tony, thanks very much

[directorscut82]

if you can provide an example would be great. Thank you