[Question] Can we obtain more fine-grained answers through fine-tuning

[CrazyBrick]

Question

Can fine-tuning be used on custom datasets to obtain more accurate outputs. For example, for some less famous people, can the model recognize them and know their names. If possible, which parameter need to be adjusted, how many samples are needed, and whether the Q&A data can only be in English. Looking forward to hearing from you.

[laserwave]

1. Did you use a LLM model that is pretrained with Chinese data of large scale (and ensure the tokenizer is suitable for Chinese tokenization) ?
2. Did you use full parameter fine-tuning or at least tune the first and last layer of the LLM?

[CrazyBrick]

1. Did you use a LLM model that is pretrained with Chinese data of large scale (and ensure the tokenizer is suitable for Chinese tokenization) ?
2. Did you use full parameter fine-tuning or at least tune the first and last layer of the LLM?

Hi @laserwave, thank you for your reply. q1:no q2:no Now I just tried the lora_finetune from readme considering my limited data situation, so for I did not use full parameter fine-tuning.

If you have a limited task-specific data, we recommend finetuning from LLaVA checkpoints with LoRA following this script

I just find the llava-1.5 have the ability to conduct Chinese Q&A, I thought its training data included some Chinese, but later someone reminded me of this issue. For convenience, I have converted my data into English.

And about tune the first and last layer of the LLM, could you plz share more, I don't know how to adjust the code to make it run successfully.

[laserwave]

And about tune the first and last layer of the LLM, could you plz share more, I don't know how to adjust the code to make it run successfully.

This may be the reason you cannot get Chinese response after fine-tuning with Chinese data.

You can check the exact parameter names of these layers and set p.requires_grad = True. Make sure it's not set to False again after this setting. Or you can also try full parameter fine-tuning.

[CrazyBrick]

You can check the exact parameter names of these layers and set p.requires_grad = True. Make sure it's not set to False again after this setting. Or you can also try full parameter fine-tuning.

I may not have enough data to support full parameter fine-tuning.

[fisher75]

You can check the exact parameter names of these layers and set p.requires_grad = True. Make sure it's not set to False again after this setting. Or you can also try full parameter fine-tuning.

I may not have enough data to support full parameter fine-tuning.

how much data is required for full parameter ft? I got 100k images wonder if it's enough.