Starry

### Describe the issue Issue: I want to fine-tune the full parameters of llava1.5, not only the llm and projector, but also the vision module. However, after I saved the...

很棒的工作！！！ 但是有一个问题我有点疑惑，我发现在train_llava15.py文件中vision_tower似乎是带有梯度的，这样训练出来的模型视觉模块的参数应该是训练过的，但是在我下载的huggingface上rlaif-v 7b的参数到本地后运行chat.py文件会报以下的警告，似乎是视觉模块的参数没有被加载进来，并且还需要再从huggingface上下载clip-vit的参数： ```bash Some weights of the model checkpoint at ./models/RLAIF-V-7B were not used when initializing LlavaLlamaForCausalLM: ['model.vision_tower.vision_tower.vision_model.encoder.layers.7.mlp.fc2.weight', 'model.vision_tower.vision_tower.vision_model.encoder.layers.7.layer_norm1.bias', 'model.vision_tower.vision_tower.vision_model.encoder.layers.20.self_attn.v_proj.bias', 'model.vision_tower.vision_tower.vision_model.encoder.layers.20.self_attn.out_proj.weight', 'model.vision_tower.vision_tower.vision_model.encoder.layers.6.layer_norm2.bias', 'model.vision_tower.vision_tower.vision_model.encoder.layers.9.mlp.fc1.bias', 'model.vision_tower.vision_tower.vision_model.encoder.layers.13.mlp.fc2.bias', 'model.vision_tower.vision_tower.vision_model.encoder.layers.14.layer_norm1.bias', 'model.vision_tower.vision_tower.vision_model.encoder.layers.21.mlp.fc2.weight', 'model.vision_tower.vision_tower.vision_model.encoder.layers.5.mlp.fc2.weight', 'model.vision_tower.vision_tower.vision_model.encoder.layers.22.self_attn.k_proj.weight', 'model.vision_tower.vision_tower.vision_model.encoder.layers.0.layer_norm1.bias',...