xtuner

xtuner copied to clipboard

LLaVAModel finetuning with llava-phi-3-mini-xtuner

1

``` llava_name_or_path='hub/llava-phi-3-mini-xtuner' model = dict( type=LLaVAModel, freeze_llm=True, freeze_visual_encoder=True, pretrained_pth=llava_name_or_path, llm=dict( type=AutoModelForCausalLM.from_pretrained, pretrained_model_name_or_path=llm_name_or_path, attn_implementation="eager", trust_remote_code=True), visual_encoder=dict( type=CLIPVisionModel.from_pretrained, pretrained_model_name_or_path=visual_encoder_name_or_path)) ``` when loadding model, the following error occurs: ``` [rank7]: obj = obj_cls(**args)...

shockjiang

HIT-cwh

I need a LLM in the GGUF format. I have noticed that llama.cpp can convert files from the HF format to the GGUF format, and it can also convert files...

LiyanJin

The sequence parallel is open when I don't use it.

4

## version `05/09 21:16:21 - mmengine - INFO - 0.1.18` ## how to reproduce `CUDA_VISIBLE_DEVICES=4,5,6,7 NPROC_PER_NODE=4 xtuner train qwen1_5_0_5b_chat_qlora_alpaca_e3` ## log I only change the batch_size to 4 in config...

amulil

支持自定义视觉编码器么（llava-llama3）?

3

支持自定义视觉编码器么（llava-llama3）? 例如将clip换成siglip? 该如何实现？哪些代码需要修改？

Yanllan

failed to convert convert llava-llama3 model to hf format

9

In the project: https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llava/llama3_8b_instruct_clip_vit_large_p14_336, it gives an examples how to convert llava-llama3 model to hf format: ` python ./convert_xtuner_weights_to_hf.py --text_model_id ./iter_39620_xtuner --vision_model_id ./iter_39620_visual_encoder --projector_weight ./iter_39620_xtuner/projector/model.safetensors --save_path ./iter_39620_llava` I follow it...

Jason8Kang

PROMPT_TEMPLATE.llama2_chat效果下降

7

llama2_7b_qlora_alpaca_enzh_e3.py作为模板，qlora微调gsm8k，修改PROMPT_TEMPLATE.llama2_chat为PROMPT_TEMPLATE.llama3_chat，acc从62下降到28，可能是什么原因导致的？

dongjiancheng77

- [x] get_started - [x] training - [x] accelerate - [ ] InternEVO

pppppM

HIT-cwh

llava-llama3-8b llm+llm adapter merge error

2

- description i follow the turial for llama3 ft :[https://github.com/SmartFlowAI/Llama3-Tutorial/blob/main/docs/llava.md](https://github.com/SmartFlowAI/Llama3-Tutorial/blob/main/docs/llava.md) i use `llava_llama3_8b_instruct_qlora_clip_vit_large_p14_336_lora_e1_finetune.py` to fineture on my dataset, and want to get a llava-llama38b multimodal model on my datasets. after...

ztfmars