Open-LLaVA-NeXT

Open-LLaVA-NeXT copied to clipboard

I wanted to know why the `prepare_input_labels_for_multimodal` function in `llava_arch.py` is designed to [throw an exception ](https://github.com/xiaoachen98/Open-LLaVA-NeXT/blob/master/llava/model/llava_arch.py#L205)during pretraining if the `mm_use_im_start_end` option is enabled: ```python # TODO: image start /...

Ali2500

loss curve of llava-next-llama3

2

Thanks for your great work! I'm wondering if u can share the loss curve for training llava-next-llama3? I've observed some different behaviour compared to training llava-next-vicuna-7b. I'm wondering if it's...

simplelifetime

Would you plan to adapt it to qwen2-7B?

3

Nastu-Ho

loss curve of SFT on vicuna-7b

6

Hi, I got a training curve like this, is it normal? Do you mind sharing your trainer_state.json? thx!

Xiaohui9607

What is the difference between the anyres implementation in this project and the S2-based implementation in llava?

LIO-H-ZEN

I conducted experiments on lllama3-8b. This repository can achieve comparable performance to lllama-next-llama3-8B on MME (1613 vs. 1603), but not on MMMU (37.4 vs. 41.7). Do you have any ideas?

LIO-H-ZEN

Is the param "self.image_newline" only tuned in SFT?

1

I found that the 'self.image_newline' in LlavaMetaModel was not tuned during the pre-training stage. Have you tried enabling this parameter during pre-training?

huangrh99

Hi, thank you for sharing this repo. I've been working on preparing the SFT dataset, but I've encountered an issue with collecting all the necessary data. It seems that the...

mmderakhshani