Yu-won Lee
Yu-won Lee
I think the training code isn't open yet for the LLaVA-NEXT.
Basically it's not quite different but, the training code might need to changed a bit. I'll try for supporting the model later on. I'm trying to work on Preference optimization...
I was looking into torchtune and found ``` def compile_model( model: Union[TransformerDecoder, DeepFusionModel], verbose: bool = True, ) -> None: """ Utility to compile a transformer model inplace. On PyTorch...
The dataset required for the grounding should be a bit strict. It should be a json formatted string with coordinates so, it shouldn't be that hard. I'll make an option...
Well, I haven't use the eval with huggingface trainier. I'll give it a try.
Sorry for the issue, looke like you are using QLoRA right? I found that liger kernel does not support QLoRA. I will make an option to not use liger kernel.
If you want to use files like "sample.mp4" or some other files like videos, you could just pass the video directories. But the tag in the text should be `\n`
@lucasxu777 I've updated the code for turninig off liger kernel. Maybe it should work now.
@lucasxu777 The error is caused by the transformers version. Please download the version I've wrote it in the README.
`pip install git+https://github.com/huggingface/transformers.git@9d2056f12b66e64978f78a2dcb023f65b2be2108 accelerate` This is the correct version I wrote it in the README. The code is not based on the latest release version yet. It caused some other...