7AtAri

LLaVa-NeXT/Fine_tune_LLaVaNeXT_on_a_custom_dataset_(with_PyTorch_Lightning).ipynb fails at training

1

--------------------------------------------------------------------------- ValueError Traceback (most recent call last) [](https://localhost:8080/#) in () 17 ) 18 ---> 19 trainer.fit(model_module) 24 frames [/usr/local/lib/python3.10/dist-packages/transformers/models/llava_next/modeling_llava_next.py](https://localhost:8080/#) in _merge_input_ids_with_image_features(self, image_features, feature_lens, inputs_embeds, input_ids, attention_mask, position_ids, labels, image_token_index, ignore_index)...

Fine tuning with regression head

1

I adapted your fine tuning notebook to a regression task. Unfortunately the model gets unstable during training and only returns NANs as hidden states to my regression head. Also I...