Train LLM decoder

[bluesky314]

There is a way to train the vision encoder in the config https://github.com/Vision-CAIR/MiniGPT-4/blob/main/minigpt4/configs/models/minigpt4.yaml#L16 by:

freeze_vit: True freeze_qformer: True

Is there a way to finetune the llm decoder also with this library

[bluesky314]

@Zhwt @TsuTikgiau @junchen14 @lx709 Can anyone help? Need for research

[joaopedrosdmm]

I'm also looking for a way to change the decoder. Is there a way to get the weights for only the progression layers?

[dirtycomputer]

Neeeeeed!