video-salmonn vicuna question

Open HeChengHui opened this issue 1 year ago • 2 comments

@BriansIDP Thank you for your work.

what is the vram requirement to run inference? i am having OOM using lmsys/vicuna-13b, but lmsys/vicuna-7b is giving me size mismatch error.
Or am i using the wrong model?

Nov 29 '24 06:11 HeChengHui

Thank you for the question. Video-SALMONN is trained with vicuna-13b so the input dimension for the 7b model would not match the Q-Former output of video-SALMONN. It would be helpful to try quantization (with a bit of performance loss).

Nov 29 '24 07:11 BriansIDP

@BriansIDP does that mean i can use something like TheBloke/vicuna-13B-v1.5-16K-AWQ by just setting it in the config?

Nov 29 '24 08:11 HeChengHui