InternVL icon indicating copy to clipboard operation
InternVL copied to clipboard
Published 3 months ago • verified publisher icon OpenGVLab

4-bit version

Open lihan opened this issue 1 year ago • 5 comments

Is there a plan to release a normalized 4-bit version?

lihan avatar Apr 25 '24 11:04 lihan

Hello, thank you for your interest. We have successfully implemented the 8-bit version; however, we have encountered some issues with 4-bit quantization. We will resolve these as soon as possible and release the quantized weights. Thank you for your patience.

czczup avatar Apr 26 '24 17:04 czczup

@czczup our team LMDeploy is working on quantizing VLMs into 4bit by AWQ The relative PR is https://github.com/InternLM/lmdeploy/pull/1553 Can we collaborate on it?

lvhan028 avatar May 07 '24 13:05 lvhan028

@czczup our team LMDeploy is working on quantizing VLMs into 4bit by AWQ The relative PR is InternLM/lmdeploy#1553 Can we collaborate on it?

That sounds great! I'm thrilled at the prospect of collaborating on this initiative.

czczup avatar May 08 '24 16:05 czczup

Hi @czczup @lvhan028.

How about the bit 4 going?

Thank you for your good work.

tairen99 avatar May 23 '24 18:05 tairen99

We are working on it. https://github.com/InternLM/lmdeploy/pull/1553 This PR is going to be merged soon. We'll release lmdeploy v0.4.2 next week

lvhan028 avatar May 24 '24 03:05 lvhan028

v0.4.2 has been released. You can have a try

lvhan028 avatar May 27 '24 09:05 lvhan028

The 4-bit version of the model has been released. Check it out at OpenGVLab/InternVL-Chat-V1-5-AWQ. Thanks to the lmdeploy team for their support with model quantization.

I'm closing this issue now, but if you encounter any problems, please don't hesitate to reopen it.

czczup avatar May 30 '24 12:05 czczup