intel-extension-for-transformers Llava Models

Llava Models

Open choochtech opened this issue 1 year ago • 7 comments

trafficstars

Does the Llava part work ?

https://github.com/intel/intel-extension-for-transformers/tree/main/intel_extension_for_transformers/transformers/modeling/llava_models

If so are they optimized for Intel Device and are there any examples ?

Thanks for building this library. I have seen the token generation performance to be very good compared to OpenVino.

Great work !

Thanks

Dec 04 '23 03:12 choochtech

Thanks for your usage!!! Unfortunately, we are not support LLaVA for now. But we may consider to support it in future, please stay tune.

Regrads, Bo

Dec 04 '23 03:12 a32543254

It's for multi-model training, but optimization is WIP.

Dec 04 '23 07:12 kevinintel

@kevinintel how do you do the optimization for the llava model and use it ?

Dec 04 '23 08:12 choochtech

Someone tried low-bits for llava: https://arxiv.org/pdf/2306.00978.pdf and we will try to quantize it.

Dec 04 '23 08:12 kevinintel

Thanks @kevinintel

Dec 04 '23 22:12 choochtech

Hi, support for quantization of multimodal models is currently planned, and any updates will be communicated here.

Mar 07 '24 05:03 WeiweiZhang1

we can optimize llava in https://github.com/intel/neural-compressor/pull/1797 will add examples

Jun 05 '24 08:06 kevinintel