Lyu Han

PyTorch has to be installed.

The architecture of InternLM2 is different from InternLM. The former adopts GQA and has no attention bias. Unlike other GQA models, it packed q, k, v weights into one tensor....

Hi, @hyhhyh402 have you submitted a PR?

Based on FasterTransformer, we have implemented an efficient inference engine - [TurboMind](https://github.com/InternLM/lmdeploy#introduction) - It supports llama and llama-2 - It modeled the inference of a conversational LLM as a persistently...

> > Based on FasterTransformer, we have implemented an efficient inference engine - [TurboMind](https://github.com/InternLM/lmdeploy#introduction) > > > > * It supports llama and llama-2 > > * It modeled the...

GQA has been supported by [LMDeploy](https://github.com/InternLM/lmdeploy), which is developed based on FasterTransformer

Only 70B model is GQA.

@AnyangAngus GQA in LMDeploy/TurboMind doesn't distinguish between 7B, 13B, or 70B models. But as far as I know, [llama-2-7b/13b](https://huggingface.co/meta-llama/Llama-2-70b) doesn't have GQA block 

LMDeploy v0.4.1 can help deploying InternVL. This is a guide https://github.com/OpenGVLab/InternVL/pull/152

When deploying VLMs on GPUs with limited memory, such as the V100 and T4, it is typically essential to utilize multiple GPUs because of the large size of the model....