mistral and mixtral inference[BUG]

[Akash08naik]

trafficstars

Describe the bugMistral and Mixtral models not able to infer When i give the name of the model as i do for other models in case of mistral there is a key error from the configuration_auto.py file in llm_analysis module. This is because there is no key with mistral in the config_map.

So could you also add all the models from hugging face which are not yet defined!!

[cli99]

Please try updating the transformers library,pip install -U transformers. If not using a local model json file as the model_config, llm-analysis will rely on transformers to find the corresponding model configuration on the hf hub, meaning information of newer models only exist after certain version of the transformers library.

[Akash08naik]

yeah that worked . And also could you tell how to use this for big models with more parameters . like in my case when i try to load a 34B billion parameter the error i get is AssertionError. assert memory_left>0 . The model is too large to fit in total GPU memory.

[cli99]

there are multiple ways to address OOM: use GPUs with larger memory (e.g. 80gb); use more GPUs and apply tensor parallelism or expert parallelism (if it's a moe model); use quantization (use different dtype rather than the default "w16a16e16").

I will add an mistral example soon

[Akash08naik]

I have made changes in the code in place of memory and have given more space. Now the model is running . But will this change alter the results from normal?