'hf' model "Mixtral-8x7b" unable to load.

[vegansquirrel]

While loading the model "ehartford/dolphin-2.5-mixtral-8x7b" into the Lida llm, I am getting the error shown in the image. I am loading it in a CPU. The llama models are loading fine but the new Mixtral is facing a problem.

I would highly appreciate a detailed response and solution.

[victordibia]

The mixtral models have not been tested with lida/llmx.

I would recommend the following.

• load the model using a tool like vllm which supports mixtral
• vllm provides an openai compatible api
• use the openai llm approach, but point it to the vllm server.

# spin up a vllm endpoint 

from lida import Manager, TextGenerationConfig , llm

model_name =  # mixtral model 
model_details = [{'name': model_name, 'max_tokens': 2596, 'model': {'provider': 'openai', 'parameters': {'model': model_name}}}]

# assuming your vllm endpoint is running on localhost:8000
text_gen = llm(provider="openai",  api_base="http://localhost:8000/v1", api_key="EMPTY", models=model_details)
lida = Manager(text_gen = text_gen)

[anshumankmr]

On a somewhat related point, is there a way to use HuggingFace dedicated endpoint here?