Fail to load fixie-ai/ultravox-v0_4_1-llama-3_1-70b with device_map 'auto'

[MatthewCYM]

Hi,

When I load the model into 4 gpus with model parallelism:

transformers.pipeline(model='fixie-ai/ultravox-v0_4_1-llama-3_1-70b', trust_remote_code=True, device_map='auto')

It gives the below error:

ValueError: weight is on the meta device, we need a `value` to put in on 0.

[Madoshakalaka]

facing same problem here with two 48GB A6000 GPUs

[farzadab]

Hi there,

I've taken a look before and I wasn't able to get any good performance (if at all) out of it, so currently for 70B inference we use VLLM instead.

Can I ask why you want to do inference with the 70B model? For example, do you care about performance or is it just to test the model out.

[Madoshakalaka]

thanks! We are just poking around, checking if the 70B model provides better output. After some testing, it seems that the 8B one is already very capable though. Actually in out application, a timely response from the LLM is crucial, so if 70B is slower in that regard it's indeed a worse choice. Thanks for helping out!