fastchat-t5-3b-v1.0 on macOS?

[fdstevex]

Hi, I'm trying to use fastchat-t5-3b-v1.0 on macOS following the instructions in the README.

Simply run the line below to start chatting. It will automatically download the weights from a Hugging Face repo.
python3 -m fastchat.serve.cli --model-path lmsys/fastchat-t5-3b-v1.0

When I do this, I get the error "Torch not compiled with CUDA enabled".

I added --device mps --load-8bit to run it as

python3 -m fastchat.serve.cli --model-path lmsys/fastchat-t5-3b-v1.0 --device mps --load-8bit

And that gives a different error: "Unrecognized configuration class <class 'transformers.models.t5.configuration_t5.T5Config'> for this kind of AutoModel: AutoModelForCausalLM".

Is this usable without CUDA?

[Kimiko-AI]

Something wrong with the code here, not your device

[sqzhang-jeremy]

Macbook M1. I met the same problem. It seemed that the function AutoModelForCausalLM was not supported fastchat-t5-3b-v1.0. ValueError: Unrecognized configuration class <class 'transformers.models.t5.configuration_t5.T5Config'> for this kind of AutoModel: AutoModelForCausalLM. Model type should be one of BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, CodeGenConfig, CpmAntConfig, CTRLConfig, Data2VecTextConfig, ElectraConfig, ErnieConfig, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, LlamaConfig, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MvpConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, Speech2Text2Config, TransfoXLConfig, TrOCRConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig.

[EugeneVanchugov]

Have the following issue on M2 Pro:

python3 -m fastchat.serve.cli --model-path lmsys/fastchat-t5-3b-v1.0 --device mps --style rich
Human:
hi

Assistant:
loc("mps_clamp"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/97f6331a-ba75-11ed-a4bc-863efbbaf80d/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":268:0)): error: input types 'tensor<1x600x2048xf16>' and 'tensor<1x1x1xf32>' are not broadcast compatible
LLVM ERROR: Failed to infer result type(s).
fish: Job 1, 'python3 -m fastchat.serve.cli -…' terminated by signal SIGABRT (Abort)

[zhisbug]

@DachengLi1 is looking into it

[salym]

it seems that Pytorch has issues with mps.

[codeisnotcode]

Same issues on Windows 10.

[willie]

Glad I found this issue, I'm having the same problem.

[SCzfdf]

我使用下面命令可以运行 python3 -m fastchat.serve.cli --model-path lmsys/fastchat-t5-3b-v1.0 --device cpu

[fdstevex]

It works on cpu, but it's very slow. Should this work on mps on devices with an M1/M2?

[lovecactus]

"--device cpu" works for me as well, extremely slow though. I hope there is a perfect solution for MacBook in the future.

[salym]

@SCzfdf

用CPU执行的速度太慢

[kamroot]

I am having the same issue. Is there an issue on PyTorch repo that we can track and bump up in priority?

[Suji04]

Having the same issue with M1 Pro 16 GB.

If I use --device mps error: input types 'tensor<1x601x2048xf16>' and 'tensor<1x1x1xf32>' are not broadcast compatible LLVM ERROR: Failed to infer result type(s).

If I use --device mps --load-8bit ValueError: Unrecognized configuration class <class 'transformers.models.t5.configuration_t5.T5Config'> for this kind of AutoModel: AutoModelForCausalLM. Model type should be one of BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, CodeGenConfig, CpmAntConfig, CTRLConfig, Data2VecTextConfig, ElectraConfig, ErnieConfig, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, LlamaConfig, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MvpConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, Speech2Text2Config, TransfoXLConfig, TrOCRConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig.

Works on cpu.

[gkozyrev]

Same thing on my Macbook Pro 13 M1

[ExpressGit]

python3 -m fastchat.serve.cli --model-path /Users/username/Documents/workspace/fastchatt53bv1.0 --device cpu

same problem

device:M1 macbook pro

I use --device cpu ValueError: Unrecognized configuration class <class 'transformers.models.t5.configuration_t5.T5Config'> for this kind of AutoModel: AutoModelForCausalLM. Model type should be one of BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, CodeGenConfig, CpmAntConfig, CTRLConfig, Data2VecTextConfig, ElectraConfig, ErnieConfig, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, LlamaConfig, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MvpConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, Speech2Text2Config, TransfoXLConfig, TrOCRConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig.