Tian Tang

Import Error, not compatible with transformer package

1

File "/home/Qwen/infer.py", line 63, in import mii File "/usr/local/lib/python3.10/dist-packages/mii/__init__.py", line 7, in from .legacy import MIIServer, MIIClient, mii_query_handle, deploy, terminate, DeploymentType, TaskType, aml_output_path, MIIConfig, ModelConfig, get_supported_models File "/usr/local/lib/python3.10/dist-packages/mii/legacy/__init__.py", line 7,...

Bug: Failed to run qwen2-57b-a14b-instruct-fp16.

3

### What happened? I am trying to run Qwen2-57B-A14B-instruct, and I used llama-gguf-split to merge the gguf files from [Qwen/Qwen2-57B-A14B-Instruct-GGUF](https://huggingface.co/Qwen/Qwen2-57B-A14B-Instruct-GGUF). But it's aborted with `terminate called after throwing an instance...

non-persistent example doesn't work on Mixtral-8*7B-v0.1

1

``` import mii pipe = mii.pipeline("/data/mixtral/Mixtral-8x7B-v0.1") response = pipe(["DeepSpeed is"], max_new_tokens=128) print(response) ``` Run this by 'deepspeed --num_gpus=4' will report following error for each rank: [rank0]: response = pipe(["DeepSpeed is"],...