inference icon indicating copy to clipboard operation
inference copied to clipboard

Model link for Llama-3-instruct 70B is wrong

Open slobentanzer opened this issue 2 months ago • 1 comments

Describe the bug

Trying to use the builtin Llama-3-instruct 70B (gguf) from the Python client (.launch_model()) fails with this Traceback (excerpt):

Traceback (most recent call last):
  File "/Users/slobentanzer/miniforge3/envs/xinference/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 286, in hf_raise_for_status
    response.raise_for_status()
  File "/Users/slobentanzer/miniforge3/envs/xinference/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/lmstudio-community/Meta-Llama-3-70B-Instruct-GGUF/resolve/main/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf

I guess it should be /Meta-Llama-3-70B-Instruct-Q4_K_M.gguf in the file path, right?

I did (xinference 0.3.10):

client.launch_model(
            model_name="llama-3-instruct",
            model_size_in_billions=70,
            model_format="ggufv2",
            quantization="Q4_K_M",
        )

I don't think other details matter in this issue, but feel free to correct me in that. :)

slobentanzer avatar Apr 24 '24 09:04 slobentanzer

@slobentanzer Thanks for reporting. Fix this by PR #1370 for the next release.

ChengjieLi28 avatar Apr 24 '24 10:04 ChengjieLi28