LocalAI
LocalAI copied to clipboard
`/models/available` broken for Huggingface gallery
LocalAI version:
OK:
-
local-ai-avx2-Linux-x86_64-1.40.0
-
local-ai-avx2-Linux-x86_64-2.0.0
-
local-ai-avx2-Linux-x86_64-2.8.0
-
local-ai-avx2-Linux-x86_64-2.8.2
-
local-ai-avx2-Linux-x86_64-2.9.0
Broken:
-
local-ai-cuda12-Linux-x86_64-2.10.1
-
local-ai-cuda12-Linux-x86_64-2.11.0
-
local-ai-cuda12-Linux-x86_64-2.12.0
-
local-ai-cuda12-Linux-x86_64-2.12.4
-
local-ai-avx2-Linux-x86_64-2.12.4
-
local-ai-cuda12-Linux-x86_64-2.13.0
Environment, CPU architecture, OS, and Version:
$ uname -a
Linux 2d083df80d05 6.5.0-1018-oem #19-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar 14 21:40:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Describe the bug
Cannot get Huggingface gallery in recent versions:
-
GALLERIES='[{"url": "github:go-skynet/model-gallery/huggingface.yaml","name":"huggingface"}]'
"Index" model gallery works:
-
GALLERIES='[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}]'
To Reproduce
- Set Huggingface gallery:
-
export GALLERIES='[{"url": "github:go-skynet/model-gallery/huggingface.yaml","name":"huggingface"}]'
-
- Start LocalAI binary
- e.g.
./local-ai-avx2-Linux-x86_64-2.12.4
- e.g.
- Try to view Gallery in Browser: http://localhost:8080/models/available
- up to
v2.9.0
:- Log:
200 - GET /models/available
- Browser: JSON result is shown
- Log:
- from
v2.10.1
tov2.12.4
:- Log:
500 - GET /models/available
- Browser:
{"error":{"code":500,"message":"json: unsupported type: map[interface {}]interface {}","type":""}}
- Log:
- up to
Expected behavior
JSON for Huggingface gallery should be returned as in versions v2.9.0
and prior. Or as it is returned for standard 'model-gallery'.
Logs
Debug-Log is 15MB, so not attaching it here.
There is a really long line with what looks like a hex array:
- Starts with:
10:32AM DBG Models found from galleries: [0xc00032a9c0 0xc00032ad00 0xc00032add0 0xc00032b040
- Ends with:
0xc0090a2f70 0xc0090a3040 0xc0090a3110 0xc0090a31e0 0xc0090a32b0 0xc0090a3380 0xc0090a3450 0xc0090a3520 0xc0090a35f0 0xc0090a36c0 0xc0090a3790 0xc0090a3860 0xc0090a3930 0xc0090a3a00]
Excerpt with Model found
messages:
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_12000_1e_thebloke_mistralic-7b-1-gptq_lora_causal_lm__instruct_en-vi_12000_1e__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_12000_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:Instruct_en-vi_12000_1e/adapter_model.bin]] AdditionalFiles:[{Filename:Instruct_en-vi_12000_1e/adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_12000_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/Instruct_en-vi_12000_1e/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_14000_1e_b60_lr2e-4_thebloke_mistralic-7b-1-gptq_lora_causal_lm__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_14000_1e_b60_lr2e-4_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:adapter_model.bin]] AdditionalFiles:[{Filename:adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_14000_1e_b60_lr2e-4_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_14000_1e_b60_lr2e-4_thebloke_mistralic-7b-1-gptq_lora_causal_lm__instruct_en-vi_14000_1e_b60_lr2e-4__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_14000_1e_b60_lr2e-4_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:Instruct_en-vi_14000_1e_b60_lr2e-4/adapter_model.bin]] AdditionalFiles:[{Filename:Instruct_en-vi_14000_1e_b60_lr2e-4/adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_14000_1e_b60_lr2e-4_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/Instruct_en-vi_14000_1e_b60_lr2e-4/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_14500_1e_thebloke_mistralic-7b-1-gptq_lora_causal_lm__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_14500_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:adapter_model.bin]] AdditionalFiles:[{Filename:adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_14500_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_14500_1e_thebloke_mistralic-7b-1-gptq_lora_causal_lm__instruct_en-vi_14500_1e__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_14500_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:Instruct_en-vi_14500_1e/adapter_model.bin]] AdditionalFiles:[{Filename:Instruct_en-vi_14500_1e/adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_14500_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/Instruct_en-vi_14500_1e/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_7500_1epoch_thebloke_mistralic-7b-1-gptq_lora_causal_lm__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_7500_1epoch_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:adapter_model.bin]] AdditionalFiles:[{Filename:adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_7500_1epoch_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_7500_1epoch_thebloke_mistralic-7b-1-gptq_lora_causal_lm__instruct_en-vi_7500_1epoch__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_7500_1epoch_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:Instruct_en-vi_7500_1epoch/adapter_model.bin]] AdditionalFiles:[{Filename:Instruct_en-vi_7500_1epoch/adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_7500_1epoch_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/Instruct_en-vi_7500_1epoch/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_8000_1e_thebloke_mistralic-7b-1-gptq_lora_causal_lm__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_8000_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:adapter_model.bin]] AdditionalFiles:[{Filename:adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_8000_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
Additional context
Running from release binaries on a rented VM. Binaries downloaded from GitHub and renamed to include version number.
It seems like the issue you're facing is related to a compatibility problem between the Hugging Face transformers
library version and the models used. The error messages suggest that the models found in the galleries are not compatible with the version of transformers
you are using.
The models mentioned are likely SF-T5 models, which require a specific version of the transformers
library to work properly. You mentioned using transformers==4.6.0
, but these models may require a different version. Specifically, the error suggests that you may need to use transformers==4.5.0
.
To resolve this issue, you should update the transformers
library in your environment to version 4.5.0. You can do this with pip by running the following command:
pip install transformers==4.5.0
After installing the correct version of the transformers
library, try running your code again to see if the issue is resolved. If you continue to encounter problems, make sure that the models you are trying to load are compatible not only with the transformers
library version but also with the Python version you are using.
This is not about running any model.
Just trying to list the gallery as described in https://localai.io/models/#list-models
Repro:
- Start LocalAI
- List gallery
- Error
$ curl http://localhost:8080/models/available
{"error":{"code":500,"message":"json: unsupported type: map[interface {}]interface {}","type":""}}
Maybe the full debug log will help:
The only request to LocalAI after startup is listing gallery models:
10:08AM INF core/startup process completed!
10:08AM DBG No configuration file found at /tmp/localai/upload/uploadedFiles.json
10:08AM DBG No configuration file found at /tmp/localai/config/assistants.json
10:08AM DBG No configuration file found at /tmp/localai/config/assistantsFile.json
10:08AM DBG Listing models from galleries: [{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface}]
I'm also able to replicate this, model-gallery works, hugging face does not.
I'm also able to replicate this, model-gallery works, hugging face does not.
Another confirmation here. I can get a model list using only the main gallery but not using only huggingface gallery, using both at the same time also fails.
Currently running v2.12.4 (0004ec8be3ca150ce6d8b79f2991bfe3a9dc65ad)
with the AIO docker image latest-aio-gpu-nvidia-cuda-11 a6ffa9444dd5
With DEBUG=true I can see that LocalAI reads and parses entire huggingface.yaml
, last lines from the log matches with last lines from the YAML file), but finally fails to build the model list.
$ curl $LOCAL_AI/models/available
{"error":{"code":500,"message":"json: unsupported type: map[interface {}]interface {}","type":""}}
Relevant log lines:
localai | 8:00AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:zhaijunxiao__pygmalion-13b-ggml__ggml-model-q4_0.bin Description: License: URLs:[https://huggingface.co/zhaijunxiao/Pygmalion-13B-ggml] Icon: Tags:[region:us] ConfigFile:map[] Overrides:map[parameters:map[model:ggml-model-q4_0.bin]] AdditionalFiles:[{Filename:ggml-model-q4_0.bin SHA256: URI:https://huggingface.co/zhaijunxiao/Pygmalion-13B-ggml/resolve/main/ggml-model-q4_0.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:model-gallery} Installed:false}
localai | 8:00AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:zinccat__santacoder-ggml-quantized__gpt_bigcode-santacoder-ggml-q4_1.bin Description: License:bigcode-openrail-m URLs:[https://huggingface.co/zinccat/santacoder-ggml-quantized] Icon: Tags:[license:bigcode-openrail-m region:us] ConfigFile:map[] Overrides:map[parameters:map[model:gpt_bigcode-santacoder-ggml-q4_1.bin]] AdditionalFiles:[{Filename:gpt_bigcode-santacoder-ggml-q4_1.bin SHA256: URI:https://huggingface.co/zinccat/santacoder-ggml-quantized/resolve/main/gpt_bigcode-santacoder-ggml-q4_1.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:model-gallery} Installed:false}
Same error on my side too with huggingface, using latest-aio-gpu-nvidia-cuda-12
docker image