inference icon indicating copy to clipboard operation
inference copied to clipboard
Published 1 month ago • verified publisher icon xorbitsai

最新的cpu版本，模型起不来

Open luofeng0603 opened this issue 11 months ago • 13 comments

System Info / 系統信息

尝试了 registry.cn-hangzhou.aliyuncs.com/xprobe_xinference/xinference:latest-cpu (镜像id： 7892712e1798) 和 xprobe/xinference:nightly-main-cpu 容器启动没问题，但是加载模型的时候报错： (base) root@31a37f59a02e:/# xinference launch --model-name bge-large-zh-v1.5 --model-type embedding --n-gpu none Launch model name: bge-large-zh-v1.5 with kwargs: {} Traceback (most recent call last): File "/opt/conda/bin/xinference", line 8, in sys.exit(cli()) ^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1161, in call return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1082, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1697, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1443, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 788, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/xinference/deploy/cmdline.py", line 908, in model_launch model_uid = client.launch_model( ^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/xinference/client/restful/restful_client.py", line 999, in launch_model raise RuntimeError( RuntimeError: Failed to launch model, detail: [address=0.0.0.0:43015, pid=523] Failed to import transformers.models.timm_wrapper.configuration_timm_wrapper because of the following error (look up to see its traceback): cannot import name 'ImageNetInfo' from 'timm.data' (/opt/conda/lib/python3.11/site-packages/timm/data/init.py)

应该是 timm版本问题，默认我看安装的是 0.6.7，然后我尝试强制升级 timm pip install --upgrade timm ，它给我升级到了 1.0.14 ，但是提示我 跟 controlnet-aux 0.0.9 不兼容了，这个要求 timm的版本是<=0.6.7 但是 上面的模型加载可以正常执行了

这个问题还是请官方修复一下吧，我也不知道 我强制升级 timm会不会导致其他的问题

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

  • [x] docker / docker
  • [ ] pip install / 通过 pip install 安装
  • [ ] installation from source / 从源码安装

Version info / 版本信息

registry.cn-hangzhou.aliyuncs.com/xprobe_xinference/xinference:latest-cpu

The command used to start Xinference / 用以启动 xinference 的命令

docker run --name xinference -d -p 11021:9997 -v /data/xinference:/data -e XINFERENCE_MODEL_SRC='modelscope' -e XINFERENCE_HOME=/data --restart always registry.cn-hangzhou.aliyuncs.com/xprobe_xinference/xinference:latest-cpu xinference-local -H 0.0.0.0 --auth-config=/data/config/config.conf --log-level debug

Reproduction / 复现过程

  1. 启动容器
  2. 进入容器 执行命令加载已缓存的模型，就报错了，或者在 web管理台中 加载模型也会有一样的报错

Expected behavior / 期待表现

希望修复此问题

luofeng0603 avatar Feb 06 '25 07:02 luofeng0603

This issue is stale because it has been open for 7 days with no activity.

github-actions[bot] avatar Feb 13 '25 19:02 github-actions[bot]

same issue with model jina-embeddings-v2-base-zh and jina-embeddings-v3 xinference cpu docker image version is v1.2.2-cpu

he-ding avatar Feb 17 '25 03:02 he-ding

/bge-m3

yuanbaobaoo avatar Feb 18 '25 08:02 yuanbaobaoo

Encounter same issue on model : multilingual-e5-large-instruct

TimeAshore avatar Feb 19 '25 07:02 TimeAshore

的确更新如下包可以解决。

docker exec -it xinference bash pip install "timm>=0.9.16" controlnet-aux

darrentangdt avatar Feb 22 '25 06:02 darrentangdt

I've tried to run this framework with 'xinference-local --host 0.0.0.0 --port 9997' instead of 'docker', it worked.

TimeAshore avatar Feb 24 '25 02:02 TimeAshore

用官方的docker-cpu版本，一堆问题，搞了一个星期放弃了，后面改成空白docker容器内部安装conda和xinference，反而能正常跑模型了，记得别用官方docker-cpu，就是坑。。。

hpx52 avatar Mar 11 '25 09:03 hpx52

官方最新版本：1.4.1-cpu版本容器无法启动： 启动命令： docker run -d --name xinference141 -v /data/xinference:/xinference -e XINFERENCE_HOME=/xinference -p 9997:9997 xprobe/xinference:v1.4.1-cpu xinference-local -H 0.0.0.0 --log-level debug 2cec77fd76dda267cf8fdbc0f23b81d879599ec73e6c8a5fcb1cc715a5ef2a83 容器日志： `2025-04-08 06:46:31,510 xinference.core.supervisor 24 INFO Xinference supervisor 0.0.0.0:44110 started 2025-04-08 06:46:31,540 xinference.core.worker 24 INFO Starting metrics export server at 0.0.0.0:None 2025-04-08 06:46:31,543 xinference.core.worker 24 INFO Checking metrics export server... 2025-04-08 06:46:33,908 xinference.core.worker 24 INFO Metrics server is started at: http://0.0.0.0:37593 2025-04-08 06:46:33,909 xinference.core.worker 24 INFO Purge cache directory: /xinference/cache 2025-04-08 06:46:33,911 xinference.core.supervisor 24 DEBUG [request 35f63ec2-1445-11f0-87c6-0242ac110002] Enter add_worker, args: <xinference.core.supervisor.SupervisorActor object at 0x7f30c2b5cad0>,0.0.0.0:44110, kwargs: 2025-04-08 06:46:33,912 xinference.core.supervisor 24 DEBUG Worker 0.0.0.0:44110 has been added successfully 2025-04-08 06:46:33,912 xinference.core.supervisor 24 DEBUG [request 35f63ec2-1445-11f0-87c6-0242ac110002] Leave add_worker, elapsed time: 0 s 2025-04-08 06:46:33,912 xinference.core.worker 24 INFO Connected to supervisor as a fresh worker 2025-04-08 06:46:33,934 xinference.core.worker 24 INFO Xinference worker 0.0.0.0:44110 started 2025-04-08 06:46:33,938 xinference.core.supervisor 24 DEBUG Worker 0.0.0.0:44110 resources: {'cpu': ResourceStatus(usage=0.0, total=8, memory_used=3172597760, memory_available=12951879680, memory_total=16637235200)} 2025-04-08 06:46:36,488 xinference.core.supervisor 24 DEBUG Enter get_status, args: <xinference.core.supervisor.SupervisorActor object at 0x7f30c2b5cad0>, kwargs: 2025-04-08 06:46:36,488 xinference.core.supervisor 24 DEBUG Leave get_status, elapsed time: 0 s 2025-04-08 06:46:39,322 xinference.api.restful_api 1 INFO Starting Xinference at endpoint: http://0.0.0.0:9997 /opt/conda/lib/python3.11/site-packages/xinference/api/restful_api.py:845: UserWarning: Xinference ui is not built at expected directory: /opt/conda/lib/python3.11/site-packages/xinference/web/ui/build/ To resolve this warning, navigate to /opt/conda/lib/python3.11/site-packages/xinference/web/ui/ And build the Xinference ui by running "npm run build"

warnings.warn( 2025-04-08 06:46:39,475 uvicorn.error 1 INFO Uvicorn running on http://0.0.0.0:9997 (Press CTRL+C to quit)` UI页面访问显示：

Image

Nuyoah66 avatar Apr 08 '25 06:04 Nuyoah66

使用镜像xprobe/xinference:latest-cpu启动docker服务可以正常启动，但是无法加载运行模型，换用镜像docker-registry.neuedu.com/xprobe/xinference:v0.15.2-cpu启动docker容器可以正常运行，加载bge-m3\bge-rerank都可以启动并运行，curl\postman调用接口均没有问题，但是api调用向量化模型报错：2025-04-09 09:03:07,738 xinference.api.restful_api 1 ERROR Handling request http://192.168.3.54:9997/v1/embeddings failed: 7 validation errors for CreateEmbeddingRequest input str type expected (type=type_error.str) input -> 0 str type expected (type=type_error.str) input -> 1 str type expected (type=type_error.str) input -> 0 value is not a valid integer (type=type_error.integer) input -> 1 value is not a valid integer (type=type_error.integer) input -> 0 value is not a valid list (type=type_error.list) input -> 1 value is not a valid list (type=type_error.list) Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call await self.app(scope, receive, _send) File "/opt/conda/lib/python3.11/site-packages/aioprometheus/asgi/middleware.py", line 184, in call await self.asgi_callable(scope, receive, wrapped_send) File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call await self.app(scope, receive, send) File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 756, in call await self.middleware_stack(scope, receive, send) File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle await self.app(scope, receive, send) File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 72, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/xinference/api/restful_api.py", line 1256, in create_embedding body = CreateEmbeddingRequest.parse_obj(payload) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/pydantic/v1/main.py", line 526, in parse_obj return cls(**obj) ^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/pydantic/v1/main.py", line 341, in init raise validation_error pydantic.v1.error_wrappers.ValidationError: 7 validation errors for CreateEmbeddingRequest input str type expected (type=type_error.str) input -> 0 str type expected (type=type_error.str) input -> 1 str type expected (type=type_error.str) input -> 0 value is not a valid integer (type=type_error.integer) input -> 1 value is not a valid integer (type=type_error.integer) input -> 0 value is not a valid list (type=type_error.list) input -> 1 value is not a valid list (type=type_error.list) 数据格式已经检查没有任何问题，使用gpu版本的xinference部署同一个向量化模型及同样的接口调用没有问题，不确定是否该cpu安装包有没有问题？请求解决。

postman调用接口cpu返参如下：

Image

postman调用接口gpu返参如下：

Image

sunyue-s avatar Apr 10 '25 01:04 sunyue-s

官方最新版本：1.4.1-cpu版本容器无法启动： 启动命令： docker run -d --name xinference141 -v /data/xinference:/xinference -e XINFERENCE_HOME=/xinference -p 9997:9997 xprobe/xinference:v1.4.1-cpu xinference-local -H 0.0.0.0 --log-level debug 2cec77fd76dda267cf8fdbc0f23b81d879599ec73e6c8a5fcb1cc715a5ef2a83 容器日志： `2025-04-08 06:46:31,510 xinference.core.supervisor 24 INFO Xinference supervisor 0.0.0.0:44110 started 2025-04-08 06:46:31,540 xinference.core.worker 24 INFO Starting metrics export server at 0.0.0.0:None 2025-04-08 06:46:31,543 xinference.core.worker 24 INFO Checking metrics export server... 2025-04-08 06:46:33,908 xinference.core.worker 24 INFO Metrics server is started at: http://0.0.0.0:37593 2025-04-08 06:46:33,909 xinference.core.worker 24 INFO Purge cache directory: /xinference/cache 2025-04-08 06:46:33,911 xinference.core.supervisor 24 DEBUG [request 35f63ec2-1445-11f0-87c6-0242ac110002] Enter add_worker, args: <xinference.core.supervisor.SupervisorActor object at 0x7f30c2b5cad0>,0.0.0.0:44110, kwargs: 2025-04-08 06:46:33,912 xinference.core.supervisor 24 DEBUG Worker 0.0.0.0:44110 has been added successfully 2025-04-08 06:46:33,912 xinference.core.supervisor 24 DEBUG [request 35f63ec2-1445-11f0-87c6-0242ac110002] Leave add_worker, elapsed time: 0 s 2025-04-08 06:46:33,912 xinference.core.worker 24 INFO Connected to supervisor as a fresh worker 2025-04-08 06:46:33,934 xinference.core.worker 24 INFO Xinference worker 0.0.0.0:44110 started 2025-04-08 06:46:33,938 xinference.core.supervisor 24 DEBUG Worker 0.0.0.0:44110 resources: {'cpu': ResourceStatus(usage=0.0, total=8, memory_used=3172597760, memory_available=12951879680, memory_total=16637235200)} 2025-04-08 06:46:36,488 xinference.core.supervisor 24 DEBUG Enter get_status, args: <xinference.core.supervisor.SupervisorActor object at 0x7f30c2b5cad0>, kwargs: 2025-04-08 06:46:36,488 xinference.core.supervisor 24 DEBUG Leave get_status, elapsed time: 0 s 2025-04-08 06:46:39,322 xinference.api.restful_api 1 INFO Starting Xinference at endpoint: http://0.0.0.0:9997 /opt/conda/lib/python3.11/site-packages/xinference/api/restful_api.py:845: UserWarning: Xinference ui is not built at expected directory: /opt/conda/lib/python3.11/site-packages/xinference/web/ui/build/ To resolve this warning, navigate to /opt/conda/lib/python3.11/site-packages/xinference/web/ui/ And build the Xinference ui by running "npm run build"

warnings.warn( 2025-04-08 06:46:39,475 uvicorn.error 1 INFO Uvicorn running on http://0.0.0.0:9997 (Press CTRL+C to quit)` UI页面访问显示：

Image

我跟你遇到同样的问题，这个版本镜像前端代码没编译导致的，降低镜像版本就行了

IT-xiaobai avatar Apr 16 '25 06:04 IT-xiaobai

官方最新版本：1.4.1-cpu版本容器无法启动： 启动命令： docker run -d --name xinference141 -v /data/xinference:/xinference -e XINFERENCE_HOME=/xinference -p 9997:9997 xprobe/xinference:v1.4.1-cpu xinference-local -H 0.0.0.0 --log-level debug 2cec77fd76dda267cf8fdbc0f23b81d879599ec73e6c8a5fcb1cc715a5ef2a83 容器日志： 2025-04-08 06:46:31,510 xinference.core.supervisor 24 INFO Xinference supervisor 0.0.0.0:44110 started 2025-04-08 06:46:31,540 xinference.core.worker 24 INFO Starting metrics export server at 0.0.0.0:None 2025-04-08 06:46:31,543 xinference.core.worker 24 INFO Checking metrics export server... 2025-04-08 06:46:33,908 xinference.core.worker 24 INFO Metrics server is started at: http://0.0.0.0:37593 2025-04-08 06:46:33,909 xinference.core.worker 24 INFO Purge cache directory: /xinference/cache 2025-04-08 06:46:33,911 xinference.core.supervisor 24 DEBUG [request 35f63ec2-1445-11f0-87c6-0242ac110002] Enter add_worker, args: <xinference.core.supervisor.SupervisorActor object at 0x7f30c2b5cad0>,0.0.0.0:44110, kwargs: 2025-04-08 06:46:33,912 xinference.core.supervisor 24 DEBUG Worker 0.0.0.0:44110 has been added successfully 2025-04-08 06:46:33,912 xinference.core.supervisor 24 DEBUG [request 35f63ec2-1445-11f0-87c6-0242ac110002] Leave add_worker, elapsed time: 0 s 2025-04-08 06:46:33,912 xinference.core.worker 24 INFO Connected to supervisor as a fresh worker 2025-04-08 06:46:33,934 xinference.core.worker 24 INFO Xinference worker 0.0.0.0:44110 started 2025-04-08 06:46:33,938 xinference.core.supervisor 24 DEBUG Worker 0.0.0.0:44110 resources: {'cpu': ResourceStatus(usage=0.0, total=8, memory_used=3172597760, memory_available=12951879680, memory_total=16637235200)} 2025-04-08 06:46:36,488 xinference.core.supervisor 24 DEBUG Enter get_status, args: <xinference.core.supervisor.SupervisorActor object at 0x7f30c2b5cad0>, kwargs: 2025-04-08 06:46:36,488 xinference.core.supervisor 24 DEBUG Leave get_status, elapsed time: 0 s 2025-04-08 06:46:39,322 xinference.api.restful_api 1 INFO Starting Xinference at endpoint: http://0.0.0.0:9997 /opt/conda/lib/python3.11/site-packages/xinference/api/restful_api.py:845: UserWarning: Xinference ui is not built at expected directory: /opt/conda/lib/python3.11/site-packages/xinference/web/ui/build/ To resolve this warning, navigate to /opt/conda/lib/python3.11/site-packages/xinference/web/ui/ And build the Xinference ui by running "npm run build" warnings.warn( 2025-04-08 06:46:39,475 uvicorn.error 1 INFO Uvicorn running on http://0.0.0.0:9997 (Press CTRL+C to quit) UI页面访问显示： Image

我跟你遇到同样的问题，这个版本镜像前端代码没编译导致的，降低镜像版本就行了

所以您鏡像降版就可以正常使用jina的embeddings model了嗎? 想在問一下可以用的話是降到第幾版?

iga64828 avatar Apr 17 '25 03:04 iga64828

官方最新版本：1.4.1-cpu版本容器无法启动： 启动命令： docker run -d --name xinference141 -v /data/xinference:/xinference -e XINFERENCE_HOME=/xinference -p 9997:9997 xprobe/xinference:v1.4.1-cpu xinference-local -H 0.0.0.0 --log-level debug 2cec77fd76dda267cf8fdbc0f23b81d879599ec73e6c8a5fcb1cc715a5ef2a83 容器日志： 2025-04-08 06:46:31,510 xinference.core.supervisor 24 INFO Xinference supervisor 0.0.0.0:44110 started 2025-04-08 06:46:31,540 xinference.core.worker 24 INFO Starting metrics export server at 0.0.0.0:None 2025-04-08 06:46:31,543 xinference.core.worker 24 INFO Checking metrics export server... 2025-04-08 06:46:33,908 xinference.core.worker 24 INFO Metrics server is started at: http://0.0.0.0:37593 2025-04-08 06:46:33,909 xinference.core.worker 24 INFO Purge cache directory: /xinference/cache 2025-04-08 06:46:33,911 xinference.core.supervisor 24 DEBUG [request 35f63ec2-1445-11f0-87c6-0242ac110002] Enter add_worker, args: <xinference.core.supervisor.SupervisorActor object at 0x7f30c2b5cad0>,0.0.0.0:44110, kwargs: 2025-04-08 06:46:33,912 xinference.core.supervisor 24 DEBUG Worker 0.0.0.0:44110 has been added successfully 2025-04-08 06:46:33,912 xinference.core.supervisor 24 DEBUG [request 35f63ec2-1445-11f0-87c6-0242ac110002] Leave add_worker, elapsed time: 0 s 2025-04-08 06:46:33,912 xinference.core.worker 24 INFO Connected to supervisor as a fresh worker 2025-04-08 06:46:33,934 xinference.core.worker 24 INFO Xinference worker 0.0.0.0:44110 started 2025-04-08 06:46:33,938 xinference.core.supervisor 24 DEBUG Worker 0.0.0.0:44110 resources: {'cpu': ResourceStatus(usage=0.0, total=8, memory_used=3172597760, memory_available=12951879680, memory_total=16637235200)} 2025-04-08 06:46:36,488 xinference.core.supervisor 24 DEBUG Enter get_status, args: <xinference.core.supervisor.SupervisorActor object at 0x7f30c2b5cad0>, kwargs: 2025-04-08 06:46:36,488 xinference.core.supervisor 24 DEBUG Leave get_status, elapsed time: 0 s 2025-04-08 06:46:39,322 xinference.api.restful_api 1 INFO Starting Xinference at endpoint: http://0.0.0.0:9997 /opt/conda/lib/python3.11/site-packages/xinference/api/restful_api.py:845: UserWarning: Xinference ui is not built at expected directory: /opt/conda/lib/python3.11/site-packages/xinference/web/ui/build/ To resolve this warning, navigate to /opt/conda/lib/python3.11/site-packages/xinference/web/ui/ And build the Xinference ui by running "npm run build" warnings.warn( 2025-04-08 06:46:39,475 uvicorn.error 1 INFO Uvicorn running on http://0.0.0.0:9997 (Press CTRL+C to quit) UI页面访问显示： Image

我跟你遇到同样的问题，这个版本镜像前端代码没编译导致的，降低镜像版本就行了

所以您鏡像降版就可以正常使用jina的embeddings model了嗎? 想在問一下可以用的話是降到第幾版?

我用的镜像是xprobe/xinference:v1.4.0-cpu，启动命令如下：

docker run -d \
--name xinference \
-e XINFERENCE_MODEL_SRC=modelscope \
-e XINFERENCE_HOME=/root/xinference \
-v ~/xinference:/root/xinference \
-p 9997:9997 \
xprobe/xinference:v1.4.0-cpu \
xinference-local -H 0.0.0.0 --log-level debug

启动后页面可以打开，但启动向量模型报错，需要进入容器里安装下面的库

docker exec -it xinference bash
pip install "timm>=0.9.16" controlnet-aux

安装好后，重新在页面部署向量模型就可以了，我用的bge-large-zh-v1.5模型验证的没问题。

IT-xiaobai avatar Apr 18 '25 06:04 IT-xiaobai

xprobe/xinference:v1.4.1-cpu没前端，xprobe/xinference:v1.4.0-cpu启动embedding模型报错： RuntimeError: [address=0.0.0.0:44065, pid=247] Failed to import transformers.models.timm_wrapper.configuration_timm_wrapper because of the following error (look up to see its traceback): cannot import name 'ImageNetInfo' from 'timm.data' (/opt/conda/lib/python3.11/site-packages/timm/data/init.py)

进入容器执行timm升级后目前看可用 pip install timm==1.0.13

gyj0825 avatar Apr 18 '25 08:04 gyj0825