QAnything
QAnything copied to clipboard
LLM服务启动超时`bash ./run.sh -c local -i 0 -b hf -m Qwen-7B-QAnything -t qwen-7b-qanything`
zi@yo:~/QAnything$ bash ./run.sh -c local -i 0 -b hf -m Qwen-7B-QAnything -t qwen-7b-qanything From https://github.com/netease-youdao/QAnything
- branch master -> FETCH_HEAD 当前master分支已是最新,无需更新。 请输入您使用的大模型B数(示例:1.8B/3B/7B): 1.8B model_size=1.8B GPUID1=0, GPUID2=0, device_id=0 GPU1 Model: NVIDIA GeForce RTX 4060 Laptop GPU Compute Capability: null OCR_USE_GPU=False because null >= 7.5 ==================================================== ******************** 重要提示 ******************** ====================================================
./run.sh: line 207: [: 1.8: integer expression expected 您当前的显存为 8188 MiB 推荐部署1.8B的大模型,包括在线的OpenAI API ./run.sh: line 219: [: 1.8: integer expression expected llm_api is set to [local] device_id is set to [0] runtime_backend is set to [hf] model_name is set to [Qwen-7B-QAnything] conv_template is set to [qwen-7b-qanything] tensor_parallel is set to [1] gpu_memory_utilization is set to [0.81] models 文件夹已存在,无需下载。 检查模型版本成功,当前版本为 v2.1.0。 Model directories check passed. (0/8) 模型路径和模型版本检查通过. (0/8) Do you want to use the previous host: localhost? (yes/no) 是否使用上次的host: localhost?(yes/no) 回车默认选yes,请输入:yes Running under WSL Container qanything-container-local Stopping Container qanything-container-local Stopped Container qanything-container-local Removing Container qanything-container-local Removed Container mysql-container-local Stopping Container milvus-standalone-local Stopping Container milvus-standalone-local Stopped Container milvus-standalone-local Removing Container milvus-standalone-local Removed Container milvus-etcd-local Stopping Container milvus-minio-local Stopping Container milvus-etcd-local Stopped Container milvus-etcd-local Removing Container milvus-etcd-local Removed Container milvus-minio-local Stopped Container milvus-minio-local Removing Container milvus-minio-local Removed Container mysql-container-local Stopped Container mysql-container-local Removing Container mysql-container-local Removed Network qanything_milvus_mysql_local Removing Network qanything_milvus_mysql_local Removed [+] Running 5/6 ⠙ Network qanything_milvus_mysql_local Created 3.2s ✔ Container milvus-etcd-local Started 1.3s ✔ Container milvus-minio-local Started 1.3s ✔ Container mysql-container-local Started 1.3s ✔ Container milvus-standalone-local Started 1.5s ✔ Container qanything-container-local Started 2.8s qanything-container-local | qanything-container-local | ============================= qanything-container-local | == Triton Inference Server == qanything-container-local | ============================= qanything-container-local | qanything-container-local | NVIDIA Release 23.05 (build 61161506) qanything-container-local | Triton Server Version 2.34.0 qanything-container-local | qanything-container-local | Copyright (c) 2018-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. qanything-container-local | qanything-container-local | Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved. qanything-container-local | qanything-container-local | This container image and its contents are governed by the NVIDIA Deep Learning Container License. qanything-container-local | By pulling and using the container, you accept the terms and conditions of this license: qanything-container-local | https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license qanything-container-local | qanything-container-local | llm_api is set to [local] qanything-container-local | device_id is set to [0] qanything-container-local | runtime_backend is set to [hf] qanything-container-local | model_name is set to [Qwen-7B-QAnything] qanything-container-local | conv_template is set to [qwen-7b-qanything] qanything-container-local | tensor_parallel is set to [1] qanything-container-local | gpu_memory_utilization is set to [0.81] qanything-container-local | checksum 93bdf95e60bef460ff7e8061e23d6d13 qanything-container-local | default_checksum 77275c133c7dfcf1553a7b5ef043168d qanything-container-local | Obtaining file:///workspace/qanything_local/third_party/FastChat qanything-container-local | Installing build dependencies ... done qanything-container-local | Checking if build backend supports build_editable ... done qanything-container-local | Getting requirements to build editable ... done qanything-container-local | Installing backend dependencies ... done qanything-container-local | Preparing editable metadata (pyproject.toml) ... done qanything-container-local | Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (3.9.1) qanything-container-local | Requirement already satisfied: fastapi in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (0.109.0) qanything-container-local | Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (0.26.0) qanything-container-local | Requirement already satisfied: markdown2[all] in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (2.4.12) qanything-container-local | Requirement already satisfied: nh3 in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (0.2.15) qanything-container-local | Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (1.23.4) qanything-container-local | Requirement already satisfied: prompt-toolkit>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (3.0.43) qanything-container-local | Requirement already satisfied: pydantic<2,>=1 in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (1.10.13) qanything-container-local | Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (2.31.0) qanything-container-local | Requirement already satisfied: rich>=10.0.0 in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (13.7.0) qanything-container-local | Requirement already satisfied: shortuuid in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (1.0.11) qanything-container-local | Requirement already satisfied: tiktoken in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (0.4.0) qanything-container-local | Requirement already satisfied: uvicorn in /usr/local/lib/python3.10/dist-packages (from fschat==0.2.35) (0.25.0) qanything-container-local | Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit>=3.0.0->fschat==0.2.35) (0.2.13) qanything-container-local | Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<2,>=1->fschat==0.2.35) (4.9.0) qanything-container-local | Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.0.0->fschat==0.2.35) (3.0.0) qanything-container-local | Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.0.0->fschat==0.2.35) (2.17.2) qanything-container-local | Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->fschat==0.2.35) (23.2.0) qanything-container-local | Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->fschat==0.2.35) (6.0.4) qanything-container-local | Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->fschat==0.2.35) (1.9.4) qanything-container-local | Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->fschat==0.2.35) (1.4.1) qanything-container-local | Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->fschat==0.2.35) (1.3.1) qanything-container-local | Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->fschat==0.2.35) (4.0.3) qanything-container-local | Requirement already satisfied: starlette<0.36.0,>=0.35.0 in /usr/local/lib/python3.10/dist-packages (from fastapi->fschat==0.2.35) (0.35.1) qanything-container-local | Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx->fschat==0.2.35) (4.2.0) qanything-container-local | Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx->fschat==0.2.35) (2023.11.17) qanything-container-local | Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx->fschat==0.2.35) (1.0.2) qanything-container-local | Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx->fschat==0.2.35) (3.6) qanything-container-local | Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->fschat==0.2.35) (1.3.0) qanything-container-local | Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx->fschat==0.2.35) (0.14.0) qanything-container-local | Requirement already satisfied: wavedrom in /usr/local/lib/python3.10/dist-packages (from markdown2[all]->fschat==0.2.35) (2.0.3.post3) qanything-container-local | Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->fschat==0.2.35) (3.3.2) qanything-container-local | Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->fschat==0.2.35) (2.1.0) qanything-container-local | Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken->fschat==0.2.35) (2023.10.3) qanything-container-local | Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn->fschat==0.2.35) (8.1.7) qanything-container-local | Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=10.0.0->fschat==0.2.35) (0.1.2) qanything-container-local | Requirement already satisfied: exceptiongroup>=1.0.2 in /usr/local/lib/python3.10/dist-packages (from anyio->httpx->fschat==0.2.35) (1.2.0) qanything-container-local | Requirement already satisfied: svgwrite in /usr/local/lib/python3.10/dist-packages (from wavedrom->markdown2[all]->fschat==0.2.35) (1.4.3) qanything-container-local | Requirement already satisfied: six in /usr/lib/python3/dist-packages (from wavedrom->markdown2[all]->fschat==0.2.35) (1.16.0) qanything-container-local | Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from wavedrom->markdown2[all]->fschat==0.2.35) (6.0.1) qanything-container-local | Building wheels for collected packages: fschat qanything-container-local | Building editable for fschat (pyproject.toml) ... done qanything-container-local | Created wheel for fschat: filename=fschat-0.2.35-0.editable-py3-none-any.whl size=14132 sha256=261df339f5da3b48299f16e8c7012475984d5756c4e7699446cb1b206133eea5 qanything-container-local | Stored in directory: /tmp/pip-ephem-wheel-cache-gs7blx23/wheels/10/b5/d4/f972394c64bd92755f0d9ca5f41a850723ea5d306f7bc1b157 qanything-container-local | Successfully built fschat qanything-container-local | Installing collected packages: fschat qanything-container-local | Attempting uninstall: fschat qanything-container-local | Found existing installation: fschat 0.2.35 qanything-container-local | Uninstalling fschat-0.2.35: qanything-container-local | Successfully uninstalled fschat-0.2.35 qanything-container-local | Successfully installed fschat-0.2.35 qanything-container-local | WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv qanything-container-local | qanything-container-local | [notice] A new release of pip is available: 23.3.2 -> 24.0 qanything-container-local | [notice] To update, run: python3 -m pip install --upgrade pip qanything-container-local | qanything-container-local | [notice] A new release of pip is available: 23.3.2 -> 24.0 qanything-container-local | [notice] To update, run: python3 -m pip install --upgrade pip qanything-container-local | GPU ID: 0, 0 qanything-container-local | The triton server for embedding and reranker will start on 0 GPUs qanything-container-local | Executing hf runtime_backend qanything-container-local | The rerank service is ready! (2/8) qanything-container-local | rerank服务已就绪! (2/8) qanything-container-local | The ocr service is ready! (3/8) qanything-container-local | OCR服务已就绪! (3/8) qanything-container-local | The qanything backend service is ready! (4/8) qanything-container-local | qanything后端服务已就绪! (4/8) qanything-container-local | Waiting for [npm run install](5/8)
qanything-container-local | added 1 package in 2s qanything-container-local | npm notice qanything-container-local | npm notice New minor version of npm available! 10.2.3 -> 10.4.0 qanything-container-local | npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.4.0 qanything-container-local | npm notice Run npm install -g [email protected] to update! qanything-container-local | npm notice qanything-container-local | [npm run install] Installed successfully(5/8) qanything-container-local | Waiting for npm run build qanything-container-local | qanything-container-local | > [email protected] build qanything-container-local | > vite build qanything-container-local | qanything-container-local | The CJS build of Vite's Node API is deprecated. See https://vitejs.dev/guide/troubleshooting.html#vite-cjs-node-api-deprecated for more details. qanything-container-local | vite v5.1.3 building for production...
qanything-container-local | ~@/styles/fonts/FZLanTingHeiS-L-GB.TTF referenced in /workspace/qanything_local/front_end/src/styles/common/global.scss didn't resolve at build time, it will remain unchanged to be resolved at runtime qanything-container-local | qanything-container-local | ~@/styles/fonts/FZYaSongS-B-GB.TTF referenced in /workspace/qanything_local/front_end/src/styles/common/global.scss didn't resolve at build time, it will remain unchanged to be resolved at runtime [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/Defaultpage.vue qanything-container-local | 51:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/OptionList.vue qanything-container-local | 196:5 warning Unexpected console statement no-console qanything-container-local | 206:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/Chat.vue qanything-container-local | 271:7 warning Unexpected console statement no-console qanything-container-local | 273:9 warning Unexpected console statement no-console qanything-container-local | 286:13 warning Unexpected console statement no-console qanything-container-local | 292:7 warning Unexpected console statement no-console qanything-container-local | 294:7 warning Unexpected console statement no-console qanything-container-local | 310:7 warning Unexpected console statement no-console qanything-container-local | 311:7 warning Unexpected console statement no-console qanything-container-local | 321:7 warning Unexpected console statement no-console qanything-container-local | 336:3 warning Unexpected console statement no-console qanything-container-local | 367:5 warning Unexpected console statement no-console qanything-container-local | 387:7 warning Unexpected console statement no-console qanything-container-local | 391:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 12 problems (0 errors, 12 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/Head.vue qanything-container-local | 65:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/Defaultpage.vue qanything-container-local | 51:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/Chat.vue qanything-container-local | 271:7 warning Unexpected console statement no-console qanything-container-local | 273:9 warning Unexpected console statement no-console qanything-container-local | 286:13 warning Unexpected console statement no-console qanything-container-local | 292:7 warning Unexpected console statement no-console qanything-container-local | 294:7 warning Unexpected console statement no-console qanything-container-local | 310:7 warning Unexpected console statement no-console qanything-container-local | 311:7 warning Unexpected console statement no-console qanything-container-local | 321:7 warning Unexpected console statement no-console qanything-container-local | 336:3 warning Unexpected console statement no-console qanything-container-local | 367:5 warning Unexpected console statement no-console qanything-container-local | 387:7 warning Unexpected console statement no-console qanything-container-local | 391:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 12 problems (0 errors, 12 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/OptionList.vue qanything-container-local | 196:5 warning Unexpected console statement no-console qanything-container-local | 206:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/Head.vue qanything-container-local | 65:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/OptionList.vue qanything-container-local | 196:5 warning Unexpected console statement no-console qanything-container-local | 206:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/Chat.vue qanything-container-local | 271:7 warning Unexpected console statement no-console qanything-container-local | 273:9 warning Unexpected console statement no-console qanything-container-local | 286:13 warning Unexpected console statement no-console qanything-container-local | 292:7 warning Unexpected console statement no-console qanything-container-local | 294:7 warning Unexpected console statement no-console qanything-container-local | 310:7 warning Unexpected console statement no-console qanything-container-local | 311:7 warning Unexpected console statement no-console qanything-container-local | 321:7 warning Unexpected console statement no-console qanything-container-local | 336:3 warning Unexpected console statement no-console qanything-container-local | 367:5 warning Unexpected console statement no-console qanything-container-local | 387:7 warning Unexpected console statement no-console qanything-container-local | 391:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 12 problems (0 errors, 12 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/Defaultpage.vue qanything-container-local | 51:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/Head.vue qanything-container-local | 65:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/OptionList.vue qanything-container-local | 196:5 warning Unexpected console statement no-console qanything-container-local | 206:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/Chat.vue qanything-container-local | 271:7 warning Unexpected console statement no-console qanything-container-local | 273:9 warning Unexpected console statement no-console qanything-container-local | 286:13 warning Unexpected console statement no-console qanything-container-local | 292:7 warning Unexpected console statement no-console qanything-container-local | 294:7 warning Unexpected console statement no-console qanything-container-local | 310:7 warning Unexpected console statement no-console qanything-container-local | 311:7 warning Unexpected console statement no-console qanything-container-local | 321:7 warning Unexpected console statement no-console qanything-container-local | 336:3 warning Unexpected console statement no-console qanything-container-local | 367:5 warning Unexpected console statement no-console qanything-container-local | 387:7 warning Unexpected console statement no-console qanything-container-local | 391:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 12 problems (0 errors, 12 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/store/useKnowledgeModal.ts qanything-container-local | 59:7 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/AddInput.vue qanything-container-local | 43:5 warning Unexpected console statement no-console qanything-container-local | 51:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/SiderCard.vue qanything-container-local | 98:3 warning Unexpected console statement no-console qanything-container-local | 123:3 warning Unexpected console statement no-console qanything-container-local | 128:3 warning Unexpected console statement no-console qanything-container-local | 149:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 4 problems (0 errors, 4 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/UrlUploadDialog.vue qanything-container-local | 75:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/DeleteModal.vue qanything-container-local | 45:3 warning Unexpected console statement no-console qanything-container-local | 64:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/store/useOptiionList.ts qanything-container-local | 43:7 warning Unexpected console statement no-console qanything-container-local | 45:9 warning Unexpected console statement no-console qanything-container-local | 52:9 warning Unexpected console statement no-console qanything-container-local | 55:7 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 4 problems (0 errors, 4 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/AddInput.vue qanything-container-local | 43:5 warning Unexpected console statement no-console qanything-container-local | 51:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/SiderCard.vue qanything-container-local | 98:3 warning Unexpected console statement no-console qanything-container-local | 123:3 warning Unexpected console statement no-console qanything-container-local | 128:3 warning Unexpected console statement no-console qanything-container-local | 149:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 4 problems (0 errors, 4 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/UrlUploadDialog.vue qanything-container-local | 75:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/DeleteModal.vue qanything-container-local | 45:3 warning Unexpected console statement no-console qanything-container-local | 64:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/UrlUploadDialog.vue qanything-container-local | 75:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/SiderCard.vue qanything-container-local | 98:3 warning Unexpected console statement no-console qanything-container-local | 123:3 warning Unexpected console statement no-console qanything-container-local | 128:3 warning Unexpected console statement no-console qanything-container-local | 149:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 4 problems (0 errors, 4 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/DeleteModal.vue qanything-container-local | 45:3 warning Unexpected console statement no-console qanything-container-local | 64:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/AddInput.vue qanything-container-local | 43:5 warning Unexpected console statement no-console qanything-container-local | 51:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/SiderCard.vue qanything-container-local | 98:3 warning Unexpected console statement no-console qanything-container-local | 123:3 warning Unexpected console statement no-console qanything-container-local | 128:3 warning Unexpected console statement no-console qanything-container-local | 149:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 4 problems (0 errors, 4 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/UrlUploadDialog.vue qanything-container-local | 75:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/UploadInput.vue qanything-container-local | 74:1 warning Unexpected console statement no-console qanything-container-local | 99:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/DeleteModal.vue qanything-container-local | 45:3 warning Unexpected console statement no-console qanything-container-local | 64:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/UploadInput.vue qanything-container-local | 74:1 warning Unexpected console statement no-console qanything-container-local | 99:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/services/axiosInterceptor/interceptors/autoRetry.ts qanything-container-local | 17:7 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/services/axiosInterceptor/interceptors/forceRetry.ts qanything-container-local | 5:5 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 1 problem (0 errors, 1 warning) qanything-container-local | [plugin:vite-plugin-eslint] qanything-container-local | /workspace/qanything_local/front_end/src/components/UploadInput.vue qanything-container-local | 74:1 warning Unexpected console statement no-console qanything-container-local | 99:3 warning Unexpected console statement no-console qanything-container-local | qanything-container-local | ✖ 2 problems (0 errors, 2 warnings) qanything-container-local | ✓ 3394 modules transformed. dist/qanything/index.html 0.81 kB │ gzip: 0.48 kB qanything-container-local | dist/qanything/assets/ai-avatar-x9recamI.png 4.48 kB qanything-container-local | dist/qanything/assets/logo-small-b0CKaHl6.png 7.12 kB qanything-container-local | dist/qanything/assets/icon-file-Bxz0aRpP.png 12.36 kB qanything-container-local | dist/qanything/assets/index-3tlEN2xo.css 4.72 kB │ gzip: 1.54 kB qanything-container-local | dist/qanything/assets/Home-BGs_Xdjt.css 10.80 kB │ gzip: 2.43 kB qanything-container-local | dist/qanything/assets/index-c1YhHJDz.css 11.93 kB │ gzip: 2.47 kB qanything-container-local | dist/qanything/assets/index-uBefNhMQ.js 89.02 kB │ gzip: 31.29 kB qanything-container-local | dist/qanything/assets/index-DpfymmAX.js 280.88 kB │ gzip: 94.49 kB qanything-container-local | dist/qanything/assets/useOptiionList-C-hLbyxo.js 298.47 kB │ gzip: 105.41 kB qanything-container-local | dist/qanything/assets/Home-DYJCKxMn.js 560.64 kB │ gzip: 157.15 kB qanything-container-local | qanything-container-local | (!) Some chunks are larger than 500 kB after minification. Consider: qanything-container-local | - Using dynamic import() to code-split the application qanything-container-local | - Use build.rollupOptions.output.manualChunks to improve chunking: https://rollupjs.org/configuration-options/#output-manualchunks qanything-container-local | - Adjust chunk size limit for this warning via build.chunkSizeWarningLimit. qanything-container-local | ✓ built in 18.72s qanything-container-local | qanything-container-local | > [email protected] postbuild qanything-container-local | > node writeVersion.js qanything-container-local | qanything-container-local | [npm run build] build successfully(6/8) qanything-container-local | Waiting for the front-end service to start... qanything-container-local | 等待启动前端服务 qanything-container-local | qanything-container-local | > [email protected] serve qanything-container-local | > vite preview --port 5052 qanything-container-local | qanything-container-local | The CJS build of Vite's Node API is deprecated. See https://vitejs.dev/guide/troubleshooting.html#vite-cjs-node-api-deprecated for more details. qanything-container-local | Waiting for the front-end service to start... qanything-container-local | 等待启动前端服务 qanything-container-local | ➜ Local: http://localhost:5052/qanything qanything-container-local | ➜ Network: http://172.19.0.6:5052/qanything qanything-container-local | The front-end service is ready!...(7/8) qanything-container-local | 前端服务已就绪!...(7/8) qanything-container-local | I0220 08:36:02.635682 139 infer_handler.cc:629] New request handler for ModelInferHandler, 0 qanything-container-local | I0220 08:36:02.635710 139 infer_handler.h:1025] Thread started for ModelInferHandler qanything-container-local | I0220 08:36:02.635762 139 stream_infer_handler.cc:122] New request handler for ModelStreamInferHandler, 0 qanything-container-local | I0220 08:36:02.635787 139 infer_handler.h:1025] Thread started for ModelStreamInferHandler qanything-container-local | I0220 08:36:02.635792 139 grpc_server.cc:2450] Started GRPCInferenceService at 0.0.0.0:9001 qanything-container-local | I0220 08:36:02.636266 139 http_server.cc:3555] Started HTTPService at 0.0.0.0:9000 qanything-container-local | I0220 08:36:02.694275 139 http_server.cc:185] Started Metrics Service at 0.0.0.0:9002 qanything-container-local | W0220 08:36:03.623936 139 metrics.cc:573] Unable to get power limit for GPU 0. Status:Success, value:0.000000 qanything-container-local | W0220 08:36:04.624398 139 metrics.cc:573] Unable to get power limit for GPU 0. Status:Success, value:0.000000 qanything-container-local | W0220 08:36:05.628659 139 metrics.cc:573] Unable to get power limit for GPU 0. Status:Success, value:0.000000 qanything-container-local | I0220 08:36:23.188765 139 http_server.cc:3449] HTTP request: 0 /v2/health/ready qanything-container-local | The embedding and rerank service is ready!. (7.5/8) qanything-container-local | Embedding 和 Rerank 服务已准备就绪!(7.5/8) qanything-container-local | 2024-02-20 16:36:00 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=7801, worker_address='http://0.0.0.0:7801', controller_address='http://0.0.0.0:7800', model_path='/model_repos/CustomLLM/Qwen-7B-QAnything', revision='main', device='cuda', gpus='0', num_gpus=1, max_gpu_memory=None, dtype='bfloat16', load_8bit=True, cpu_offloading=False, gptq_ckpt=None, gptq_wbits=16, gptq_groupsize=-1, gptq_act_order=False, awq_ckpt=None, awq_wbits=16, awq_groupsize=-1, enable_exllama=False, exllama_max_seq_len=4096, exllama_gpu_split=None, exllama_cache_8bit=False, enable_xft=False, xft_max_seq_len=4096, xft_dtype=None, model_names=None, conv_template='qwen-7b-qanything', embed_in_truncate=False, limit_worker_concurrency=5, stream_interval=2, no_register=False, seed=None, debug=False, ssl=False) qanything-container-local | 2024-02-20 16:36:00 | INFO | model_worker | Loading the model ['Qwen-7B-QAnything'] on worker 8cf97c2f ... qanything-container-local | 2024-02-20 16:36:01 | INFO | stdout | Warning: import flash_attn rotary fail, please install FlashAttention rotary to get better performance https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary qanything-container-local | 2024-02-20 16:36:01 | INFO | stdout | Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get better performance https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm qanything-container-local | 2024-02-20 16:36:01 | INFO | stdout | Warning: import flash_attn fail, please install FlashAttention https://github.com/Dao-AILab/flash-attention 0%| | 0/2 [00:00<?, ?it/s]20 16:36:04 | ERROR | stderr | 50%|█████ | 1/2 [00:18<00:18, 18.56s/it]:22 | ERROR | stderr | qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 42 0 --:--:-- --:--:-- --:--:-- 43 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 1270 0 --:--:-- --:--:-- --:--:-- 1444 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | /workspace/qanything_local/scripts/run_for_local_option.sh: line 404: 153 Killed CUDA_VISIBLE_DEVICES=$gpus nohup python3 -m fastchat.serve.model_worker --host 0.0.0.0 --port 7801 --controller-address http://0.0.0.0:7800 --worker-address http://0.0.0.0:7801 --model-path /model_repos/CustomLLM/$LLM_API_SERVE_MODEL --load-8bit --gpus $gpus --num-gpus $tensor_parallel --dtype bfloat16 --conv-template $LLM_API_SERVE_CONV_TEMPLATE > /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 2>&1 (wd: /workspace/qanything_local/logs/debug_logs/fastchat_logs) qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 16089 0 --:--:-- --:--:-- --:--:-- 13000 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 11265 0 --:--:-- --:--:-- --:--:-- 13000 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 13903 0 --:--:-- --:--:-- --:--:-- 13000 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 10655 0 --:--:-- --:--:-- --:--:-- 13000 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 15873 0 --:--:-- --:--:-- --:--:-- 13000 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 6319 0 --:--:-- --:--:-- --:--:-- 13000 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 7926 0 --:--:-- --:--:-- --:--:-- 13000 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 8855 0 --:--:-- --:--:-- --:--:-- 13000 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | 启动 LLM 服务超时,自动检查 /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 中是否存在Error... qanything-container-local | /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 中未检测到明确的错误 信息。请手动排查 /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 以获取更多信息。
W11 单卡(NVIDIA GeForce RTX 4060 Laptop GPU),显存为 8188 MiB 莫名LLM服务无法启动
日志(/workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log
)为
2024-02-20 17:06:08 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=7801, worker_address='http://0.0.0.0:7801', controller_address='http://0.0.0.0:7800', model_path='/model_repos/CustomLLM/Qwen-7B-QAnything', revision='main', device='cuda', gpus='0', num_gpus=1, max_gpu_memory=None, dtype='bfloat16', load_8bit=True, cpu_offloading=False, gptq_ckpt=None, gptq_wbits=16, gptq_groupsize=-1, gptq_act_order=False, awq_ckpt=None, awq_wbits=16, awq_groupsize=-1, enable_exllama=False, exllama_max_seq_len=4096, exllama_gpu_split=None, exllama_cache_8bit=False, enable_xft=False, xft_max_seq_len=4096, xft_dtype=None, model_names=None, conv_template='qwen-7b-qanything', embed_in_truncate=False, limit_worker_concurrency=5, stream_interval=2, no_register=False, seed=None, debug=False, ssl=False)
2024-02-20 17:06:08 | INFO | model_worker | Loading the model ['Qwen-7B-QAnything'] on worker 15512206 ...
2024-02-20 17:06:09 | INFO | stdout | Warning: import flash_attn rotary fail, please install FlashAttention rotary to get better performance https://github.com/Dao-AILab/flash-attention/tree/main/csrc/rotary
2024-02-20 17:06:09 | INFO | stdout | Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get better performance https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm
2024-02-20 17:06:09 | INFO | stdout | Warning: import flash_attn fail, please install FlashAttention https://github.com/Dao-AILab/flash-attention
2024-02-20 17:06:11 | ERROR | stderr |
0%| | 0/2 [00:00<?, ?it/s]
2024-02-20 17:06:27 | ERROR | stderr |
50%|█████ | 1/2 [00:15<00:15, 15.86s/it]
8G 应该不行,启动后消耗17G
8G 应该不行,启动后消耗17G
要求这么高吗?不是连1050 TI(4G) 都支持吗?
其他组件比如paddle ocr可能也会占用了
@AprildreamMI
For 7B model, >=24GB VRAM (hf/vllm backend) For 7B model, >=16GB VRAM (default backend with some GPUs) For 3B model, <16GB VRAM For OpenAI API, 4GB VRAM
系统: win10 内存: 32G 显卡: 3060 12G 后端模型选择:3B
遇到同样的问题.
启动指令:bash run.sh -c local -i 0 -b hf -m Qwen-7B-QAnything -t qwen-7b-qanything
不是显存的问题,根据日志文件显示是 pip 包flash-attention没有安装导致无法引用进而报错。
于是我在虚拟环境中,根据 flash-attention文档,通过pip一一安装这几个包,pip install packing
,pip install flash-attn --no-build-isolation
.
此时依然出现无法引用的报错,怀疑是 flash-attention在 window中不兼容
遇到同样的问题.
bash run.sh -c local -i 0 -b hf -m Qwen-7B-QAnything -t qwen-7b-qanything
我是3060 12G,选择3B模型.
不是显存的问题,是 pip 包没有安装导致无法引用。但是不知道python的环境在哪个目录里,也没办法单独安装
pip install FlashAttention
![]()
也遇到同样的问题。4060Ti 16GB,代码均为最新,2月初曾经成功启动过1~2次(当时也报FlashAttention的依赖包错误,所以个人认为不是依赖包的事情),10多天前停止服务后再启动,就一直报相同的超时错误。相关日志文件没有给出根源问题。由于python环境是打包在docker镜像里的,所以个人无法进行任何调整。总之,建议能改进日志文件的错误记录功能,给出真正的错误原因。
存在同样的问题 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 2340 0 --:--:-- --:--:-- --:--:-- 2600 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) 100%|██████████| 2/2 [05:23<00:00, 150.72s/it]50 | ERROR | stderr | 100%|██████████| 2/2 [05:23<00:00, 161.67s/it]50 | ERROR | stderr | qanything-container-local | 2024-03-11 09:32:50 | ERROR | stderr | qanything-container-local | 2024-03-11 09:32:50 | INFO | model_worker | Register to controller qanything-container-local | 2024-03-11 09:32:51 | ERROR | stderr | INFO: Started server process [147] qanything-container-local | 2024-03-11 09:32:51 | ERROR | stderr | INFO: Waiting for application startup. qanything-container-local | 2024-03-11 09:32:51 | ERROR | stderr | INFO: Application startup complete. qanything-container-local | 2024-03-11 09:32:51 | ERROR | stderr | INFO: Uvicorn running on http://0.0.0.0:7801 (Press CTRL+C to quit) qanything-container-local | 启动 LLM 服务超时,自动检查 /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 中是否存在Error... qanything-container-local | /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 中未 检测到明确的错误信息。请手动排查 /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 以获取更多信息。
曾经成功启动过,目前也是遇到了这个问题 qanything-container-local | 2024-03-12 19:37:31 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=7801, worker_address='http://0.0.0.0:7801', controller_address='http://0.0.0.0:7800', model_path='/model_repos/CustomLLM/Qwen-7B-QAnything', revision='main', device='cuda', gpus='0', num_gpus=1, max_gpu_memory=None, dtype='bfloat16', load_8bit=True, cpu_offloading=False, gptq_ckpt=None, gptq_wbits=16, gptq_groupsize=-1, gptq_act_order=False, awq_ckpt=None, awq_wbits=16, awq_groupsize=-1, enable_exllama=False, exllama_max_seq_len=4096, exllama_gpu_split=None, exllama_cache_8bit=False, enable_xft=False, xft_max_seq_len=4096, xft_dtype=None, model_names=None, conv_template='qwen-7b-qanything', embed_in_truncate=False, limit_worker_concurrency=5, stream_interval=2, no_register=False, seed=None, debug=False, ssl=False) qanything-container-local | 2024-03-12 19:37:31 | INFO | model_worker | Loading the model ['Qwen-7B-QAnything'] on worker 7995a860 ... qanything-container-local | 2024-03-12 19:37:32 | INFO | stdout | Warning: import flash_attn rms_norm fail, please install FlashAttention layer_norm to get better performance https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm qanything-container-local | 2024-03-12 19:37:32 | INFO | stdout | Warning: import flash_attn fail, please install FlashAttention https://github.com/Dao-AILab/flash-attention 0%| | 0/2 [00:00<?, ?it/s]12 19:37:35 | ERROR | stderr | qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 78 0 --:--:-- --:--:-- --:--:-- 78 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :) qanything-container-local | % Total % Received % Xferd Average Speed Time Time Time Current qanything-container-local | Dload Upload Total Spent Left Speed 100 13 100 13 0 0 583 0 --:--:-- --:--:-- --:--:-- 590 qanything-container-local | The llm service is starting up, it can be long... you have time to make a coffee :) qanything-container-local | LLM 服务正在启动,可能需要一段时间...你有时间去冲杯咖啡 :)
官方出来解决一下呗,好多人有这问题。
同样问题+1
同样问题,我是2080ti 22G,改成minichat就可以跑起来,用Qwen-7B-QAnything,就一直喝咖啡然后超时。是显卡性能不够吗?
相同问题,RTX4060: 启动 LLM 服务超时,自动检查 /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 中是否存在Error... qanything-container-local | /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 中未 检测到明确的错误信息。请手动排查 /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 以获取更多信息。
/workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log的信息:
2024-03-15 23:01:27 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=7801, worker_address='http://0.0.0.0:7801', controller_address='http://0.0.0.0:7800', model_path='/model_repos/CustomLLM/MiniChat-2-3B', revision='main', device='cuda', gpus='0', num_gpus=1, max_gpu_memory=None, dtype='bfloat16', load_8bit=True, cpu_offloading=False, gptq_ckpt=None, gptq_wbits=16, gptq_groupsize=-1, gptq_act_order=False, awq_ckpt=None, awq_wbits=16, awq_groupsize=-1, enable_exllama=False, exllama_max_seq_len=4096, exllama_gpu_split=None, exllama_cache_8bit=False, enable_xft=False, xft_max_seq_len=4096, xft_dtype=None, model_names=None, conv_template='minichat', embed_in_truncate=False, limit_worker_concurrency=5, stream_interval=2, no_register=False, seed=None, debug=False, ssl=False)
2024-03-15 23:01:27 | INFO | model_worker | Loading the model ['MiniChat-2-3B'] on worker 80969698 ...
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the legacy
(previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False
. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
2024-03-15 23:01:27 | ERROR | stderr |
0%| | 0/1 [00:00<?, ?it/s]
尝试了三个模型,分别是minichat,qwen和glm3,但是都出现LLM服务超时的问题,使用的是3B模型
收到Helix大神的消息,“一样的错误, 试试换用这个命令启动:bash ./run.sh -c local -i 0 -b default -m Qwen-7B-QAnything -t qwen-7b-qanything”。我换了命令成功了。之前把QAnything-win升级到1.2.1,配置文件指向1.2.1版也不行,没想到改参数可以解决(hf改default)。我看run.sh文件里default代表使用FasterTransformer,支持40系显卡。
但是default自动选择7B模型啊,我的显存不足以使用7B,如何选择3B呢
@ADZX03 你可以修改根目录下的run.sh
# 获取大模型B数 if [ $llm_api = 'cloud' ]; then model_size='0B' elif [ $runtime_backend = 'default' ]; then model_size='7B'
将7B
修改为3B
即可
相同问题,RTX4080: 启动 LLM 服务超时,自动检查 /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 中是否存在Error... qanything-container-local | /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 中未 检测到明确的错误信息。请手动排查 /workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log 以获取更多信息。
/workspace/qanything_local/logs/debug_logs/fastchat_logs/fschat_model_worker_7801.log的信息: 2024-03-15 23:01:27 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=7801, worker_address='http://0.0.0.0:7801', controller_address='http://0.0.0.0:7800', model_path='/model_repos/CustomLLM/MiniChat-2-3B', revision='main', device='cuda', gpus='0', num_gpus=1, max_gpu_memory=None, dtype='bfloat16', load_8bit=True, cpu_offloading=False, gptq_ckpt=None, gptq_wbits=16, gptq_groupsize=-1, gptq_act_order=False, awq_ckpt=None, awq_wbits=16, awq_groupsize=-1, enable_exllama=False, exllama_max_seq_len=4096, exllama_gpu_split=None, exllama_cache_8bit=False, enable_xft=False, xft_max_seq_len=4096, xft_dtype=None, model_names=None, conv_template='minichat', embed_in_truncate=False, limit_worker_concurrency=5, stream_interval=2, no_register=False, seed=None, debug=False, ssl=False) 2024-03-15 23:01:27 | INFO | model_worker | Loading the model ['MiniChat-2-3B'] on worker 80969698 ... You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the
legacy
(previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, setlegacy=False
. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in huggingface/transformers#24565 2024-03-15 23:01:27 | ERROR | stderr | 0%| | 0/1 [00:00<?, ?it/s]使用MiniChat-2-3B模型,一直报这个错误,有什么解决方案嘛