SII-Auraithm
SII-Auraithm
Is there any reference document?
In addition to updating via the API, does the pipeline also support updating weights? Is there any documentation available?
How should this request be constructed? And, if it's offline inference, is it possible to update directly from the model instead of converting it to base64 or something similar?
What is the version of lmdeploy? I used the version 0.10.1. ``` await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 186, in __call__ raise exc File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 164, in __call__...
``` export LMDEPLOY_EXECUTOR_BACKEND=ray lmdeploy serve api_server /inspire/hdd/global_user/liuxiaoran-240108120089/public/SDAR-8B-Chat \ --model-name "SDAR-8B-Chat" \ --server-port 23333 \ --tp 8 ``` What needs to be modified? I added this environment variable by referring to...
``` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torch.AcceleratorError: CUDA error: invalid resource handle CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider...
> > ``` > > export LMDEPLOY_EXECUTOR_BACKEND=ray > > lmdeploy serve api_server /inspire/hdd/global_user/liuxiaoran-240108120089/public/SDAR-8B-Chat \ > > --model-name "SDAR-8B-Chat" \ > > --server-port 23333 \ > > --tp 8 > >...
··· > > ``` > > export LMDEPLOY_EXECUTOR_BACKEND=ray > > lmdeploy serve api_server /inspire/hdd/global_user/liuxiaoran-240108120089/public/SDAR-8B-Chat \ > > --model-name "SDAR-8B-Chat" \ > > --server-port 23333 \ > > --tp 8 >...
> [@Auraithm](https://github.com/Auraithm) hi, pls. follow this. > > ``` > # step 1 sleep > > response = requests.post(f"{BASE_URL}/sleep", headers=headers, params=dict(tags=['weights', 'kv_cache'], level=1)) > assert response.status_code == 200, response.status_code >...
> [@Auraithm](https://github.com/Auraithm) Hi, pls refer to https://lmdeploy.readthedocs.io/en/latest/advance/update_weights.html Thank you. If I want to use proxy_server to deploy on multi nodes like https://lmdeploy.readthedocs.io/en/latest/llm/proxy_server.html, and I want to update weights，which url is...