byzer-llm

byzer-llm copied to clipboard

按照文档运行 bge 模型不正常

2

代码 ``` import ray from byzerllm.utils.client import ByzerLLM,LLMRequest,InferBackend ray.init(address="auto",namespace="default",ignore_reinit_error=True) llm = ByzerLLM() llm.setup_gpus_per_worker(0.4).setup_num_workers(2).setup_infer_backend(InferBackend.Transformers) llm.deploy( model_path="/home/dev/model/assets/embeddings/BAAI/bge-base-zh-v1.5/", pretrained_model_type="custom/bge", udf_name="emb", infer_params={} ) ``` Failed to look up actor with name 'emb'. This could...

shell-nlp

- **现象** 调用llm.stream_chat_oai接口，前后推理的重复数据较多  - **排查** 通过本地加打点debug，发现是同一条数据被get_item多次导致  

Hansen523

csaimd