Jianyu Zhan

Hi, @Ying1123 , @zhyncs , now this PR is complete, and passed all CI tests(The previous e2e-test failure is due to missing PYTHONPATH setting, so I added one in the...

I have rebased upon the latest upstream/main branch, and it passed all CI tests now.

@zhyncs @Ying1123. Yes, the main goal is to provide a server-free (or offline) `Engine`, and the api is quite similar to the `triton_python_backend` you mentioned: https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/serve/turbomind/triton_python_backend/model.py#L113. You can refer to...

@Ying1123 , I did try to reuse most current code. The addition/deleltion line comparsion is dauting, mostly because I forgot to remove the old `serer*py`, and now I re-push the...

@DragonFive , Add "LLM" and "SamplingParams" in `sglang/__init__.py::__all__`, it is now imported in this __init__.py but not exposed, or you can try `from sglang.api import LLM, SamplingParams`. The code lags...

@jischein , Thanks for testing. I don't have this multi-GPU environment to test. Per my analysis, your error looks like the `tp procs` are not terminated properly in `Engine::shutdown()` ....

@DragonFive it is because the vllm dependency is upgraded, I think, you shall update your local installation dependency as well: `pip install --upgrade pip pip install -e "python[all]"`, check the...

> 您好，是哪个模型呢？ 您好，我用的是一个基于[这个文档]( https://paddlepaddle.github.io/PaddleOCR/ppstructure/model_train/train_layout.html )训练并导出的模型, 是基于 `picodet_lcnet_x1_0_layout`训练的模型。

这个至少在我的case里，是能稳定复现的。所以我现在关了GPU推理，明显快很多

**magic-pdf.json**: ``` $ cat ~/magic-pdf.json { "bucket_info": { "bucket-name-1": [ "ak", "sk", "endpoint" ], "bucket-name-2": [ "ak", "sk", "endpoint" ] }, "models-dir": "/home/ubuntu/.cache/huggingface/hub/models--opendatalab--PDF-Extract-Kit-1.0/snapshots/38e484355b9acf5654030286bf72490e27842a3c/models", "layoutreader-model-dir": "/home/ubuntu/.cache/huggingface/hub/models--hantian--layoutreader/snapshots/641226775a0878b1014a96ad01b9642915136853", "device-mode": "cpu", "layout-config": { "model":...