Jiaxin Shan

 I think the problem is probably due to the env miss https://github.com/vllm-project/aibrix/pull/776 this change, and it forward request to worker pod. Note, worker use different probe from head. After...

@simon-mo @Yard1 Do you know the status of this PR? We are building control planes for LoRa and need this change. I'm unsure if the original author is still working...

1800 seems too high. Let's hold this change and please run more benchmarks so we can tune this number later

Due to limited time, I will untrack this issue from v0.2.0.

## use logits_processors in request extra_body ``` def infer(img_base64): response = client.chat.completions.create( model=model, messages=[ { "role": "user", "content": [ { "type": "image_url", # "image_url": {"url": f"data:image/png;base64,{img_base64}"}, "image_url": {"url": "https://jeroen.github.io/images/testocr.png"}, },...

## specify logits_processors in engine startup command

### work version 1: ``` vllm serve deepseek-ai/DeepSeek-OCR ``` ``` from openai import OpenAI import base64 import os os.environ["OPENAI_API_KEY"] = "" client = OpenAI(base_url="http://localhost:8000/v1") model = "deepseek-ai/DeepSeek-OCR" def encode_image(image_path): with...

## work version 2 vllm source doc

## work version 3 wrapper + standard openai api ``` # server.py import base64 import io import os from typing import List, Optional, Union, Dict, Any from fastapi import FastAPI,...

I will also help take a look this weekend