Failed to create rerank model when use vllm rerank.

[HeZiGang]

Self Checks

• [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [x] Please do not modify this template :) and fill in all the required fields.

Dify version

langgenius/dify-api:0.15.3 ｜ langgenius/dify-web:0.15.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Recently, I integrated a reranking model using vllm and encountered issues when creating reranking tasks in Dify. The API consistently returned a 400 error, which after reviewing the logs, was traced to line 67 in rerank.py where top_n was set to None. This caused the vllm /v1/rerank endpoint to reject the request due to an invalid parameter. To resolve this, I modified the default value of top_n on line 36 to 3, which resolved the issue.

However, since these changes were made directly within the Docker container, they lack persistence. I recommend implementing a more robust solution by modifying the source code to properly support the vllm rerank API.

Steps to Reproduce:

1. Start the rerank model using vllm Run the following command to initialize and serve the reranking model: vllm serve BAAI/bge-reranker-base --served-model-name bge-reranker-base --dtype auto --port 8889 --api-key token-abc123

2. Configure the model provider in Dify Navigate to the settings > Provider > OpenAI-API-compatible section and create a new model configuration.

3. Select the rerank model type and set parameters Choose "rerank" as the model type and input the required parameters for the model.

4. Save the configuration After configuring the settings, click "Save" to apply the changes and trigger the issue.

BTW, if I don’t change the content length, the save button stays disabled.

✔️ Expected Behavior

Creation is successful

❌ Actual Behavior

Triggers an HTTP 400 error

[dosubot[bot]]

Hey @HeZiGang! 👋 I'm here to help you with any bugs, questions, or contributions. Let's squash those bugs together!

To address the issue of the top_n parameter being set to None and causing an HTTP 400 error with the vllm rerank API, you can implement a more persistent solution by modifying the source code to ensure top_n has a default value. Here's a suggested approach:

1. Modify the Source Code: Update the rerank.py file to set a default value for top_n if it is not provided. This can be done by adding a default value in the _invoke method where top_n is used. For example, you can set top_n to a default of 3 if it is None.

   def _invoke(self, model: str, credentials: dict,
               query: str, docs: list[str], score_threshold: Optional[float] = None, top_n: Optional[int] = None,
               user: Optional[str] = None) -> RerankResult:
       if top_n is None:
           top_n = 3  # Set a default value
       # Rest of the code...
   

2. Persist Changes: Ensure that these changes are made in the source code outside of the Docker container. This way, they will persist across container restarts and deployments. You can achieve this by editing the source code in your local development environment and then rebuilding the Docker image.

3. Rebuild and Deploy: After making the changes, rebuild your Docker image and redeploy the container to apply the updated code.

By implementing these changes, you can ensure that the top_n parameter is always set to a valid default value, preventing the HTTP 400 error when using the vllm rerank API.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[kenwoodjw]

But master branch remove the code api/core/model_runtime/model_providers/openai_api_compatible/rerank/rerank.py, so which branch should I checkout. @crazywoola

[quanjinzhang]

Hey @HeZiGang! 👋 I'm here to help you with any bugs, questions, or contributions. Let's squash those bugs together!

To address the issue of the top_n parameter being set to None and causing an HTTP 400 error with the vllm rerank API, you can implement a more persistent solution by modifying the source code to ensure top_n has a default value. Here's a suggested approach:

1. Modify the Source Code: Update the rerank.py file to set a default value for top_n if it is not provided. This can be done by adding a default value in the _invoke method where top_n is used. For example, you can set top_n to a default of 3 if it is None. def _invoke(self, model: str, credentials: dict, query: str, docs: list[str], score_threshold: Optional[float] = None, top_n: Optional[int] = None, user: Optional[str] = None) -> RerankResult: if top_n is None: top_n = 3 # Set a default value # Rest of the code...
2. Persist Changes: Ensure that these changes are made in the source code outside of the Docker container. This way, they will persist across container restarts and deployments. You can achieve this by editing the source code in your local development environment and then rebuilding the Docker image.
3. Rebuild and Deploy: After making the changes, rebuild your Docker image and redeploy the container to apply the updated code.

By implementing these changes, you can ensure that the top_n parameter is always set to a valid default value, preventing the HTTP 400 error when using the vllm rerank API.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

Could you please help to point out the command to rebuild and deploy docker after modify the source code outside docker. Thank you very much !

[quanjinzhang]

Hey @HeZiGang! 👋 I'm here to help you with any bugs, questions, or contributions. Let's squash those bugs together!

To address the issue of the top_n parameter being set to None and causing an HTTP 400 error with the vllm rerank API, you can implement a more persistent solution by modifying the source code to ensure top_n has a default value. Here's a suggested approach:

1. Modify the Source Code: Update the rerank.py file to set a default value for top_n if it is not provided. This can be done by adding a default value in the _invoke method where top_n is used. For example, you can set top_n to a default of 3 if it is None. def _invoke(self, model: str, credentials: dict, query: str, docs: list[str], score_threshold: Optional[float] = None, top_n: Optional[int] = None, user: Optional[str] = None) -> RerankResult: if top_n is None: top_n = 3 # Set a default value # Rest of the code...
2. Persist Changes: Ensure that these changes are made in the source code outside of the Docker container. This way, they will persist across container restarts and deployments. You can achieve this by editing the source code in your local development environment and then rebuilding the Docker image.
3. Rebuild and Deploy: After making the changes, rebuild your Docker image and redeploy the container to apply the updated code.

By implementing these changes, you can ensure that the top_n parameter is always set to a valid default value, preventing the HTTP 400 error when using the vllm rerank API.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

[@dosu] Could you please help to point out the command to rebuild docker image and deploy docker after modify the source code outside docker. Thank you very much !

[jcvv0n]

PR#362 fixed in plugin require dify>=1.0.0, plugin OpenAI-API-compatible >= 0.0.6 @HeZiGang @kenwoodjw @quanjinzhang

[dosubot[bot]]

Hi, @HeZiGang. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• You reported a bug with the Dify vllm rerank model causing a 400 error when top_n was set to None.
• A temporary fix was applied by setting top_n to 3, with further instructions I provided for a permanent solution.
• Kenwoodjw noted the relevant code was removed from the master branch, and Quanjinzhang sought guidance on Docker image deployment.
• The issue was resolved with a fix in PR#362, requiring Dify version 1.0.0 and plugin version 0.0.6.

Next Steps:

• Please confirm if the issue is still relevant to the latest version of the Dify repository by commenting here.
• If no further updates are provided, the issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!