How to set up InternVL-Chat as an API

[BIGBALLON]

First of all, thank you very much for your work, which is very constructive.

I would like to ask how to deploy this model and turn it into an API for others to call. Can you provide the corresponding code? Thank you again.

[czczup]

Thank you for your interest. Currently, we do not offer such code to deploy the model as an API; our existing code only supports deploying the model as a local demo.

I plan to investigate and see if there are mature tools in the community that can facilitate a better deployment of VLLMs. This may take a bit of time, so I kindly ask for your patience during this period.

[BIGBALLON]

@czczup Thank you for your prompt and detailed response. I appreciate the effort you’re putting into this. I’ll be looking forward to any updates.

[czczup]

You can now use lmdeploy to deploy InternVL-Chat as an API, or you can use this existing API.