OpenLLM
OpenLLM copied to clipboard
Published
20 hours ago •
bentoml
bentoml
feat: HTTPS support
Feature request
Hey there :)
I have deployed an OpenLLM on a managed service that provides HTTPS out of the box:
curl -X 'POST' \
'https://themodel.url/v1/generate' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"prompt": "What is the difference between a pigeon",
"llm_config": {
"use[ . . . and so on]
}
}'
Curl works like a charm.
In LangChain, I try to create my new llm as such:
llm = OpenLLM(server_url="https://themodel.url/v1/generate", temperature=0.2)
It seems that only HTTP is supported as for now as I get this output from server:
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx</center>
</body>
</html>
And I believe this is an issue on this basecode and not on langchain lib :)
Motivation
HTTPS would be useful to deploy the image without any frontend nor reverse proxy for internal routing of requests
Other
Thanks a lot!
