llm-ls icon indicating copy to clipboard operation
llm-ls copied to clipboard

When the backend is 'tgi', `build_url(...)` should append `/generate` to the URL

Open spew opened this issue 1 year ago • 0 comments

When the backend is tgi, the build_url(...) function is simply returning the supplied URL parameter. When a user passes in the base URL of their TGI server, the result is that the a request is made against the root path and it is routed to /compat_generate. Most users would not expect to pass in ${TGI_BASE_URL}/generate. In addition, llm-ls doesn't appear to be compatible with the /generate_stream endpoint so there is no value in allowing the user to choose between the two.

Suggestion: detect and append /generate to the URL (or build out some more robust logic).

spew avatar Feb 16 '24 18:02 spew