Using LiteLLM

[shindere]

As I understand it, LiteLLM uses the same API than OpenAI so it should already be possible to use it with llm. In practice, though, this would require the ability to add a variant of the OpenAI facility that lets one specify an API endpoint, which I didn't find how to do yet.

Is this already possible and, if not, would it be possible to provide this facility, please?

Thanks

[pharmacologic]

yes, it appears possible to add models with their own api base url through the openai plugin via an extra-openai-models.yaml file in dirname "$(llm logs path)"

https://llm.datasette.io/en/stable/other-models.html#openai-compatible-models

https://llm.datasette.io/en/stable/openai-models.html#openai-extra-models

[shindere]

Thank you!

The first URL looked promising, indeed.

However, I must admit that my attempts so far have not been successful.

Conceptually, what I do not really understand is that the LiteLLM instance I am trying to plug llm to is actually serving several models, not just one.

So am I supposed to have one entry in extra-openai-models.yaml for each model proposed by the LiteLLM instance, with the API's base URL repeated in each entry?

That looks a bit suboptimal to me.

Also, even with one entry things didn't work and I got a message saying:

openai.AuthenticationError: Error code: 401 - {'error': {'message':
"Authentication Error, LiteLLM Virtual Key expected. Received=DUMMY_KEY,
expected to start with 'sk-'.", 'type': 'auth_error', 'param': 'None',
'code': '401'}}

So I don't really know what to do as I did set-up the key associated to the model.

I assume I must be missing something really obvious?

[gasull]

After much trying, I was able to run llm pointing to the Jan internal server API (not very useful, but at that time I didn't know about llm-gguf plugin). I think LiteLLM configuration might be similar.

I added this to .config/io.datasette.llm/extra-openai-models.yaml:

- model_id: Qwen2.5-Coder
  model_name: Qwen2.5-Coder
  aliases: ["qwen"]
  api_base: "http://localhost:1337/v1"
  api_key_name: jan