model for llama.cpp

[Pellarin]

About your locally hosted example

from paperqa import Settings, ask

local_llm_config = dict(
    model_list=[
        dict(
            model_name="my_llm_model",
            litellm_params=dict(
                model="my_llm_model",
                api_base="http://127.0.0.1:8080/",
                api_key="sk-no-key-required",
                temperature=0.1,
                frequency_penalty=1.5,
                max_tokens=512,
            ),
        )
    ]
)

answer = ask(
    "What manufacturing challenges are unique to bispecific antibodies?",
    settings=Settings(
        llm="my_llm_model",
        llm_config=local_llm_config,
        summary_llm="my_llm_model",
        summary_llm_config=local_llm_config,
    ),
)

I get this

  File "paper-qa/conda/lib/python3.12/site-packages/litellm/litellm_core_utils/get_llm_provider_logic.py", line 502, in get_llm_provider
    raise e
  File "paper-qa/conda/lib/python3.12/site-packages/litellm/litellm_core_utils/get_llm_provider_logic.py", line 479, in get_llm_provider
    raise litellm.exceptions.BadRequestError(  # type: ignore
litellm.exceptions.BadRequestError: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=llama
 Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers

What provider/model one should use for llama.cpp? Tried "ollama/llava", which sounded the most logical one, and does not work.

  File "conda/lib/python3.12/site-packages/litellm/main.py", line 449, in acompletion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "paper-qa/conda/lib/python3.12/site-packages/litellm/utils.py", line 8187, in exception_type
    raise e
  File "paper-qa/conda/lib/python3.12/site-packages/litellm/utils.py", line 8152, in exception_type
    raise APIConnectionError(
litellm.exceptions.APIConnectionError: litellm.APIConnectionError: OllamaException - File Not Found
Received Model Group=ollama/llava
Available Model Group Fallbacks=None LiteLLM Retried: 2 times, LiteLLM Max Retries: 3

[dosubot[bot]]

The provider or model that should be used for llama.cpp in the given code example is specified as follows:

from paperqa import Settings, ask

local_llm_config = dict(
    model_list=[
        dict(
            model_name="my_llm_model",
            litellm_params=dict(
                model="my-llm-model",
                api_base="http://localhost:8080/v1",
                api_key="sk-no-key-required",
                temperature=0.1,
                frequency_penalty=1.5,
                max_tokens=512,
            ),
        )
    ]
)

answer = ask(
    "What manufacturing challenges are unique to bispecific antibodies?",
    settings=Settings(
        llm="my-llm-model",
        llm_config=local_llm_config,
        summary_llm="my-llm-model",
        summary_llm_config=local_llm_config,
    ),
)

This configuration uses a locally hosted LLM model named "my-llm-model" with llama.cpp ^[1].

To continue talking to Dosu, mention @dosu.