api_base settings not useful

[ganansuan647]

I'm running in jupyter notebook with paper-qa==5.3.0, tyring to use an alternative OPENAI compatiable api. here is my script

# make sure to run in jupyter
import os
import nest_asyncio
nest_asyncio.apply()

# then I import paperqa and set litellm verbose
from paperqa import Docs, Settings
import litellm
litellm.set_verbose=True

# add docs for testing
doc_paths = ("Cui2021.pdf", "Ghosh2018.pdf")
docs = Docs()
for doc in doc_paths:
    docs.add(doc)

I follow the instruction in readme and #492, with the config like this:

llm_config = dict(
    model_list=[
        dict(
            model_name='gpt-4o-2024-08-06',
            litellm_params=dict(
                model='gpt-4o-2024-08-06',
                api_key=os.environ["OPENAI_API_KEY"],
                api_base="https://api.deepbricks.ai/v1/"
            )
        ),
    ],
)

and then make docs.query:

answer = docs.query(
    "What challenges are unique to bispecific antibodies?",
    settings=Settings(llm="gpt-4o-2024-08-06",
                      llm_config=llm_config,
                      summary_llm="gpt-4o-2024-08-06",
                      summary_llm_config=llm_config)
)
print(answer.formatted_answer)

Unfortunatedly I got following output:

Token Counter - using OpenAI token counter, for model=gpt-4o-2024-08-06
LiteLLM: Utils - Counting tokens for OpenAI model=gpt-4o-2024-08-06
Token Counter - using OpenAI token counter, for model=gpt-4o-2024-08-06
LiteLLM: Utils - Counting tokens for OpenAI model=gpt-4o-2024-08-06
set cache: key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_async_client; value: <openai.AsyncOpenAI object at 0x00000257096CDA90>
InMemoryCache: set_cache. current size= 0
set cache: key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_client; value: <openai.OpenAI object at 0x00000257096CF9B0>
InMemoryCache: set_cache. current size= 1
set cache: key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_stream_async_client; value: <openai.AsyncOpenAI object at 0x00000257096CF020>
InMemoryCache: set_cache. current size= 2
14:40:21 - LiteLLM:WARNING: utils.py:325 - `litellm.set_verbose` is deprecated. Please set `os.environ['LITELLM_LOG'] = 'DEBUG'` for debug logs.
set cache: key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_stream_client; value: <openai.OpenAI object at 0x00000257096D2720>
InMemoryCache: set_cache. current size= 3
get cache: cache key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25; local_only: True
get cache: cache result: None
set cache: key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25; value: 1
InMemoryCache: set_cache. current size= 4
get cache: cache key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_async_client; local_only: True
get cache: cache result: <openai.AsyncOpenAI object at 0x00000257096CDA90>
get cache: cache key: b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25_max_parallel_requests_client; local_only: True
get cache: cache result: None


Request to litellm:
litellm.acompletion(model='gpt-4o-2024-08-06', temperature=0.0, messages=[{'role': 'user', 'content': '
 _Too Long So I removed the content_
'}], caching=False, client=<openai.AsyncOpenAI object at 0x00000257096CDA90>, timeout=60.0, stream=False, metadata={'model_group': 'gpt-4o-2024-08-06', 'model_group_size': 1, 
'deployment': 'gpt-4o-2024-08-06', 'model_info': {'id': 'b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25', 'db_model': False},
 **'api_base': None**, 'caching_groups': None}, model_info={'id': 'b622c96ddd11ae0ab2d5badac10abf2bb7977f0b5de7d310a0dc617035bb4e25', 'db_model': False}, max_retries=0)


ASYNC kwargs[caching]: False; litellm.cache: None; kwargs.get('cache'): None
Final returned optional params: {'temperature': 0.0, 'stream': False, 'max_retries': 0, 'extra_body': {}}


POST Request Sent from LiteLLM:
curl -X POST \
**https://api.openai.com/v1/** \
-H 'Authorization: Bearer sk-eE9N********************************************' \
-d '{'model': 'gpt-4o-2024-08-06', 'messages': [{'role': 'user', 'content': '_Also Too Long_'}], 'temperature': 0.0, 'stream': False, 'extra_body': {}}'
...
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

InMemoryCache: set_cache. current size= 5
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?6400e798-a497-47f1-ae13-294c29d9ba0d) or open in a [text editor](command:workbench.action.openLargeOutput?6400e798-a497-47f1-ae13-294c29d9ba0d). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...
[d:\ProgramData\anaconda3\envs\paperqa\Lib\site-packages\executing\executing.py:171](file:///D:/ProgramData/anaconda3/envs/paperqa/Lib/site-packages/executing/executing.py:171): RuntimeWarning: coroutine 'Docs.aadd' was never awaited
  self._nodes_by_line[lineno].append(node)
RuntimeWarning: Enable tracemalloc to get the object al

I wonder what have i done wrong in this situation, litellm obviously got 'api_base': None and send request to api.openai.com by default. How should I config for this api_base?

[dosubot[bot]]

Hello, @ganansuan647! I'm here to assist you with any bugs, questions, or concerns you may have. Let's work together to resolve your issue.

I found a similar closed issue regarding the use of an alternative API base with paperqa and litellm. The issue suggested that version 5 of paperqa outsources all LLM management to BerriAI/litellm ^[1].

To ensure that litellm uses the provided api_base, you should verify that the api_base parameter is correctly set in the litellm_params dictionary within your llm_config. Your configuration seems correct, but the issue might be related to how litellm processes this parameter internally.

Here are a few steps you can try to troubleshoot and resolve the issue:

1. Check Environment Variables: Ensure that there are no conflicting environment variables that might override your api_base setting.

2. Update litellm and paperqa: Make sure you are using the latest versions of both litellm and paperqa, as updates might have fixed related issues.

3. Debugging: Set the environment variable LITELLM_LOG to DEBUG to get more detailed logs, which might help identify why the api_base is not being used.

4. Directly Check litellm Configuration: If possible, inspect the litellm library's source code or documentation to ensure that the api_base parameter is being correctly passed and utilized.

If these steps do not resolve the issue, it might be necessary to look deeper into the litellm library's handling of the api_base parameter, as there might be an internal issue or a specific configuration requirement not documented.

To continue talking to Dosu, mention @dosu.

[whitead]

I actually don't know if this works with litellm, but you can try just setting the environment variable:

export OPENAI_API_BASE=https://api.deepbricks.ai/v1/

before executing.

[bwnjnOEI]

I actually don't know if this works with litellm, but you can try just setting the environment variable:

export OPENAI_API_BASE=https://api.deepbricks.ai/v1/

before executing.

Thanks for your reply, but I'm afraid it doesn‘t work for me, I've tried setting env variables like OPENAI_API_BASE,ANTHROPIC_API_BASE,LITELLM_API_BASE to deepbricks.ai. Unfortunately they all fails.

You may not have set the api of agent_llm, like

from paperqa AgentSettings

settings=Settings(
                     llm="gpt-4o-2024-08-06",
                     llm_config=llm_config,
                     summary_llm="gpt-4o-2024-08-06",
                     summary_llm_config=llm_config,
                     agent=AgentSettings(
                     	agent_llm='gpt-4o-2024-08-06', 
                     	agent_llm_config=llm_config)
))