dspy
dspy copied to clipboard
Published
20 hours ago •
stanfordnlp
stanfordnlp
[Feature] Cannot use http_client for ssl_context or proxy settings via LiteLLM
What feature would you like to see?
I cannot see any documentation or figure out how to use the dspy.LM adapter with http_client parameter (standard openai) that sets proxy and ssl context settings. This was added to the old adapter here. Is there something I'm missing?
Would you like to contribute?
- [ ] Yes, I'd like to help implement this.
- [ ] No, I just want to request it.
Additional Context
No response
Hi @satyaloka93 , is this LiteLLM documentation useful? seems like an external litellm configuration you can set and then just call dspy.LM in the same way as you would.
Ok, when I try setting litellm.client_session = httpx_client, then:
import litellm
import os
response = litellm.completion(
model="openai/claude-3-5-sonnet-v2", # add `openai/` prefix to model so litellm knows to route to OpenAI
api_key="sk-xxxxxx, # api key to your openai compatible endpoint
api_base="http://0.0.0.0:4000", # set API Base of your Custom OpenAI Endpoint
messages=[
{
"role": "user",
"content": "Hey, how's it going?",
}
],
)
print(response)
It works! However, setting the client_session first then using dspy.LM similarly (model="openai/claude-3-5-sonnet-v2"),, I get:
APIError: litellm.APIError: APIError: OpenAIException - Internal Server Error (litellm_core_utils/exception_mapping_utils.py:404)
Sorry, typing manually from another system, cannot paste entire stack. But it appears simply that dspy.LM does not take the standard required pre-pended provider openai, befor the model, which works using a standard litellm.completion. If you don't prepend 'openai', there is an error about not supplying the provider. I've tried putting provider='openai' as an argument, but that does not work, same error about 'LLM Provider NOT provided'.
Had a chance to look at this @arnavsinghvi11? I would really love to use the new adapter, stuck on old dspy version because of lack of http_client support still.
Hey @satyaloka93 , dspy.LM works with prepending 'openai/' before your model type. Can you check if your httpx_client configuration follows the documentation above and is also compliant with the LiteLLM SSL settings? Setting a proxy server might help as well https://docs.litellm.ai/docs/providers/openai#set-ssl_verifyfalse
I'm in the same situation. I'm having a AWS lambda function proxying an OpenAI compatible LLM. I've written an example class called GenAI_AWSSigV4Transport which basically connects to AWS, authenticates, sends the request to the lambda function and returns the chat completion as returned by the API. This all works fine with litellm but not with dspy.
Any pointers on what to change would be highly appreciated.
>>> import dspy
>>> import litellm
>>> import os
>>> from typing import Literal
>>> litellm.client_session = httpx.Client(transport = GenAI_AWSSigV4Transport(
... function = settings["GENAI_GATEWAY_FUNCTION"],
... role = settings["LAMBDA_ROLE_ARN"]))
>>> response = litellm.completion(
... model="openai/gpt-4o-mini",
... api_key="not-used",
... api_base=settings["GENAI_GATEWAY_LAMBDA_VPC_ENDPOINT"],
... messages=[
... {
... "role": "user",
... "content": "Hey, how's it going?",
... }
... ],
... )
>>> print(response)
ModelResponse(id='chatcmpl-ByPXmUm6SoXEvKlXd8PAjycJvOolx', created=1753737570, model='gpt-4o-mini-2024-07-18', object='chat.completion', system_fingerprint='fp_efad92c60b', choices=[Choices(finish_reason='stop', index=0, message=Message(content="I'm doing well, thanks! How about you?", role='assistant', tool_calls=None, function_call=None, provider_specific_fields={'refusal': None}, annotations=[]), provider_specific_fields={'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}})], usage=Usage(completion_tokens=11, prompt_tokens=14, total_tokens=25, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None)), service_tier=None, prompt_filter_results=[{'prompt_index': 0, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}], cache_hit=None)
>>>
>>> class Classify(dspy.Signature):
... sentence: str = dspy.InputField()
... sentiment: Literal["positive", "negative", "neutral"] = dspy.OutputField()
... confidence: float = dspy.OutputField()
...
>>> lm = dspy.LM(
... model = "openai/gpt-4o-mini",
... api_key="not-used",
... api_base=settings["GENAI_GATEWAY_LAMBDA_VPC_ENDPOINT"],
... provider='openai')
>>> dspy.configure(lm=lm)
>>> classify = dspy.Predict(Classify)
>>> classify(sentence="This book was super fun to read, though not the last chapter.")
2025/07/28 23:19:33 WARNING dspy.adapters.json_adapter: Failed to use structured output format, falling back to JSON mode.
Traceback (most recent call last):
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/llms/openai/openai.py", line 725, in completion
raise e
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/llms/openai/openai.py", line 653, in completion
) = self.make_sync_openai_chat_completion_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/litellm_core_utils/logging_utils.py", line 149, in sync_wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/llms/openai/openai.py", line 471, in make_sync_openai_chat_completion_request
raise e
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/llms/openai/openai.py", line 453, in make_sync_openai_chat_completion_request
raw_response = openai_client.chat.completions.with_raw_response.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/openai/_legacy_response.py", line 364, in wrapped
return cast(LegacyAPIResponse[R], func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/openai/_utils/_utils.py", line 287, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/openai/resources/chat/completions/completions.py", line 1087, in create
return self._post(
^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/openai/_base_client.py", line 1256, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/openai/_base_client.py", line 1044, in request
raise self._make_status_error_from_response(err.response) from None
openai.PermissionDeniedError: <AccessDeniedException>
<Message>Unable to determine service/operation name to be authorized</Message>
</AccessDeniedException>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/main.py", line 1966, in completion
raise e
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/main.py", line 1939, in completion
response = openai_chat_completions.completion(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/llms/openai/openai.py", line 736, in completion
raise OpenAIError(
litellm.llms.openai.common_utils.OpenAIError: <AccessDeniedException>
<Message>Unable to determine service/operation name to be authorized</Message>
</AccessDeniedException>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/utils.py", line 1205, in wrapper
result = original_function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/main.py", line 3427, in completion
raise exception_type(
^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2301, in exception_type
raise e
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 528, in exception_type
raise APIError(
litellm.exceptions.APIError: litellm.APIError: APIError: OpenAIException - <AccessDeniedException>
<Message>Unable to determine service/operation name to be authorized</Message>
</AccessDeniedException>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/adapters/json_adapter.py", line 69, in __call__
return super().__call__(lm, lm_kwargs, signature, demos, inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/adapters/chat_adapter.py", line 50, in __call__
raise e
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/adapters/chat_adapter.py", line 42, in __call__
return super().__call__(lm, lm_kwargs, signature, demos, inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/adapters/base.py", line 119, in __call__
outputs = lm(messages=inputs, **lm_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/utils/callback.py", line 326, in sync_wrapper
return fn(instance, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/clients/base_lm.py", line 96, in __call__
response = self.forward(prompt=prompt, messages=messages, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/clients/lm.py", line 127, in forward
results = completion(
^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/clients/cache.py", line 232, in sync_wrapper
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/clients/lm.py", line 304, in litellm_completion
return litellm.completion(
^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/utils.py", line 1310, in wrapper
return litellm.completion_with_retries(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/main.py", line 3465, in completion_with_retries
return retryer(original_function, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/tenacity/__init__.py", line 420, in exc_check
raise retry_exc.reraise()
^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/tenacity/__init__.py", line 187, in reraise
raise self.last_attempt.result()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/gitpod/.pyenv/versions/3.12.3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/home/gitpod/.pyenv/versions/3.12.3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/utils.py", line 1330, in wrapper
raise e
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/utils.py", line 1205, in wrapper
result = original_function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/main.py", line 3427, in completion
raise exception_type(
^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2301, in exception_type
raise e
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 528, in exception_type
raise APIError(
litellm.exceptions.APIError: litellm.APIError: APIError: OpenAIException - <AccessDeniedException>
<Message>Unable to determine service/operation name to be authorized</Message>
</AccessDeniedException>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/predict/predict.py", line 85, in __call__
return super().__call__(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/utils/callback.py", line 326, in sync_wrapper
return fn(instance, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/primitives/program.py", line 60, in __call__
return self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/predict/predict.py", line 157, in forward
completions = adapter(lm, lm_kwargs=config, signature=signature, demos=demos, inputs=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/adapters/chat_adapter.py", line 51, in __call__
return JSONAdapter()(lm, lm_kwargs, signature, demos, inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/.pyenv_mirror/user/3.12.3/lib/python3.12/site-packages/dspy/adapters/json_adapter.py", line 75, in __call__
raise RuntimeError(
RuntimeError: Both structured output format and JSON mode failed. Please choose a model that supports `response_format` argument. Original error: litellm.APIError: APIError: OpenAIException - <AccessDeniedException>
<Message>Unable to determine service/operation name to be authorized</Message>
</AccessDeniedException>
Which seems to indicate it hits the openai endpoint instead of my lambda function. If I change it to the below dspy.LM setup, it does hit my custom transport layer (output not shown but it gives an error as non-existing-model does not exist) - probably related to this: https://github.com/stanfordnlp/dspy/blob/cd9ae6e6334c3bcab88fcf387ab9e1aea321f991/dspy/clients/openai.py#L99-L100
lm = dspy.LM(
model = "openai/non-existing-model",
api_key="not-used",
api_base=settings["GENAI_GATEWAY_LAMBDA_VPC_ENDPOINT"],
provider='openai')
I could bypass this in my own GenAI_AWSSigV4Transport to make sure model = "openai/dspy-gpt-4o-mini" goes to "openai/gpt-4o-mini" but maybe there is a better way?
