BerriAI
[Bug]: ValueError: invalid literal for int() with base 10: 'generateContent' (gemini)
What happened?
In version 1.58.2 the above model occurs when I use a gemini model (might for others as well, haven't checked)
Relevant log output
In version `1.57.8` it still says
POST Request Sent from LiteLLM:
curl -X POST \
https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash-8b:generateContent?key=AIzaSyAaG-RsLaAfB0azNPxU-Xl3Bqv1lp1wvqg \
In version `1.58.2` it says
POST Request Sent from LiteLLM:
curl -X POST \
https://generativelanguage.googleapis.com:generateContent \
Are you a ML Ops Team?
No
What LiteLLM version are you on ?
1.58.2
Twitter / LinkedIn details
@j0yk1ll
ValueError: invalid literal for int() with base 10: 'generateContent
can you share the stacktrace for this error? not sure i follow
ValueError: invalid literal for int() with base 10: 'generateContent
can you share the stacktrace for this error? not sure i follow
POST Request Sent from LiteLLM:
curl -X POST \
https://generativelanguage.googleapis.com:generateContent \
-H 'Content-Type: *****' -H 'Authorization: Be********************************************' \
-d '{'contents': [{'role': 'user', 'parts': [{'text': '\n"...'}]}, 'generationConfig': {'stop_sequences': [], 'max_output_tokens': 1500}}'
[0m
2025-01-17 00:13:06,693 [DEBUG] Logging Details: logger_fn - None | callable(logger_fn) - False
2025-01-17 00:13:06,698 [DEBUG] Logging Details LiteLLM-Failure Call: []
2025-01-17 00:13:06,699 [WARNING] A litellm.APIConnectionError occurred: litellm.APIConnectionError: Invalid port: 'generateContent'
Traceback (most recent call last):
File "[...]\.venv\Lib\site-packages\httpx\_urlparse.py", line 364, in normalize_port
port_as_int = int(port)
^^^^^^^^^
ValueError: invalid literal for int() with base 10: 'generateContent'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "[...]\.venv\Lib\site-packages\litellm\main.py", line 2217, in completion
response = vertex_chat_completion.completion( # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[...]\.venv\Lib\site-packages\litellm\llms\vertex_ai\gemini\vertex_and_google_ai_studio_gemini.py", line 1282, in completion
response = client.post(url=url, headers=headers, json=data) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[...]\.venv\Lib\site-packages\litellm\llms\custom_httpx\http_handler.py", line 529, in post
raise e
File "[...]\.venv\Lib\site-packages\litellm\llms\custom_httpx\http_handler.py", line 504, in post
req = self.client.build_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[...]\.venv\Lib\site-packages\httpx\_client.py", line 346, in build_request
url = self._merge_url(url)
^^^^^^^^^^^^^^^^^^^^
File "[...]\.venv\Lib\site-packages\httpx\_client.py", line 376, in _merge_url
merge_url = URL(url)
^^^^^^^^
File "[...]\.venv\Lib\site-packages\httpx\_urls.py", line 117, in __init__
self._uri_reference = urlparse(url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "[...]\.venv\Lib\site-packages\httpx\_urlparse.py", line 260, in urlparse
parsed_port: int | None = normalize_port(port, scheme)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[...]\.venv\Lib\site-packages\httpx\_urlparse.py", line 366, in normalize_port
raise InvalidURL(f"Invalid port: {port!r}")
httpx.InvalidURL: Invalid port: 'generateContent'
If you look at the target url, it not constructed correctly.
It was https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash-8b:generateContent in the older version now it's just https://generativelanguage.googleapis.com:generateContent so it's missing the /v1beta/models/gemini-1.5-flash-8b. Because of that it thinks the :generateContent is a port, which obviously it can not parse.
Is there any update on this issue as I am also facing the same problem trying to call gemini model endpoints.
I'm running into a similar issue with streamGenerateContent, and I suspect the cause is the same.
In this method:
def _check_custom_proxy(
self,
api_base: Optional[str],
custom_llm_provider: str,
gemini_api_key: Optional[str],
endpoint: str,
stream: Optional[bool],
auth_header: Optional[str],
url: str,
) -> Tuple[Optional[str], str]:
"""
for cloudflare ai gateway - https://github.com/BerriAI/litellm/issues/4317
## Returns
- (auth_header, url) - Tuple[Optional[str], str]
"""
if api_base:
if custom_llm_provider == "gemini":
url = "{}:{}".format(api_base, endpoint)
if gemini_api_key is None:
raise ValueError(
"Missing gemini_api_key, please set `GEMINI_API_KEY`"
)
auth_header = (
gemini_api_key # cloudflare expects api key as bearer token
)
else:
url = "{}:{}".format(api_base, endpoint)
if stream is True:
url = url + "?alt=sse"
return auth_header, url
The url parameter here is something like 'https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash-exp:streamGenerateContent?key=<key redacted>&alt=sse, which is the correct url format. If, however, api_base is provided, the url is overridden to be just the value passed in for api base, and the endpoint, which, in my case is streamGenerateContent. If the value for api_base passed in is the result of the provider's get_api_base() call, which seems like something that should logically work, the value of api_base in this instance is https://generativelanguage.googleapis.com, and the resulting url is, therefore is 'https://generativelanguage.googleapis.com:streamGenerateContent', which is an invalid value.
Hey @michelle-avery do you have a test script i can use to repro the problem?
Here you go:
from litellm import completion
import os
GEMINI_API_KEY = # your api key
# This works
response = completion(model="gemini/gemini-1.5-flash-latest", messages=[{"role": "user", "content": "What model are you?"}], api_key=GEMINI_API_KEY)
print(response)
# This fails. Note that the endpoint is actually the default, merely providing it makes it fail.
response = completion(model="gemini/gemini-1.5-flash-latest", messages=[{"role": "user", "content": "What model are you?"}], base_url="https://generativelanguage.googleapis.com", api_key=GEMINI_API_KEY)
print(response)
A quick fix would be to replace "https://generativelanguage.googleapis.com" with api_base content if provided
def _check_custom_proxy(
self,
api_base: Optional[str],
custom_llm_provider: str,
gemini_api_key: Optional[str],
endpoint: str,
stream: Optional[bool],
auth_header: Optional[str],
url: str,
) -> Tuple[Optional[str], str]:
"""
for cloudflare ai gateway - https://github.com/BerriAI/litellm/issues/4317
## Returns
- (auth_header, url) - Tuple[Optional[str], str]
"""
if api_base:
url = url.replace("https://generativelanguage.googleapis.com", api_base)
if custom_llm_provider == "gemini":
if gemini_api_key is None:
raise ValueError(
"Missing gemini_api_key, please set `GEMINI_API_KEY`"
)
auth_header = (
gemini_api_key # cloudflare expects api key as bearer token
)
return auth_header, url
This fixed it for me ^, happy to commit , if ok
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}