Intermittent grpc._channel._InactiveRPCError / google.api_core.exceptions.InternalServerError: 500

[luo-anthony]

Description of the bug:

When making several sequential requests to generate_content, every 5-10 requests seems to result in a grpc._channel._InactiveRpcError / google.api_core.exceptions.InternalServerError: 500. These requests are well within the rate limit.

Usage

gemini_model = GenerativeModel("gemini-pro")
...
response = gemini_model.generate_content(
            contents=input_messages,
            generation_config=genai.types.GenerationConfig(temperature=0.1),
        )

Error Trace

Traceback (most recent call last):
  File "my_local_path/.venv/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 79, in error_remapped_callable
    return callable_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/grpc/_channel.py", line 1160, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting"
        debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B2607:f8b0:4009:803::200a%5D:443 {grpc_message:"An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting", grpc_status:13, created_time:"2024-03-09T14:01:25.23488-06:00"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "my_local_path/.venv/bin/developergpt", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/developergpt/cli.py", line 359, in evaluate
    model_output = gemini_adapter.model_command(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/developergpt/gemini_adapter.py", line 209, in model_command
    response = gemini_model.generate_content(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/google/generativeai/generative_models.py", line 232, in generate_content
    response = self._client.generate_content(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/google/ai/generativelanguage_v1beta/services/generative_service/client.py", line 566, in generate_content
    response = rpc(
               ^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/google/api_core/retry.py", line 372, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/google/api_core/retry.py", line 207, in retry_target
    result = target()
             ^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "my_local_path/.venv/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 81, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InternalServerError: 500 An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting

Any other information you'd like to share?

Running on a M2 Pro Macbook Pro

[BA2Ops]

The phenomenon I observed today is that the moment when the 500 error occurs may be related to the consumed token. After you get a long reply, the next request will most likely encounter 500

[suqi]

I got a lot of the same 500 error everyday recently as well.

[LindaLawton]

This is a known issue with the model and not related to the library see #236 236

[yzeng58]

Same here!

[MarkDaoust]

These intermittent ones are often because the model is overloaded. We're going to start 503 for those, which should trigger automatic backoff and retry by the SDK. Also, you can use the "request_options" argument to pass a retry configuration.

https://github.com/google-gemini/generative-ai-python/issues/467