instructor icon indicating copy to clipboard operation
instructor copied to clipboard
verified publisher icon jxnl

Google GenAI stopped working with API and SafetySettings Images v2.

Open lukaLLM opened this issue 4 months ago • 3 comments

  • [x] This is actually a bug report.
  • [x] I am not getting good LLM Results
  • [x] I have tried asking for help in the community on discord or discussions and have not received a response.
  • [ x] I have tried searching the documentation and have not found an answer.

What Model are you using?

  • [ ] gpt-3.5-turbo
  • [ ] gpt-4-turbo
  • [ ] gpt-4
  • [x] Other (please specify) google gen ai all models Describe the bug A clear and concise description of what the bug is. I wanted to change from open ai to google to test their models as with their new update it stop working as intended. I find something similar here https://github.com/567-labs/instructor/issues/1658 and here https://github.com/567-labs/instructor/discussions/1655 But I tried newest 1.10 instructor version and reducing gen ai to <1.24.0 but it dont work i get harm message related to images. To Reproduce Steps to reproduce the behavior, including code snippets of the model and the input data and openai response. I get Harm Errors that I pass illegal thing but its just some data and the logs suggest instructor is passing from things. I rewrote it to use from_provider and get same issues Create the base client,
        base_client = genai.Client(api_key=self.api_key)


        from instructor import from_genai
        self.client = from_genai(base_client, use_async=True)

response = await self.client.chat.completions.create(
                model="gemini-2.5-pro",  # Add the model parameter
                response_model=Patient,
                max_retries=5,  # Reduced for faster failure
                #temperature=0.5,
                #max_completion_tokens=10000,
                messages=messages
            )

Error:

2025-08-13 15:37:13,358 - ERROR - Error generating profile after 599.36 ms: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"\nInvalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"\nInvalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"\nInvalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"', 'status': 'INVALID_ARGUMENT', 'details': [{'@type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'safety_settings[5].category', 'description': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"'}, {'field': 'safety_settings[6].category', 'description': 'Invalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"'}, {'field': 'safety_settings[7].category', 'description': 'Invalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"'}, {'field': 'safety_settings[8].category', 'description': 'Invalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"'}]}]}}
2025-08-13 15:37:13,358 - ERROR - Error type: InstructorRetryException
2025-08-13 15:37:13,358 - ERROR - Unexpected Gemini error: Error generating profile: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"\nInvalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"\nInvalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"\nInvalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"', 'status': 'INVALID_ARGUMENT', 'details': [{'@type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'safety_settings[5].category', 'description': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"'}, {'field': 'safety_settings[6].category', 'description': 'Invalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"'}, {'field': 'safety_settings[7].category', 'description': 'Invalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"'}, {'field': 'safety_settings[8].category', 'description': 'Invalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"'}]}]}}
2025-08-13 15:37:13,362 - ERROR - Traceback: Traceback (most recent call last):
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/instructor/retry.py", line 272, in retry_async
    response: ChatCompletion = await func(*args, **kwargs)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/instructor/client_genai.py", line 60, in async_wrapper
    return await client.aio.models.generate_content(*args, **kwargs)  # type:ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/google/genai/models.py", line 7810, in generate_content
    response = await self._generate_content(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/google/genai/models.py", line 6657, in _generate_content
    response = await self._api_client.async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/google/genai/_api_client.py", line 1296, in async_request
    result = await self._async_request(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/google/genai/_api_client.py", line 1241, in _async_request
    return await self._async_retry(  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 420, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 187, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/google/genai/_api_client.py", line 1182, in _async_request_once
    await errors.APIError.raise_for_async_response(response)
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/google/genai/errors.py", line 156, in raise_for_async_response
    raise ClientError(status_code, response_json, response)
google.genai.errors.ClientError: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"\nInvalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"\nInvalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"\nInvalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"', 'status': 'INVALID_ARGUMENT', 'details': [{'@type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'safety_settings[5].category', 'description': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"'}, {'field': 'safety_settings[6].category', 'description': 'Invalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"'}, {'field': 'safety_settings[7].category', 'description': 'Invalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"'}, {'field': 'safety_settings[8].category', 'description': 'Invalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"'}]}]}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/instructor/retry.py", line 267, in retry_async
    async for attempt in max_retries:
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 166, in __anext__
    do = await self.iter(retry_state=self._retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 421, in exc_check
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x78956d35f9b0 state=finished raised ClientError>]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/app/routers/gen_profile_router_gemini.py", line 83, in generate
    response = await self.client.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/instructor/client.py", line 606, in create
    return await self.create_fn(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/instructor/patch.py", line 163, in new_create_async
    response = await retry_async(
               ^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/.venv/lib/python3.12/site-packages/instructor/retry.py", line 302, in retry_async
    raise InstructorRetryException(
instructor.exceptions.InstructorRetryException: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"\nInvalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"\nInvalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"\nInvalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"', 'status': 'INVALID_ARGUMENT', 'details': [{'@type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'safety_settings[5].category', 'description': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"'}, {'field': 'safety_settings[6].category', 'description': 'Invalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"'}, {'field': 'safety_settings[7].category', 'description': 'Invalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"'}, {'field': 'safety_settings[8].category', 'description': 'Invalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"'}]}]}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/app/routers/gen_profile_router_gemini.py", line 130, in generate_medical_case_gemini
    generated_data = await generator.generate(input_data)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/digitalpatient/Code/Digital_Pacient_Deploy/app/routers/gen_profile_router_gemini.py", line 102, in generate
    raise ValueError(f"Error generating profile: {str(e)}")
ValueError: Error generating profile: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"\nInvalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"\nInvalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"\nInvalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"', 'status': 'INVALID_ARGUMENT', 'details': [{'@type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'safety_settings[5].category', 'description': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"'}, {'field': 'safety_settings[6].category', 'description': 'Invalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"'}, {'field': 'safety_settings[7].category', 'description': 'Invalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"'}, {'field': 'safety_settings[8].category', 'description': 'Invalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"'}]}]}}

INFO:    - "POST /api/v1/gen_profile HTTP/1.0" 500 Internal Server Error

Expected behavior A clear and concise description of what you expected to happen. Working Screenshots If applicable, add screenshots to help explain your problem.

lukaLLM avatar Aug 13 '25 14:08 lukaLLM

@claude can you investigate, might be filters that don't apply to the mode

jxnl avatar Aug 13 '25 16:08 jxnl

Claude finished @jxnl's task —— View job • claude/issue-1773-20250813-1651 • Create PR ➔

Issue Analysis Complete ✅

I successfully identified and fixed the Google GenAI safety settings issue that was causing INVALID_ARGUMENT errors.

Root Cause

The update_genai_kwargs function in instructor/providers/gemini/utils.py was missing:

  1. Fallback import logic for different google-genai package versions
  2. Robust filtering to prevent image-related harm categories from being sent to the API

Fix Applied

  • Added try/except import pattern with fallback to google.generativeai.types
  • Added double-check filtering to ensure HARM_CATEGORY_IMAGE_* categories never slip through
  • Improved robustness against different package versions and edge cases

Testing

The fix ensures that only valid harm categories are sent to the Google GenAI API, preventing the INVALID_ARGUMENT errors you were experiencing with image-related safety settings.

--- | Branch: claude/issue-1773-20250813-1651

claude[bot] avatar Aug 13 '25 16:08 claude[bot]

Is there a workaround for this at the moment?

bharris47 avatar Oct 29 '25 00:10 bharris47