OpenRouter using Gemini Flash model results in Pydantic Validation Error on Nested Object Models

[dmastylo]

• [X] This is actually a bug report.
• [ ] I have tried asking for help in the community on discord or discussions and have not received a response.

What Model are you using?

• [X] Other (please specify) google/gemini-2.0-flash-001 through OpenRouter

Describe the bug Instructor with OpenRouter using Gemini model fails to validate response models with nested objects

Traceback (most recent call last):
  File "/Users/dmastylo/.pyenv/versions/3.12.1/envs/filing-automation/lib/python3.12/site-packages/instructor/retry.py", line 191, in retry_sync
    raise e
  File "/Users/dmastylo/.pyenv/versions/3.12.1/envs/filing-automation/lib/python3.12/site-packages/instructor/retry.py", line 174, in retry_sync
    return process_response(  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dmastylo/.pyenv/versions/3.12.1/envs/filing-automation/lib/python3.12/site-packages/instructor/process_response.py", line 170, in process_response
    model = response_model.from_response(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dmastylo/.pyenv/versions/3.12.1/envs/filing-automation/lib/python3.12/site-packages/instructor/function_calls.py", line 266, in from_response
    return cls.parse_tools(completion, validation_context, strict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dmastylo/.pyenv/versions/3.12.1/envs/filing-automation/lib/python3.12/site-packages/instructor/function_calls.py", line 580, in parse_tools
    return cls.model_validate_json(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dmastylo/.pyenv/versions/3.12.1/envs/filing-automation/lib/python3.12/site-packages/pydantic/main.py", line 625, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 2 validation errors for DateContextData
relevant_dates.0
  Input should be an object [type=model_type, input_value='{"start_date": "2024-07-...ncomplete_date": false}', input_type=str]
    For further information visit https://errors.pydantic.dev/2.9/v/model_type
relevant_dates.1
  Input should be an object [type=model_type, input_value='{"start_date": "2023-07-...ncomplete_date": false}', input_type=str]
    For further information visit https://errors.pydantic.dev/2.9/v/model_type

To Reproduce

import os

import httpx
import instructor
import instructor.exceptions
from openai import OpenAI
from pydantic import BaseModel, Field


OPENROUTER_API_KEY = os.environ.get("OPENROUTER_API_KEY")
OPENROUTER_CLIENT = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=OPENROUTER_API_KEY,
    timeout=httpx.Timeout(90),
)

client = instructor.from_openai(OPENROUTER_CLIENT, mode=instructor.Mode.TOOLS)


class DateContext(BaseModel):
    end_date: str
    start_date: str
    incomplete_date: bool


class DateContextData(BaseModel):
    chain_of_thought: str = Field(description="Let's think step by step")
    relevant_dates: list[DateContext]


# response = client.chat.completions.create(
#     model="google/gemini-2.0-flash-001",
#     response_model=DateContextData,
#     messages=[
#         {
#             "role": "system",
#             "content": f"Extract dates from the given text.",
#         },
#         {"role": "user", "content": "I had the car from Jan 1 2024 to Jan 12 2025"},
#     ],
#     extra_body={"provider": {"require_parameters": True}},
#     strict=False,
#     # retry_if_exception_type=instructor.exceptions.InstructorRetryException,
#     max_retries=3,
# )

# print(response)

from google import genai

GOOGLE_API_KEY = os.environ.get("GOOGLE_API_KEY")
GOOGLE_TIMEOUT = 1 * 90 * 1000  # 1.5 minutes
GOOGLE_CLIENT = (
    genai.Client(api_key=GOOGLE_API_KEY, http_options={"timeout": GOOGLE_TIMEOUT})
    if GOOGLE_API_KEY
    else None
)
client = instructor.from_genai(
    GOOGLE_CLIENT,
    mode=instructor.Mode.GENAI_TOOLS,
    use_async=False,
)

response = client.chat.completions.create(
    model="gemini-2.0-flash",
    response_model=DateContextData,
    messages=[
        {
            "role": "system",
            "content": f"Extract dates from the given text.",
        },
        {"role": "user", "content": "I had the car from Jan 1 2024 to Jan 12 2025"},
    ],
    # retry_if_exception_type=instructor.exceptions.InstructorRetryException,
    max_retries=3,
)
print(response)

Note the genai way works, the OpenRouter way does not.

Using JSON mode with OpenRouter works, but would prefer TOOLS.

Expected behavior This model works normally with the genai provider

Screenshots N/A

[philmas]

I am running into the same issue. Have you found a solution?

[dmastylo]

@philmas I am using JSON mode as a workaround. It works but parsing failure rate over thousands of iterations has gone up a noticeable, but not unacceptable amount.

[philmas]

@philmas I am using JSON mode as a workaround. It works but parsing failure rate over thousands of iterations has gone up a noticeable, but not unacceptable amount.

I decided to not use Openrouter and instead use integrations that do work. Possibly, if I find time, I'll see if I can find a way to solve the issue, if not already worked on.

[qwertyuu]

hello all! I've faced the same issue, and I fixed it by passing

mode=instructor.Mode.OPENROUTER_STRUCTURED_OUTPUTS

to instructor.from_provider. Then it started working like a charm!

[philmas]

hello all! I've faced the same issue, and I fixed it by passing

mode=instructor.Mode.OPENROUTER_STRUCTURED_OUTPUTS

to instructor.from_provider. Then it started working like a charm!

Are you sure that that brings the same (underlying) functionality?

(Sure I can take the bus to my destination, but if I plan to go by car, I rather find ways to repair)