Gemini 2.0 Flash structured output value repetition and missing fields

[Yiling-J]

trafficstars

Description of the bug:

• When using Gemini 2.0 Flash with structured output, the model often repeats values indefinitely until it reaches the token limit.
• When using Gemini 2.0 Flash with structured output, some expected fields are not generated in the output sometimes.

Schema Used:

{
  "type": "object",
  "properties": {
    "data": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "UkLWZg": {
            "type": "string"
          },
          "gbHJdm": {
            "type": "string"
          },
          "EfhxLZ": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "VqXmZF": {
            "type": "string"
          },
          "uw2YK1": {
            "type": "integer"
          }
        }
      }
    }
  }
}

Prompt Used:

Give me 10 new rows for the table in JSON array format.  
<TableDescription>table of recipes</TableDescription>  
Generate values for the following missing columns:  
<MissingColumns>  
  <Column id="UkLWZg" name="Name" description="recipe name" type="string" >  
  <Column id="gbHJdm" name="Country" description="country" type="string" >  
  <Column id="EfhxLZ" name="Ingredients List" description="The list of ingredients for the recipe." type="array" >  
  <Column id="VqXmZF" name="Instructions" description="The instructions for preparing the recipe." type="string" >  
  <Column id="uw2YK1" name="Preparation Time" description="The total time required to prepare the recipe, in minutes." type="integer" >  
</MissingColumns>

Actual vs expected behavior:

Repeated Values:

Missing Fields:

Using the same schema and prompt, GPT-4o generates a structured response without value repetition or missing fields. I didn't test o3 mini or other OpenAI models due to OpenAI's restrictions. This report is based on tests using the GPT-4o model on GitHub Models.

Any other information you'd like to share?

No response

[redwraith2]

I have also experienced missing fields using a relatively simple schema for a list of multiple choice quiz questions.

class QuestionAnswerSet(BaseModel): question: str = Field(..., description="The quiz question based on the input image.") correct_answer: str = Field(..., description="The correct answer to the quiz question based on the input image.") false_answers: List[str] = Field(..., description="A list of exactly three false answers.")

class QuestionAnswerList(BaseModel): RootModel: List[QuestionAnswerSet]

[felixvor]

We had this exact problem in a few-shot document classification/entity extraction use case (with Gemini 1.5-pro). For now, we were able to fix it by downgrading google-cloud-aiplatform:

pip install "google-cloud-aiplatform==1.69.0" --force-reinstall

We can reproduce the error reliably in sandboxed environments by switching the package versions back and forth - from looping text output until token limit is reached (latest version) to straight forward correct output (version 1.69.0). Sadly, the few shot documents used to create this problem are proprietary and I can't share them. I really can't imagine what the package is doing to influence the model outputs this badly, but that's what we're working with...

Maybe this solves the issue for anyone else who has this problem!

[VladimirArustamian]

I have the same problem with structured output both in gemini-1.5-pro-002, but it's even worse in gemini-flash-2.0 It basically makes the new Flash model unusable for me. I also see a tendency to go into repetition when I feed it German language.

[KelSolaar]

@felixvor : Out of curiosity, do you know what could be the difference between 1.69.0 and whatever version you had before?

[felixvor]

@felixvor : Out of curiosity, do you know what could be the difference between 1.69.0 and whatever version you had before?

No idea, I checked the changelogs but they gave me no clue insofar they seemed to have nothing to do with the way the model generates text.

We are currently experimenting using the latest version again in combination with frequency penalty and presence penalty. Maybe that also helps in your case!

Update: The problem now appears with "google-cloud-aiplatform==1.69.0" as well and can easily be reproduced. So either I got extremely lucky in all my previous back and forth "version-switch" experiments, or they did some further changes in the backend that now break our downgrade fix.

[felixvor]

I found a solution 🙌

Use the required attribute in your response schema and include the names of all fields from your schema (Docs). Then use a keyword like the string "<NA>" in your prompt or empty lists in your schema to make the model respond to optional fields correctly. In our case this stopped the model from generating repeating sequences.

{
  "type": "object",
  "properties": {
    "data": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "UkLWZg": {
            "type": "string"
          },
          "gbHJdm": {
            "type": "string"
          },
          "EfhxLZ": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "VqXmZF": {
            "type": "string"
          },
          "uw2YK1": {
            "type": "integer"
          }
        }
      }
    }
  },
  "required": ["UkLWZg", "gbHJdm", "EfhxLZ", "VqXmZF", "uw2YK1"]
}

I hope this helps others who encounter this problem!

[Yiling-J]

@felixvor Thank you for your suggestion! I’ve added the required field and tested it 10 times. Everything works as expected, and no errors were encountered. I do feel the quality difference with/without the required field is a bit dramatic. It seems the issue has already been triaged, so let’s wait for an official recommendation/reply from the Gemini team.

[karayanni]

Same issue here. we use Gemini 2.0 Flash - apparently this problem has been faced across other models in the past: GPT for instance

I will go ahead and try the Required attributed - thanks @felixvor

We currently use the Python data classes for structuring:

class SomeObject(BaseModel):
    field_1: str
    field_2: str
    field_3: int

class ListOfFirstObject(BaseModel):
    objects_list: list[SomeObject]

Is there a way to add the required attribute using the python data classes? or should we switch to a hard coded schema?

[karayanni]

@markmcd Thought this might be worth taking a look at. Thanks

[felix-vorlaender-mittelstand-ai]

@karayanni Try to use pydantic Fields. You can use the ... (Ellipsis) to explicitly mark them as "required".

from pydantic import BaseModel, Field
class SomeObject(BaseModel):
    field_1: str = Field(...) 

Let me know if that improves the model output!

[markmcd]

class SomeObject(BaseModel):
    field_1: str
    field_2: str
    field_3: int

class ListOfFirstObject(BaseModel):
    objects_list: list[SomeObject]

Is there a way to add the required attribute using the python data classes? or should we switch to a hard coded schema?

The SDK does mark them as required by default. Using your schema:

>>> from google.genai import _transformers
>>> pprint(_transformers.t_schema(client, SomeObject).to_json_dict())
{'properties': {'field_1': {'type': 'STRING'},
                'field_2': {'nullable': True, 'type': 'STRING'},
                'field_3': {'type': 'INTEGER'}},
 'required': ['field_1', 'field_2', 'field_3'],
 'type': 'OBJECT'}

I'm not sure if there's a way to turn it off though (Optional fields are represented as nullable). If you want to request an SDK FR you can file it over on the SDK repo.

[karayanni]

Thanks @markmcd and @felix-vorlaender-mittelstand-ai

It appears that the default behavior with the required is not solving the structured output edge case with infinite loop.

We are adding a retry with variants of the prompt to overcome this problem.. but it seems like there is a fundamental problem with the LLMs structured output flow

If there are any additional tips / future improvements would love to be updated to figure out better approaches/solutions.

[karayanni]

@Yiling-J did you figure out a solution?

This is making Gemini 2 flash not usable as almost 20% of the calls fail and waste so much tokens.. Just infinite stream of \n\n\n\n\n\n\n\

Our team now has a fall back strategy with 3 different prompts - with few shot examples, when it fails, it fails on all 3 different prompts!

FYI - @markmcd please let us know if there are any updates - we would much appreciate it

[Yiling-J]

@karayanni Adding the required field works for me, before doing so, around 80% of my calls would fail—either due to loops or missing fields. After explicitly adding the required field, I haven’t encountered any errors. The only remaining issue for me is the noticeable quality difference between calls with and without the required field.

I actually encountered errors using the OpenAI Go SDK, not the Python Gemini SDK. But I was able to reproduce it using the Gemini web as I reported in this issue. After applying the fix, both the SDK and web are now working as expected.

[xylophone21]

@karayanni Adding the required field works for me, before doing so, around 80% of my calls would fail—either due to loops or missing fields. After explicitly adding the required field, I haven’t encountered any errors. The only remaining issue for me is the noticeable quality difference between calls with and without the required field.

I actually encountered errors using the OpenAI Go SDK, not the Python Gemini SDK. But I was able to reproduce it using the Gemini web as I reported in this issue. After applying the fix, both the SDK and web are now working as expected.

do you add required filed by code? can you give a example?

class SomeObject(BaseModel): field_1: str field_2: str field_3: int

class ListOfFirstObject(BaseModel): objects_list: list[SomeObject]

[Yiling-J]

@xylophone21 According to https://github.com/google-gemini/cookbook/issues/449#issuecomment-2665193768, the google.genai SDK should automatically add the required fields. You can try printing the schema to verify. In my case, I do manually add the required fields in code since I'm working with a dynamic schema (not from an existing class or struct) but I'm using Go with the OpenAI Go SDK ,so unfortunately, I can’t offer much help with your Python question.

[Yiling-J]

off-topic: I'm working on this tool and encountered the issue while developing it: https://github.com/Yiling-J/tablepilot. I think it's an interesting project, feel free to check it out if you're interested

[shresbm]

Thanks @markmcd and @felix-vorlaender-mittelstand-ai

It appears that the default behavior with the required is not solving the structured output edge case with infinite loop.

We are adding a retry with variants of the prompt to overcome this problem.. but it seems like there is a fundamental problem with the LLMs structured output flow

If there are any additional tips / future improvements would love to be updated to figure out better approaches/solutions.

Could yours be a property ordering issue - https://ai.google.dev/gemini-api/docs/structured-output?lang=python#property-ordering

[marcodgiudice]

Hi, I have this exact issue using structured outputs with gemini-2.0-flash. Even adding the "required" field does not solve the issue. Has anyone any other suggestions? Thanks!

[alexpeys]

gemini-2.0-flash deathlooping on structured outputs is an issue for us as well, trying with frequency penalty as a fix but that is a bandaid!

[AlbertInRC]

Hi @alexpeys @marcodgiudice , I have the issue as well. To mitigate this issue, does retry the LLM call solve this issue on your side? Thanks

[HT2Knock]

Current experience same issue of looping in structured output with gemini 2.0 flash seems like there just aren't any reliable way to resolve this right now.

[petervanwylen]

I agree that there's no way to get Gemini 2.0 flash to reliably handle structured generation. Have to stick with Flash 1.5 if you want it to work correctly.

[hugbubby]

I am using Pydantic models which (should) add the required field automatically, and am nevertheless getting back models with missing fields.

[karayanni]

For me what solved it is cleaning the text - removing non alphabet or numeric chars before sending to the LLM.

[CodeofDutyAI]

Same issue here.. And I could not reproduce using python genai package, but can easily reproduce via rest api

[yorickvP]

I'm seeing this on 2.5 flash as well. It seems that the constrained decoding can't do unicode escapes, and the model gets in a loop.

Trying to have it generate {"response": "\u00ED\u00ED"} does not work. Another example would be: { "face": "\uD83D\uDE10" }

[Gunand3043]

I suggest trying our advanced 2.5-pro model, as it appears to resolve the issue. I just tested it on my end, and it worked as expected.

[Yiling-J]

My test case (and other similar tests that generate different kinds of table data) is working with Gemini 2.0 Flash after I added the required field. A lot of comments mention they have the same issue, but very few include a reproducible example. I’m not sure whether I should close the issue, but if the Gemini team considers this resolved, please feel free to close it.