genkit icon indicating copy to clipboard operation
genkit copied to clipboard

[Dev UI] Thinking config not behaving as expected

Open MichaelDoyle opened this issue 7 months ago • 0 comments

Note: Changes likely span both framework and Dev UI

After https://github.com/firebase/genkit/pull/2851. was merged, I am seeing a few problems:

  1. "thought" parts are mixed in with all of the other text parts. The Gemini API seems to handle this with the introduction of a new boolean "thought". That boolean is not present in our responses that Genkit is getting back from the SDK. I am looking at the "raw" response from Gemini in the output to verify. I suspect this is due to our use of the legacy SDK.

    TL;DR This is not great, but the developer has to opt in to thought tokens, so its not super impactful yet, but will be confusing.

  2. Dev UI always displays thought tokens for Gemini 2.5, as long as there is a thoughtConfig object present, no matter what the settings are. This is a definite regression.

    My understanding from the docs are:

    1. Gemini has thinking in the background enabled by default, controlled by the budget. This costs money, and so you can stop it from thinking, even in the background, by setting the budget to 0.

    2. You can ask the API/SDK to give you the above thought tokens as an optional feature.

      In other words, the default behavior should be such that (a) thought tokens ARE used in the background and (b) not returned by the SDK, or displayed by the Dev UI.

    This may end up being a bit of work specifically for budget, because displaying/setting a "default" budget of 0 is incorrect, but its not quite clear if the "max" value should be the default either. There needs to be a way for it to be "unset".

  3. Dev UI shows thought config even for models that do not support it.

    Image

MichaelDoyle avatar May 07 '25 02:05 MichaelDoyle