streaming requests with tools may return a 500 error under certain prompts

[douglarek]

Description of the bug:

package main

import (
        "context"
        "fmt"
        "log"

        "github.com/google/generative-ai-go/genai"
        "google.golang.org/api/iterator"
        "google.golang.org/api/option"
)

func main() {
        ctx := context.Background()
        client, err := genai.NewClient(ctx, option.WithAPIKey("your-api-key"))
        if err != nil {
                log.Fatal(err)
        }
        defer client.Close()

        schema := &genai.Schema{
                Type: genai.TypeObject,
                Properties: map[string]*genai.Schema{
                        "location": {
                                Type:        genai.TypeString,
                                Description: "The city and state, e.g. San Francisco, CA",
                        },
                        "unit": {
                                Type: genai.TypeString,
                                Enum: []string{"celsius", "fahrenheit"},
                        },
                },
                Required: []string{"location"},
        }

        weatherTool := &genai.Tool{
                FunctionDeclarations: []*genai.FunctionDeclaration{{
                        Name:        "CurrentWeather",
                        Description: "Get the current weather in a given location",
                        Parameters:  schema,
                }},
        }

        model := client.GenerativeModel("gemini-1.5-flash-latest")
        model.Tools = []*genai.Tool{weatherTool}
        session := model.StartChat()

        iter := session.SendMessageStream(ctx, genai.Text("write a python function that takes a list of integers and returns the sum of the list")) // googleapi: Error 500:
        //iter := session.SendMessageStream(ctx, genai.Text("hello")) // ok
        if err != nil {
                log.Fatal(err)
        }

        for {
                res, err := iter.Next()
                if err == iterator.Done {
                        break
                }
                if err != nil {
                        log.Fatal(err)
                }
                printResponse(res)
        }

}

func printResponse(resp *genai.GenerateContentResponse) {
        for _, cand := range resp.Candidates {
                if cand.Content != nil {
                        for _, part := range cand.Content.Parts {
                                fmt.Println(part)
                        }
                }
        }
        fmt.Println("---")
}

Actual vs expected behavior:

when use iter := session.SendMessageStream(ctx, genai.Text("write a python function that takes a list of integers and returns the sum of the list")):

2024/05/15 15:09:45 googleapi: Error 500:
exit status 1

if use iter := session.SendMessageStream(ctx, genai.Text("hello")):

Hello
---
! How can I help you today? 😊 

---

Any other information you'd like to share?

go mod: github.com/google/generative-ai-go v0.12.0

[jba]

Thanks for the report. This repo holds the Go Gemini client, but the client doesn't control the service's responses. Try asking at https://discuss.ai.google.dev.

[douglarek]

I have already posted the issue to the forum address above. What surprises me is that this forum is supposed to be a place where users post questions, but there isn't a single staff member seriously addressing the issues raised by users. Honestly, it's appalling. This is not at all what a former giant like Google should be presenting. I know venting emotions in an issue is not appropriate, but truth be told, I cannot tolerate the façade of caring about these issues while they persist unresolved.

https://discuss.ai.google.dev/t/streaming-requests-with-tools-may-return-a-500-error-under-some-prompts-118/2216

[eliben]

@douglarek we're sorry this is taking a while to resolve. There's an internal issue opened about it and the team is looking at it along with other issues.

Even though this isn't really specific to the Go SDK, we can keep the issue open here for now as well.

[MarkDaoust]

Hi, we were just retrying this example. And I can't replicate the 500 error, I get a lot of FinishReasonRecitations, but that's working as intended. I think this is fixed.