cloud.google.com/go/vertexai: nil pointer dereferencing GenerateContentResponseIterator

[galeone]

Hi, I just released a software (website) that uses the genai package and receives a streaming response from Gemini.

Sometimes, the iteration fails unexpectedly.

This is the project: https://github.com/galeone/fitsleepinsights

The relevant part is the chat.go file also (partially) reported below: https://github.com/galeone/fitsleepinsights/blob/main/app/chat.go

Environment

Docker container, go 1.22

Code

e.g.

var responseIterator *genai.GenerateContentResponseIterator = chatSession.SendMessageStream(ctx, genai.Text(builder.String()))
begin := true
marker := "begin"
for {
	// write to socket
	if responseIterator == nil {
		marker = "end"
		if err = websocketSend("\n", marker); err != nil {
			c.Logger().Error("responseIterator is nil")
		}
		break
	}
	response, err := responseIterator.Next()
	if err == iterator.Done {
		marker = "end"
		if err = websocketSend("\n", marker); err != nil {
			c.Logger().Error(err)
			continue
		}
		break
	}
	// ....

Expected behavior

After checking for responseIterator not being null, the call to .Next() should be safe. Moreover, I expect the iterator to never be nil, but always having a value, when done, it should always return iterator.Done and never be nil.

Actual behavior

Sometimes, the call to .Next() causes a panic for a nil pointer dereferencing. This is the relevant part of the callstack

[PANIC RECOVER] runtime error: invalid memory address or nil pointer dereference goroutine 381 [running]:
github.com/galeone/fitsleepinsights/app.NewRouter.Recover.RecoverWithConfig.func44.1.1()
	/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/recover.go:100 +0x165
panic({0xe69280?, 0x19e11a0?})
	/usr/local/go/src/runtime/panic.go:770 +0x132
cloud.google.com/go/vertexai/genai.joinContent(...)
	/go/pkg/mod/cloud.google.com/go/[email protected]/genai/client.go:295
cloud.google.com/go/vertexai/genai.joinCandidateLists({0xc000808288, 0x1, 0x1}, {0xc0008082e8, 0x1, 0x10beab8?})
	/go/pkg/mod/cloud.google.com/go/[email protected]/genai/client.go:268 +0x186
cloud.google.com/go/vertexai/genai.joinResponses(...)
	/go/pkg/mod/cloud.google.com/go/[email protected]/genai/client.go:254
cloud.google.com/go/vertexai/genai.(*GenerateContentResponseIterator).Next(0xc0009133b0)
	/go/pkg/mod/cloud.google.com/go/[email protected]/genai/client.go:181 +0x227
github.com/galeone/fitsleepinsights/app.NewRouter.ChatWithData.func30.2(0xc0009ac120)
	/go/src/fitsleepingishts/app/chat.go:199 +0xc1d

chat.go, line 199 is precisely

response, err := responseIterator.Next()

[codyoss]

cc @eliben @jba