gin
gin copied to clipboard
gin-gonic
The stream is not response as a
Description
I'm using the stream to response gptchat result to front.
My Demo like :
func ChatWithGpt(c *gin.Context) {
var req []UserMessages
if err := c.BindJSON(&req); err != nil || len(req) == 0 {
return
}
messages := make([]client.ChatMessage, 0)
question := ""
for index := range req {
qa := req[index]
if index == len(req)-1 {
// the last text is user question
if strings.TrimSpace(qa.UserText) != "" {
question = strings.TrimSpace(qa.UserText)
}
break
}
if strings.TrimSpace(qa.UserText) != "" {
messages = append(messages, client.ChatMessage{
Role: openai.ChatMessageRoleUser,
Content: strings.TrimSpace(qa.UserText),
})
} else if strings.TrimSpace(qa.InitText) != "" {
messages = append(messages, client.ChatMessage{
Role: openai.ChatMessageRoleAssistant,
Content: strings.TrimSpace(qa.InitText),
})
}
}
log.Infof("msg: %v", messages)
chanStream := make(chan client.ChatResponse, 500)
stream, err := client.ChatCompletionWithStream(
context.Background(),
client.ChatRequest{
History: messages,
TopK: 1,
Temperature: 0.7,
ScoreThreshold: 1.0,
Stream: true,
Question: question,
PromptName: "text",
ModelName: setting.Aimodel,
},
)
if err != nil {
c.JSON(400, gin.H{
"anwser": "error",
})
return
}
go func() {
defer stream.Close()
defer close(chanStream)
for {
var response client.ChatResponse
response, err = stream.Recv()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
fmt.Printf("Stream error: %v\n", err)
break
}
chanStream <- response
}
}()
c.Header("Content-Type", "text/event-stream")
c.Header("Cache-Control", "no-cache")
c.Header("Connection", "keep-alive")
c.Stream(func(w io.Writer) bool {
if msg, ok := <-chanStream; ok {
if msg.Answer != "" {
m := gin.H{
"answer": msg.Answer,
"docs": msg.Docs,
"tools": msg.Tools,
"answerstep": msg.AnswerStep,
}
jcontent, _ := json.Marshal(m)
w.Write(jcontent)
} else {
m := gin.H{
"docs": msg.Docs,
"tools": msg.Tools,
"answerstep": msg.AnswerStep,
}
jcontent, _ := json.Marshal(m)
w.Write(jcontent)
}
log.Infof("message: %v\n", msg)
return true
}
return false
})
}
The response not result as a text/event-stream .It's return all the results at once.
I'm also test using c.SSEvent("message", msg), It's also returned all the results at once.
Actual result
returned all the result at once.
Environment
- go version:1.20
- gin version (or commit ref):1.9.1
- operating system: CentOS 7
Looks like chanStream received response (chanStream <- response in Goroutine) so quickly that like Gin return all results at once. How about add one line time.Sleep(time.Duration(1) * time.Second) after chanStream <- response?
