Inconsistent results from GPT with temperature = 0

[gronka]

I am getting a wide variety of results from GPT, but my temperature is 0. I'm using a custom-trained model. I contacted GPT/openai, but their customer support seems to be inadequate.

req := gogpt.CompletionRequest{
	Model:       "davinci:ft-text-friday-2022-10-02-03-58-57",
	MaxTokens:   100, // 16 is default
	Prompt:      prompt,
	Temperature: 0,
	//TopP:             1,
	//FrequencyPenalty: 0,
	//PresencePenalty:  0,
	Stop: []string{" ->", ">>>"},
}

Example 1

 TRC Respond in JSON. I want to buy water. ->
8:02AM TRC GPT 5
8:02AM TRC GPT 6
8:02AM TRC GPT 6.a
8:02AM TRC returned: null
We'll update the last message sent to match the reply, and we'll update the request.id to say:
{ 'Store': 'water', 'Category': 'latest', 'Message': 'I want to buy water', 'Date': '2016-09-21T10:52:13', 'RequestId': '20', 'Reply': 'null', 'ArrivalDate': '2016-09-21T10:52:13'}

Example 2, sent seconds later

8:02AM TRC Respond in JSON. I want to buy water. ->
8:02AM TRC GPT 5
8:02AM TRC GPT 6
8:02AM TRC GPT 6.a
8:02AM TRC returned: {'Function': 'contact_support', 'Noun': 'water', 'Adjectives': [], 'Category': '', 'ClothingType': '', 'Quantity': 0, 'ArrivalDate': '', 'Store': '', 'MinPrice': 0, 'MaxPrice': 0, 'Multiple': False}

[sauceman40]

@gronka I ran into this same problem a few months back when creating a fine-tuned classifier. If it helps you, they let me know at the time that GPT-3 is not fully deterministic even when setting Temperature = 0.

[nhobin219]

@gronka in case you or anyone else is still wondering about this, the reason this happens is that all of the numeric completion request settings are values (not pointers) in the CompletionRequest struct. You can't actually pass 0 as a numeric value otherwise it will get removed during json serialization by the encoding/json library as specified by the omitempty tag. Then, when no temperature is provided in the request, the model defaults to a temperature value of 1. This is actually pretty frustrating, but you can more or less get around it for now by passing an extremely small temperature value e.g. 0.0001. The correct fix would be to redefine all of those internal structs using pointers instead of values as PullRequestInc does in their GPT-3 client https://github.com/PullRequestInc/go-gpt3/blob/main/models.go#L91.

[nhobin219]

Ah, I think this can be closed as a duplicate of https://github.com/sashabaranov/go-openai/issues/9 actually.

[vvatanabe]

This is a duplicate issue with https://github.com/sashabaranov/go-openai/issues/9, so please discuss it in https://github.com/sashabaranov/go-openai/issues/9 from now on.