Provide prompt options on a call by call basis

[mhoshi-vm]

Our local OpenAI endpoint has a very small default max_tokens against the chat completion. We like to make it adjustable

Expected Behavior

Hoping this code.

https://github.com/spring-projects-experimental/spring-ai/blob/main/spring-ai-openai/src/main/java/org/springframework/ai/openai/client/OpenAiClient.java

Looks something like this

	private ChatCompletionRequest getChatCompletionRequest(String text) {
		List<ChatMessage> chatMessages = List.of(new ChatMessage("user", text));
		logger.trace("ChatMessages: ", chatMessages);
		ChatCompletionRequest chatCompletionRequest = ChatCompletionRequest.builder()
			.model(this.model)
			.temperature(this.temperature)
			.messages(List.of(new ChatMessage("user", text)))
                         .maxTokens(4000) /// ADD HERE
 			.build();
		logger.trace("ChatCompletionRequest: ", chatCompletionRequest);
		return chatCompletionRequest;
	}

Current Behavior

max tokens not adjustable

Context

[markpollack]

Yes. This is a big gap. I have a design idea and hope to implement this next week, after more docs are written.

Thanks for taking the time to create an issue.

BTW - I changed the title of the issue to be more generic.

[devondragon]

@markpollack I am working on another AiClient adapter, and very much want to support per call prompt options. I can do this my own way of course, but I'd prefer to align with whatever you've got in the works. Can you share your approach so I can align with it and hopefully have a smooth PR process later?

[markpollack]

Hi, A bit late, but we are working on this now. https://github.com/spring-projects/spring-ai/commits/model-options-spike has someting to look at but it isn't complete.

The basic design we are thinking of at the moment is to have a "PromptOptions" interface that acts as a marker interface. Then we will determine which "PromptOptions" are portable across all the providers. It is likely a short list, e.g. model and temperature. A builder for these portable prompt options will be available. For provider specific options, there will provider specific prompt options interfact that also implements the portable options interface. There will also be a builder for those provider specific options. The provider specific ChatClient implementation will first check for the prompt options properties that were passed at runtime and use those instead of the options that were created at bean instantiation time.

[markpollack]

We note support this. See ChatOptions and then there is a specific subclass for each of the supported chat models, e.g OpenAiChatOptions, AnthropicChatOptions etc.

It hasn't had a great deal of exposure in the wild, so if you could give it a try, that would be appreciated.