'JsonEOFException: Unexpected end-of-input: expected close marker for Object' when making synchronous openAiChatClient.call

[dolukhanov]

trafficstars

Bug description When making a basic synchronous openAiChatClient.call( prompt) with a single UserMessage, an Exception is thrown (com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1]))

I have inspected the traffic using Charles, and the OpenAI API is returning a correct JSON response.

Environment Please provide as many details as possible: Spring AI version, Java version, which vector store you use if any, etc OpenJDK 18.0.2 Spring Framework 6.1.4 Spring Boot 3.2.3 Spring Open AI 0.8.0-SNAPSHOT

Steps to reproduce Basic synchronous call

String message = "Tell me about OpenAI";
UserMessage userMessage = new UserMessage( message );
List<Message> openAiMessages = List.of( userMessage );
Prompt prompt = new Prompt( openAiMessages );
ChatResponse call = openAiChatClient.call( prompt );
Generation generation = call.getResults().get( 0 );
System.out.println( generation.getOutput().getContent() );

Expected behavior Chat response is printed.

Stack Trace

Caused by: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
  at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:406) ~[spring-web-6.1.4.jar:6.1.4]
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:354) ~[spring-web-6.1.4.jar:6.1.4]
	at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:213) ~[spring-web-6.1.4.jar:6.1.4]
	... 82 common frames omitted
Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
 at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 2]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:697) ~[jackson-core-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:512) ~[jackson-core-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:529) ~[jackson-core-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:3103) ~[jackson-core-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:757) ~[jackson-core-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:181) ~[jackson-databind-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) ~[jackson-databind-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2105) ~[jackson-databind-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1481) ~[jackson-databind-2.15.4.jar:2.15.4]
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:395) ~[spring-web-6.1.4.jar:6.1.4]
	... 84 common frames omitted

[dolukhanov]

I narrowed down the problem to the project having a dependencies on:

 <dependency>
            <groupId>org.apache.httpcomponents.client5</groupId>
            <artifactId>httpclient5</artifactId>
            <version>5.3.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.httpcomponents.client5</groupId>
            <artifactId>httpclient5-fluent</artifactId>
            <version>5.3.1</version>
        </dependency>

Once these are removed, the code functions as expected.

Leaving this issue open, should it require further investigation.

[markpollack]

I am not able to reproduce. Here is a working sample of your code. myai.zip

Are you using any ChatOptions configuration in your code? That is an area in the code path where we use Jackson, but if no special options are set, then jackson isn't involved in the code path. Maybe provide a longer stacktrace to localize it better.

[carlspring]

I am experiencing the same issue.

• org.springframework.ai:spring-ai-openai-spring-boot-starter:0.8.0
• OpenJDK: 21.0.2
• Spring Boot: 3,2,3
• Spring Framework: 6.1.4
• Jackson: 2.17.1

package com.carlspring.ai.openai;

import com.carlspring.ai.openai.config.OpenAITestConfiguration;
import org.junit.jupiter.api.Test;
import org.springframework.ai.autoconfigure.openai.OpenAiAutoConfiguration;
import org.springframework.ai.chat.ChatResponse;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.openai.OpenAiChatClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

/**
 * @author carlspring
 */
@SpringBootTest(classes = { OpenAITest.class,
                            OpenAiAutoConfiguration.class,
                            OpenAITestConfiguration.class })
public class OpenAITest
{

    @Autowired
    private OpenAiChatClient chatClient;

    @Test
    public void testBasic()
    {
        // Sync request
        ChatResponse response = chatClient.call(new Prompt("Is Linux the same as Unix?"));

        System.out.println(String.join(" ", response.getResults().stream().map((g) -> g.getOutput().toString()).toList()));
    }

}

package com.carlspring.ai.openai.config;

import org.springframework.context.annotation.Bean;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;

@Component
public class OpenAITestConfiguration
{
    
    @Bean
    public RestClient.Builder restClientBuilder()
    {
        return RestClient.builder();
    }
    
}

04:50:23.924 13-05-2024 | DEBUG | main                 | org.springframework.web.client.DefaultRestClient   | Writing [ChatCompletionRequest[messages=[ChatCompletionMessage[content=Is Linux the same as Unix?, role=USER, name=null, toolCallId=null, toolCalls=null]], model=gpt-3.5-turbo, frequencyPenalty=0.0, logitBias=null, maxTokens=null, n=1, presencePenalty=null, responseFormat=null, seed=null, stop=null, stream=false, temperature=0.7, topP=null, tools=null, toolChoice=null, user=null]] as "application/json" with org.springframework.http.converter.json.MappingJackson2HttpMessageConverter
04:50:26.018 13-05-2024 | DEBUG | main                 | org.springframework.web.client.DefaultRestClient   | Reading to [org.springframework.ai.openai.api.OpenAiApi$ChatCompletion]

org.springframework.web.client.RestClientException: Error while extracting response for type [org.springframework.ai.openai.api.OpenAiApi$ChatCompletion] and content type [application/json]

	at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:236)
	at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.readBody(DefaultRestClient.java:667)
	at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.toEntityInternal(DefaultRestClient.java:637)
	at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.toEntity(DefaultRestClient.java:626)
	at org.springframework.ai.openai.api.OpenAiApi.chatCompletionEntity(OpenAiApi.java:656)
	at org.springframework.ai.openai.OpenAiChatClient.chatCompletionWithTools(OpenAiChatClient.java:337)
	at org.springframework.ai.openai.OpenAiChatClient.lambda$call$1(OpenAiChatClient.java:151)
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:335)
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:211)
	at org.springframework.ai.openai.OpenAiChatClient.call(OpenAiChatClient.java:147)
	at com.carlspring.ai.openai.OpenAITest.testBasic(OpenAITest.java:30)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:406)
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:354)
	at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:213)
	... 13 more
Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
 at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 2]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:697)
	at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:512)
	at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:529)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:3103)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:757)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:181)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2105)
	at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1481)
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:395)
	... 15 more

It seems that the answers here might be related and that the JSON reponse needs to be read fully before trying to read it as an object.

[vwiencek]

I have the same problem and can't figure out how to solve it

[jocax]

I had the same issue and found a workaround by setting the required http header "Accept-Encoding". I applied the header in the Rest.Builder. One the header is set the response can be read without the above error.

@Bean
RestClient.Builder restClientBuilder() {
    return RestClient.builder()
            .defaultHeaders(new Consumer<HttpHeaders>() {
                @Override
                public void accept(HttpHeaders httpHeaders) {
                    // https://github.com/spring-projects/spring-ai/issues/372
                    httpHeaders.set("Accept-Encoding", "gzip, deflate");
                }
            });
}

Hope this helps.

[ThomasVitale]

Spring Boot provides an autoconfigured RestClient.Builder object, including JSON serialization configuration. Have you tried using that one instead of defining one yourself?

When you create a RestClient via RestClient.builder(), you don't get the Spring Boot autoconfiguration, so it's up to you to configure encoding and serialization. If you create one from an autowired RestClient.Builder bean, that is all configured for you and should work out-of-the-box.

[Alien2150]

So my workaround that is not that invasive as overwritting the default RestClient builder:

    @Bean
    fun chatModel(
        commonProperties: OpenAiConnectionProperties,
        chatProperties: OpenAiChatProperties,
        webClientBuilder: WebClient.Builder,
        retryTemplate: RetryTemplate,
        functionCallbackContext: FunctionCallbackContext,
        responseErrorHandler: ResponseErrorHandler,
    ): OpenAiChatModel {
        val restClientBuilder = RestClient.builder().defaultHeaders {
            it.set(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate")
        }

        val openAiApi = OpenAiApi(
            chatProperties.baseUrl ?: commonProperties.baseUrl,
            chatProperties.apiKey ?: commonProperties.apiKey,
            restClientBuilder,
            webClientBuilder,
            responseErrorHandler,
        )

        return OpenAiChatModel(
            openAiApi,
            chatProperties.options,
            functionCallbackContext,
            retryTemplate,
        )
    }

As a suggestion: Could the bean for RestClient builder not use a "Qualifier" annotation so it would not be overwritten by the default one? (OpenAiAutoConfiguration seems to use the default one)

[markpollack]

I was thining the same thing @Alien2150 in relation to setting timeouts per model, using a qualifier annotation that will pick up a specific restClientBuilder. It would seem that this problem would have been solved already with apps calling many microservices, each with different time outs... https://github.com/spring-projects/spring-ai/issues/512 and related issues discuss it more.