spring-ai icon indicating copy to clipboard operation
spring-ai copied to clipboard

StreamingChatClient returns empty results with internal issue: MissingRequiredAnnotationException

Open radhakrishna67 opened this issue 10 months ago • 2 comments

StreamingChatClient (azure open ai) returns no response with Spring AI version 0.8.1 ChatClient (azure open ai) returns the response with the same Spring AI version 0.8.1

Bug description StreamingChatClient returns empty results with internal issue: MissingRequiredAnnotationException

Environment Spring Boot 3.2.4 Spring AI version: 0.8.1, Java version: 17.0.2 (Oracle)

Steps to reproduce

Expected behavior A clear and concise description of what you expected to happen.

Minimal Complete Reproducible example `var beanOutputParser = new BeanOutputParser<>(Demo.class);

String promptStr = """ you are an AI assistant to help with xyz products. If not sure, state that don't know. You have to provide {N_QUERIES} recommended queries for generated answer of the given query.

The response should be jackson JSON parsable without any special characters that should have user query, generated answer and {N_QUERIES} recommended queries. Use Bean parser to parse output to the java bean.

PREVIOUS CONVERSATION: {CONVERSATION} {query} {format}

""";

PromptTemplate promptTemplate = new PromptTemplate(promptStr, getReplacements(promptInput, beanOutputParser)); Prompt prompt = promptTemplate.create();

// with chatClient - this works as expected.

Generation generation = chatClient.call(prompt).getResult(); Demo demo = beanOutputParser.parse(generation.getOutput().getContent());

// with streamingChatClient - this returns empty string as response Mono<Demo> recommendationMono = streamingChatClient.stream(prompt) .map(chatResponse -> { Generation result = chatResponse.getResult(); AssistantMessage output = result.getOutput(); return beanOutputParser.parse(output.getContent()); }) .single();

private static Map<String, Object> getReplacements(PromptInput promptInput, BeanOutputParser<Demo> beanOutputParser) {
    return Map.of("N_QUERIES", promptInput.getNumberOfQueryRecommendations(),
            "query", promptInput.getQuery(),
            "CONVERSATION", "",
            "format", beanOutputParser.getFormat());
}
	
	

// Exception when using streamingChatClient com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input at [Source: (String)""; line: 1, column: 0] at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59) at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4916) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4818) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3772) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3740) at org.springframework.ai.parser.BeanOutputParser.parse(BeanOutputParser.java:113) at com.demo.service.impl.AiRecommenderServiceImpl.lambda$getContextRecommendedQueries$0(AiRecommenderServiceImpl.java:96) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113) at reactor.core.publisher.FluxIterable$IterableSubscription.fastPath(FluxIterable.java:402) at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:291) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) at reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103) at reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2909) at reactor.core.publisher.StrictSubscriber.onSubscribe(StrictSubscriber.java:77) at reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96) at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201) at reactor.core.publisher.FluxStream.subscribe(FluxStream.java:69) at reactor.core.publisher.Mono.subscribe(Mono.java:4568) at org.springframework.web.servlet.mvc.method.annotation.ReactiveTypeHandler$DeferredResultSubscriber.connect(ReactiveTypeHandler.java:465) at org.springframework.web.servlet.mvc.method.annotation.ReactiveTypeHandler.handleValue(ReactiveTypeHandler.java:163) at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitterReturnValueHandler.handleReturnValue(ResponseBodyEmitterReturnValueHandler.java:154) at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:78) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:136) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:925) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:830) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.base/java.lang.Thread.run(Thread.java:833) ` image

radhakrishna67 avatar Apr 08 '24 05:04 radhakrishna67

@radhakrishna67, can you, please, confirm if this problem is present in current 1.0.0-SNAPSHOT? (0.8.1 is a milestone and the fix with come with 1.0.0-SNAPSHOT and the forthcoming 1.0.0-M1 milestone.)

Also the MissingRequiredAnnotationException is not present in the stack you've shared.

tzolov avatar Apr 09 '24 17:04 tzolov

@tzolov , I am using spring-ai-azure-openai-spring-boot-starter-0.8.1, is a milestone.

MissingRequiredAnnotationException is only seen while debugging as shown in the screenshot.

radhakrishna67 avatar Apr 13 '24 06:04 radhakrishna67

@tzolov Just bringing this top of the list for up coming release

radhakrishna67 avatar Apr 15 '24 17:04 radhakrishna67

MissingRequiredAnnotationException may not be an issue. I think it happens because IDEA tries to invoke the toString method for debugging purpose.

In your case, you are trying to use BeanOutputParser to parse the JSON output from LLM. In the streaming mode, each ChatResponse only has partial content of the whole JSON data, so the JSON parsing will fail.

alexcheng1982 avatar Apr 17 '24 00:04 alexcheng1982

It would be helpful @radhakrishna67 if you can try 1.0 snapshots, but as @alexcheng1982 mentions this maybe expected behavior in this specific environment

markpollack avatar Apr 17 '24 02:04 markpollack

Closing, please reopen if there are any issues with the current code base. Seems like in this case a partial response of json was used to create a new bean resulting in an error.

markpollack avatar Jul 22 '24 20:07 markpollack