spring-ai icon indicating copy to clipboard operation
spring-ai copied to clipboard

It seems that LLM struggle to comprehend JSON schemas that include Java-type definitions?

Open luckygqx opened this issue 7 months ago • 2 comments
trafficstars

for example

@Bean
public ChatClient chatClient(OllamaChatModel ollamaChatModel) {
    return ChatClient.builder(ollamaChatModel)
            .defaultAdvisors(new MessageChatMemoryAdvisor(new InMemoryChatMemory()))
            .build();
}
@Component
public class MatchPlanTool {
    @Tool(name = "matchPlan", returnDirect = true, description = "Analyze the user's travel plan, which includes departure location, destination, trip start time(in the user's timezone), and number of travelers (including adults and children).")
    public String matchPlan(Request request) {
        return request.desc();
    }

    @Data
    @JsonClassDescription("Analyze the user's travel plan service API request parameter")
    public static class Request {

        @JsonPropertyDescription("departure location")
        private String departure;

        @JsonPropertyDescription("destination")
        private String destination;

        @JsonPropertyDescription("trip start time")
        private String tripStartTime;

        @JsonPropertyDescription("number of adults")
        private Integer adultCount;

        @JsonPropertyDescription("number of children")
        private Integer childrenCount;

        public String desc() {
            return String.format("user need %s time,from %s  to %s,include adults %s,chlidren%s", tripStartTime, departure, destination, adultCount, childrenCount);
        }
    }
}
  @GetMapping(value = "chat")
  public String chat(@RequestParam String sessionId, @RequestParam String message) {
      return chatClient.prompt()
              .user(message)
              .advisors(i -> i.param(AbstractChatMemoryAdvisor.CHAT_MEMORY_CONVERSATION_ID_KEY, sessionId))
              .tools(matchPlanTool)
              .call()
              .content();
  }

This is the JSON schema generated by Spring AI before calling LLM

{
  "$schema" : "https://json-schema.org/draft/2020-12/schema",
  "type" : "object",
  "properties" : {
    "request" : {
      "type" : "object",
      "properties" : {
        "adults" : {
          "type" : "integer",
          "format" : "int32",
          "description" : "number of adults"
        },
        "children" : {
          "type" : "integer",
          "format" : "int32",
          "description" : "number of children"
        },
        "departure" : {
          "type" : "string",
          "description" : "departure location"
        },
        "destination" : {
          "type" : "string",
          "description" : "destination"
        },
        "tripStartTime" : {
          "type" : "string",
          "description" : "trip start time"
        }
      },
      "required" : [ "adults", "children", "departure", "destination", "tripStartTime" ],
      "description" : "Analyze the user's travel plan service API request parameter"
    }
  },
  "required" : [ "request" ],
  "additionalProperties" : false
}

It seems that LLM struggle to comprehend JSON schemas that include Java-type definitions? LLM response is this

{"request":{"departureLocation":"beijing","destination":"shanghai","numAdults":2,"numChildren":1,"tripStartTime":"2023-12-25T10:00:00+08:00"}}

look at this, field name is not match, Causing JSON deserialization failure。

Did I make a mistake somewhere?

luckygqx avatar Apr 16 '25 08:04 luckygqx