spring-ai
spring-ai copied to clipboard
spring-projects
Metrics/Observability
I was thinking it might be useful (or at least interesting) if Spring AI public metrics regarding token usage via the Actuator metrics endpoint. This information comes from the ChatResponse (formerly known as AiResponse) and (I presume) from other ???Response classes in the future when other types of interactions are supported. But I'd imagine that the best place to report it would be at the point that the response is received, that is in the various client implementations such as OpenAiClient.
Yes, this is a big feature set. Introducing a listener interface on the ChatClient is one such approach (similar to Spring Batch) as there can be many targets for this information. I did look into some of the AI specific options is this space based on the Emerging LLM Application Stack diagram and thought a listener approach would be useful to write data to these specific systems. Weights and Bias and helicone. Also, in the context of 'evaluation driven development', storing the entire set of information in order to evaluate drift/accuracy/hallucination is important as well.
I do like the idea of exporting some basic metrics via actuator, such as token usage when relevant, any perhaps other information that isn't voluminous, such as number of retries (when we add retry support), or latency times. Those might be good for micrometer style metrics that could also be expored by a Micrometer based implementation of a ChatClientListener.
I'll add this to 0.9.0 and see if we can get at least the basic interfaces/design working.
Also the idea of the LLMCache, Guardrails, and Playground from the a16z diagram are also of interest. Just a matter of time and prioritization.
In https://github.com/spring-projects/spring-ai/issues/953, I shared some initial thoughts and proposals for introducing observability in Spring AI. It would be great to get your feedback about it.
Closing this issue in favor of https://github.com/spring-projects/spring-ai/issues/953