getsentry
Support LLM monitoring in Ruby SDK.
Describe the idea Support LLM monitoring in Ruby SDK.
Why do you think it's beneficial to most of the users
Implementing LLM monitoring in the Ruby SDK allows developers to gain deeper insights into the performance and behavior of large language models in their applications. Users can track the health of their models, detect anomalies, and ensure optimal functioning, which can lead to enhanced user experiences and increased trust in AI-driven features. By providing detailed monitoring, users can make informed decisions on how to iterate on their models, improving accuracy and responsiveness while minimizing downtime.
Possible implementation
To implement LLM monitoring in the Ruby SDK, we can follow a structured approach:
Integrate Monitoring Hooks: Introduce SDK hooks that enable developers to easily add monitoring functionalities at critical points in the model's lifecycle, including initialization, inference, and error handling.
Capture Metrics: Create predefined metrics for automatic capture, such as response times, error rates, input/output token counts, and model latency. Additionally, allow users to define custom metrics relevant to their use cases.
Anomaly Detection: Implement real-time algorithms to analyze captured metrics for anomalies, such as unexpected spikes in error rates or response times. Notify developers when anomalies are detected.
Dashboard and Visualization: Develop a user-friendly dashboard that visually displays collected metrics and anomalies, providing insights into model performance over time to help developers identify trends and improvement areas.
Documentation and Examples: Offer comprehensive documentation and practical examples for enabling and using LLM monitoring within the SDK, including step-by-step guides, code snippets, and best practices for seamless integration.
Community Feedback Loop: Create a feedback mechanism for users to share their experiences and suggest improvements, such as a dedicated forum or feedback section in the documentation, allowing the SDK to evolve based on real-world usage.
Thanks for the request @khaile! We typically add new feature support based on existing popular gems in the ecosystem. Do you have any particular LLM gems/libraries you would like to see support for?
Do you have any particular LLM gems/libraries you would like to see support for?
Hi @sl0thentr0py , I would love to see if we can support for langchainrb!
Here are some other gems in the LLM space for Ruby:
- https://docs.activeagents.ai/ (high-level framework)
- https://rubyllm.com/ (high-level framework)
- https://github.com/patterns-ai-core/langchainrb (high-level framework)
- https://github.com/alexrudall/ruby-openai (OpenAI Ruby SDK)
- https://github.com/alexrudall/ruby-anthropic (Anthropic Ruby SDK)
- https://github.com/gbaptista/gemini-ai (Gemini Ruby SDK)
As for Sentry SDK, I guess you would probably just expose the relevant methods on the SDK, and then it would be up to the application or gem author [of one of the above mentioned gems] to implement the Sentry instrumentation calls?
Another approach could be to reach out to these gem authors and suggest they all standardise on a set of notification events with standardised metadata (like token count etc) and add this to their gems, for example via ActiveSupport::Notification. It would then be up to your Ruby SDK to subscribe to these notifications.
ping @crmne @TonsOfFun
