phoenix
phoenix copied to clipboard
Arize-ai
[ENHANCEMENT] An LLM Dashboard for latency, token usage, and cost
We are launching an LLM project, and I really love Phoenix given the tracing capability and clear UI. One thing we feel that would prevent us from adopting it is a aggregated stats tab as well as the alerts (e.g., connect to Slack) for each project (or multiple project). This is crucial for a production level LLM project.
Alternatively, it there could be a pointer to how I could implement such dashboard on my own would also be great. We are willing to contribute.
Something such as the following screenshots would be very helpful.
@hypernovas Glad you are liking Phoenix! Providing dashboard and monitoring capabilities is definitely on our radar but we are currently working on some infrastructure work that's foundational so we probably won't get to this as soon as you would like.
A few options:
-
try out Arize the saas platform. The screenshots you have are from those. They are quite powerful with things like custom metrics that you might like. They also already have built in monitoring.
-
this option is kinda nuts but if you would like to pull data from Phoenix, it does have a GraphQL api and you could at least see if you could stand up retool or graphana. I can't guarantee the stability of the API at the current juncture so this one is sorta admittedly a long shot
Sorry I can't be of more help. Keep the enhancement requests coming though! Would love to churn out features that unblock you.
Oh man, I'm so sorry I know I asked you to file this issue @hypernovas . Thanks again!
Oh man, I'm so sorry I know I asked you to file this issue @hypernovas . Thanks again!
Thanks for letting me know! You guys did great job.