Nick Stogner
Nick Stogner
Agree we should tackle this PR as a top priority! Great feedback, and a valid use case / expected UX.
Note: KubeAI already exposes a metrics endpoint with some boring metrics. We just need to add these to that endpoint. NOTE: Alex made a contribution to Lingo that can be...
We should do this with OpenTelemetry as that appears to be the way everything is heading.
Hey @alpe - I’ve started on this one already. I forgot to assign it. Would love a review from you though!
We should likely make this more generic: How do we want to surface how admins can customize Pod created by KubeAI?
I would prefer to avoid exposing Pod spec options on a field-by-field basis as we will trend towards replicating the entire Pod spec in the Model spec. Instead, I would...
> One example that wouldn't be as easy is figuring out the resource.requests.storage capacity to request for the PVC. The typical pattern here is to match what is on the...
KubeAI exposes some very basic metrics. Some of the backend engines like vLLM expose more granular metrics about their performance. I think it makes sense for us to add a...
Option B might be useful for engines other than vLLM. It is probably worthwhile checking to see if those engines already support loading directly from bucket URLs at this point.