Tommy Nguyen
Tommy Nguyen
https://github.com/modelcontextprotocol/registry/issues/135
@droot I pivoted my solution a bit. The idea is that I would need to rely on your existing Kind setup, as well as the LLM provider and API key...
@droot You’re right—people outside of Google Cloud Platform wouldn’t be able to run it. What do you think: would it be useful if, next time someone has an evals-related PR,...
I was also considering having it auto-trigger on every commit, but that seems excessive and would waste CI resources. So, I think it’s better to keep it as a manual...
> I am thinking if we can introduce a separate periodic workflow (tweak this PR) for running the entire eval every 4 hours or so ? yes I'm fine with...
@droot I can take on evals for openai https://github.com/GoogleCloudPlatform/kubectl-ai/issues/157
It would be awesome if we could also add these features: - Automatically run an evaluation on GitHub Actions whenever new models are added, released, or updated. - Display the...
Additionally, integrating Kubernetes Kind into CI pipeline and running the evaluations on top of it is another idea. This approach would eliminate the need to manually run eval on local...
@droot @janetkuo I’m happy to help with certain parts. Just let me know. Thanks!
@droot Thanks for creating the roadmap! Would it make sense to use GitHub Project to manage the phases and milestones? It could help us track progress, assign tasks, and keep...