kubectl-ai icon indicating copy to clipboard operation
kubectl-ai copied to clipboard

Add github action to run lightweight evals as presubmit job

Open droot opened this issue 8 months ago • 5 comments

We want to run a few eval tasks (may be difficulty level easy) as pre-submit tasks to ensure that new PRs are getting some basic e2e tests.

Details:

  • The pre-submit job should bring up a kind cluster
  • The pre-submit job should also bring up ollama or llamacpp server
  • Run evals with the difficulty level easy or to begin with --task-pattern=scale
  • Add contributing section in the README.md to use this job as e2e test

droot avatar Apr 28 '25 16:04 droot

/cc @tuannvm let me know if you will be interested in contributing towards this.

droot avatar Apr 28 '25 16:04 droot

/cc @sakajunquality

droot avatar Apr 28 '25 18:04 droot

Hey! I can have a look at that @droot

rxinui avatar Apr 28 '25 22:04 rxinui

Hey! I can have a look at that @droot

Sure. Go ahead!

droot avatar Apr 29 '25 01:04 droot

@droot ater analysing the codebase, it seems like the CI is somehow dependant to ./ci/** folder which seems to embed an internal logic I'm not aware of. As a result any attempt to manually update .github/workflows/ci-presubmit.yaml results in an error on step verify-autogen.

Therefore, I create a separate worklow on pull_request.

An analysis is run and display as a GITHUB_SUMMARY_STEP for clarity.

Note: at the moment, tasks are evaluated on the default ollama model.

rxinui avatar Apr 29 '25 22:04 rxinui