Add github action to run lightweight evals as presubmit job
We want to run a few eval tasks (may be difficulty level easy) as pre-submit tasks to ensure that new PRs are getting some basic e2e tests.
Details:
- The pre-submit job should bring up a kind cluster
- The pre-submit job should also bring up ollama or llamacpp server
- Run evals with the difficulty level
easyor to begin with--task-pattern=scale - Add contributing section in the README.md to use this job as e2e test
/cc @tuannvm let me know if you will be interested in contributing towards this.
/cc @sakajunquality
Hey! I can have a look at that @droot
@droot ater analysing the codebase, it seems like the CI is somehow dependant to ./ci/** folder which seems to embed an internal logic I'm not aware of. As a result any attempt to manually update .github/workflows/ci-presubmit.yaml results in an error on step verify-autogen.
Therefore, I create a separate worklow on pull_request.
An analysis is run and display as a GITHUB_SUMMARY_STEP for clarity.
Note: at the moment, tasks are evaluated on the default ollama model.