evals: we should have a presubmit job that runs on PRs that adds/modify a eval task.

[droot]

We expect to add many new evals, it will be nice to have a presubmit job that:

• runs only if PR adds/modifies eval tasks
• Better if it runs the affected task in the eval

for example, while reviewing this, I would have liked to see the output of the eval run.

[prasad89]

hey @droot can I pickup this?

[droot]

Thanks for volunteering @prasad89 . Only thing I am afraid of is -- you may have hard time verifying it without my help and that may require lots of back and forth.

You can definitely give it a try, and let see how it goes.

[prasad89]

sure, I'll give it a try, thanks!