alibi-detect icon indicating copy to clipboard operation
alibi-detect copied to clipboard

Separate out release vs nightly/PR CI tests

Open ascillitoe opened this issue 2 years ago • 0 comments

A number of the example notebooks have very minor bits of deprecated code. Testing the examples during CI would help pick up these issues. However, some of the notebooks involve expensive computations, so are not suitable for regular CI. To be able to test the examples we probably need to do one (or more) of the following:

  • Try to reduce the computational complexity of examples as much as possible.
  • Skip overly expensive bits of examples. Don't know if this is possible via cell metadata?
  • Use a GPU via a github actions runner (Importantly not for PR's https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories)
  • Fetch as many pre-trained models as possible (this would also be a good test of the fetching methods).
  • Only test examples during release CI (not nightly and PR's).

More delineation between nightly and release CI will also be useful in other areas e.g. see https://github.com/SeldonIO/alibi-detect/issues/315#issuecomment-911567056.

ascillitoe avatar Sep 02 '21 11:09 ascillitoe