SPEC Tools 5 = CI BEST PRACTICES

[Carreau]

See among other https://github.com/scientific-python/summit-2023/issues/3

• [ ] job that tests nightly wheels/dev versions. Maybe a separate job that tests pre-releases.
• [ ] job that tests the oldest supported versions of declared dependencies
• [ ] cron jobs: notification are not working properly on github, we should implement a small tool, workaround to get notifications to maintainers
• [ ] filterwarnings to error out on warnings: expected warnings should be handled in CI, or ignored. Erroring out by default provides enormous help to notice new deprecations while upstream dependencies are still in their dev cycle (assuming there is a job that uses the dev versions/nightly builds)

[keewis]

we should implement a small tool, workaround to get notifications to maintainers

I'm maintaining a github action that sort of does this: https://github.com/xarray-contrib/issue-from-pytest-log

My experience with it so far is pretty good, despite the fact that issues that take a long time to fix can accumulate a lot of unrelated failures (making the individual failures a bit hard to track).