python
Improving the daily stubtest check for third-party stubs
Currently, stubtest checks the stdlib and all third-party stubs daily and will create a new issue if there are discrepancies. While this is fine for the stdlib, it gets annoying for third-party stubs, and doesn't scale well. We either need to fix third-party stubs to a fixed version, which is often overly specific and causes a new stubsabot issue whenever a new micro version is released, or we need to use a range, which has the danger of triggering a new issue daily until fixed. It also means that third-party stubs get checked every night, no matter whether a new package was released or not.
Instead I suggest we extend stubsabot to run on every newly released package version. If the new release is outside the supported version range of the stubs, stubsabot will function as it does now. If the new release is within the supported version range, stubsabot will run stubtest against the new package and open an issue specific for that package if stubtest fails.
This would make fixing these issues less pressing, but also enable us to use more relaxed version ranges. Changes in the "most annoying" packages often only concern internal API and/or are backwards compatible. In the future, we might even consider opening up the ranges more, for example by using >= X.Y.Z, < (X-1) for packages following semver.
I think it's nice to have CI run on everything occasionally. A full run is hard to mess up and great for sanity.
The proposed change seems good, but maybe it's worth keeping the full run at a reduced frequency? Say every two weeks instead of daily?
I'll also note that "stubtest: third-party" also runs for PRs that contain changes to requirements-tests.txt (or daily.yml)
The proposed change seems good, but maybe it's worth keeping the full run at a reduced frequency? Say every two weeks instead of daily?
The problem with that is that with this approach we are basically guaranteed to always have some third-party problems, since we're not forced to fix them. (While this sounds bad, it is necessary long-term to prevent maintainer fatigue.) This would basically always open another spurious issue that's just a duplicate of other already opened issues.