hikari
hikari copied to clipboard
hikari-py
Move dev dependencies to smaller files
Checklist
- [x] I have run
noxand all the pipelines have passed. - [x] I have made unittests according to the code I have added/modified/deleted.
Related issues
Closes https://github.com/hikari-py/hikari/issues/883
Did you consider adding the usage of something like pip-compile (without hashing cause that doesn't tend to work between OS'es) to avoid CI being failed by a new sub-dep release?
Did you consider adding the usage of something like pip-compile (without hashing cause that doesn't tend to work between OS'es) to avoid CI being failed by a new sub-dep release?
I did, initially made use of that. But I found out later that there is no way to make a consistent environment for it, meaning that if someone adds a new requirement and runs the pip-compile pipeline, it will use their OS + python version to make decisions, which could lead to it installing unnecessary packages and/or not installing necessary ones for the CI to succeed.
I rather keep it in a predictable way that won't potentially break in the future, making it a bit of a pain for existing or new contributors by them using different systems and environments.
I rather keep it in a predictable way that won't potentially break in the future
It is currently neither of these things based on how often it has been randomly broken for previously described reasons.
I did, initially made use of that. But I found out later that there is no way to make a consistent environment for it, meaning that if someone adds a new requirement and runs the pip-compile pipeline, it will use their OS + python version to make decisions, which could lead to it installing unnecessary packages and/or not installing necessary ones for the CI to succeed.
How many dev deps are actually OS specific, does this actually have a noticeable effect or could you just build it on linux in the CI and then just not worry about windows or mac os possibly adding in one or two dev deps (also you can do os and py ver specific targets if you really want to be its a bit of a diminishing returns moment https://github.com/jazzband/pip-tools/pull/651/files)?
making it a bit of a pain for existing or new contributors by them using different systems and environments.
How many ppl are adding or removing dev-deps?
It is currently neither of these things based on how often it has been randomly broken for previously described reasons.
The only failures we have seen in CI with dependencies has been when updating the package itself, not any sub-packages.
Taking as an example the current flake8 conflicts, using pip-compile wouldnt solve any of them, as it has to do with packages not supporting them. pip-compile doesnt verify that the packages will work correctly together, it just makes sure that (on paper), they should be compatible, even tho in reality they might not be.
How many dev deps are actually OS specific, does this actually have a noticeable effect or could you just build it on linux in the CI and then just not worry about windows or mac os possibly adding in one or two dev deps (also you can do os and py ver specific targets if you really want to be its a bit of a diminishing returns moment https://github.com/jazzband/pip-tools/pull/651/files)?
I would rather have to pin a sub-dependency once in a while like I did in the past with jinja2. Additionally, this would just switch the problem from pinning a dependency to closing and blocking any potential updates to the compiled files (if we setup a pipeline to continuously update the packages to not miss new versions).
Furthemore, I would rather not have to keep different os an py versions specific files, as that would mean (for CI only) 9 different files (3 OS, 3 python versions), which will become 12 when we can support 3.11 (this doesnt account for minor versions, which projects could pin requirements to too).
The only failures we have seen in CI with dependencies has been when updating the package itself, not any sub-packages.
https://github.com/hikari-py/hikari/commit/842601aab66792b70cdf7e4300f6ae8729d4e33c
Again, would rather do that once in a while than maintain a whole list of requirement files. If you want to open a pr for that in a nice system that I can't seem to figure out rn, I would absolutely consider and review it with an open mind
Taking as an example the current flake8 conflicts, using pip-compile wouldnt solve any of them, as it has to do with packages not supporting them. pip-compile doesnt verify that the packages will work correctly together, it just makes sure that (on paper), they should be compatible, even tho in reality they might not be.
Ngl I'm not sure what your point is here, like it would fix that problem since the sub-deps would've been pinned so any changes in sub dep-vers would go through the pr process and breakages would be picked up by ci then rather than for whoever next tries to work on the project?
I would rather have to pin a sub-dependency once in a while like I did in the past with https://github.com/hikari-py/hikari/pull/1098. Additionally, this would just switch the problem from pinning a dependency to closing and blocking any potential updates to the compiled files (if we setup a pipeline to continuously update the packages to not miss new versions).
You can just not include the ver change for whatever package update is broken when the requirement files are re-compiled, takes a couple of seconds to do in github's file edit (I've done this multiple times myself lol).
(if we setup a pipeline to continuously update the packages to not miss new versions).
New sub-deps versions usually aren't a big deal in my experience (like unless it's stuff like docgen chances are you just won't notice the difference), I just have a cron action which opens a pr to bump them once a month and that works fine for me.
You can just not include the ver change for whatever package update is broken when the requirement files are re-compiled, takes a couple of seconds to do in github's file edit (I've done this multiple times myself lol).
Instead of having to keep an eye out for things like these, I rather just pin it once in a while when we need it, as it has been done in the past.
Ngl I'm not sure what your point is here, like it would fix that problem since the sub-deps would've been pinned so any changes in sub dep-vers would go through the pr process and breakages would be picked up by ci then rather than for whoever next tries to work on the project?
No, because the issue is that the sub-deps don't work with the new version. (flake8-broken-line pinning the flake8 version to 5.0.0 max or flake8-bandit trying to use removed functionality)
breakages would be picked up by ci then rather than for whoever next tries to work on the project?
I honestly don't think it's worth all the extra trouble of maintaining those new requirement files, specially because they - require to be '"built" with the OS and python version that they will compile for, there is no way to mock it.
And again, sounds like too much pain down the road to maintain them
Just want to say, I'm welcome to any other versions in the future that may utilise pip-compile. Just see it as too much of a pain rn