aiidateam
`CI`: remove matrix redundancy between `test-install` and `ci`
Fix for https://github.com/aiidateam/aiida-core/issues/6779
As of now, two set of tests, are redundant for no apparant reasons. For example, see for the first commit that I made in this PR: https://github.com/aiidateam/aiida-core/actions/runs/13631859178/job/38101273251?pr=6780#step:7:1 and https://github.com/aiidateam/aiida-core/actions/runs/13631859180/job/38101194890?pr=6780#step:7:1
The second one has only an extra flag --cov aiida, which is irrelevant in terms of redundancy.
I don't see a reason why these tests has to be run two times.
This PR, removes that to save time on CI.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 78.15%. Comparing base (
bb5f93d) to head (9d3f1d2).
Additional details and impacted files
@@ Coverage Diff @@
## main #6780 +/- ##
=======================================
Coverage 78.15% 78.15%
=======================================
Files 564 564
Lines 42643 42643
=======================================
Hits 33322 33322
Misses 9321 9321
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Thanks @danielhollas @GeigerJ2 Again, I can only repeat myself: What we really miss is a mechanism to check if dependencies are all installed correctly regardless of their channel; via pypi, lockfile, or etc. Since we don't have such mechanism we run the entire CI all over again. Which is redundant and waste of calculation resources, even though they are provided for free by GitHub!
I understand this PR, does not provide such a mechanism. But the solution is not to ignore this problem by moving on, but rather suggest to develop that mechanism.
@khsrali I don't think I understand.
What we really miss is a mechanism to check if dependencies are all installed correctly regardless of their channel; via pypi, lockfile, or etc.
Can you explain a bit more what you mean by this? I don't understand what you mean by "mechanism to check if dependencies are installed correctly"?
What we actually need to test whether the versions of dependencies that get installed (by pip or conda or whatever) are compatible with aiida-core. And the only way to ensure that is to run the tests, no?
@danielhollas
What we actually need to test whether the versions of dependencies that get installed (by pip or conda or whatever) are compatible with aiida-core. And the only way to ensure that is to run the tests, no?
I understand pip, conda, or uv resolve dependencies differently. The real question is does the final resolution of these resolver respect our pyproject.toml?
If Yes, then it has to work! if no then pyproject.toml has to be adjusted.
What I mean is: Whether dependency resolver does it's job, has to be tested by different means, rather than running the entire test suite redundantly.
The real question is does the final resolution of these resolver respect our pyproject.toml? Whether dependency resolver does it's job, has to be tested by different means, rather than running the entire test suite redundantly.
Sorry, I don't agree with these statements. :sweat_smile:
If the resolver finds a solution, it must satisfy the constraints from pyproject.toml by definition, otherwise that is a bug in the resolver --- that's not our problem and certainly not something we need to test in our CI --- those would be bugs in pip/uv/conda. What we do need to test is:
-
Whether the resolver finds a solution at all, given the constraints specified in pyproject.toml. Sometimes the resolver may fail to find a solution, or the solution truly does not exist (a trivial case for this would be if we added a PyPI package to pyproject.toml that is missing in conda-forge). Now that we use the uv lockfile, which solves for all supported python versions at once, this is less of a concern, because we at least know that the solution exists, and pip should find it as well. But we still test installation with pip just to be sure.
-
Most/all resolvers by default pick the latest available packages (within the constraints of pyproject.toml). And because we generally don't pin versions very strictly in pyproject.toml, it can happen quite easily that new package version breaks something in aiida (this is what happened with click 8.2 for example). The only way we catch this is by actually running the test suite --- I don't see any other way around this.
Point 1 can indeed be tested by just running the installation step (which is what we do in test-install.yml), but for Point 2 we must run the tests.
As for why we can't pin versions more strictly in pyproject.toml, this post explains the problem well (although I don't fully agree with all of its conclusions): https://iscinumpy.dev/post/bound-version-constraints/
Thank you @danielhollas these are exactly my points!
Most/all resolvers by default pick the latest available packages (within the constraints of pyproject.toml). And because we generally don't pin versions very strictly in pyproject.toml, it can happen quite easily that new package version breaks something in aiida (this is what happened with click 8.2 for example). The only way we catch this is by actually running the test suite --- I don't see any other way around this
Well, you are not actually testing a real life scenario. Where users have their own packages installed and the resolvers would yield different solutions. The two solution that uv, and pip provide based on pyproject.toml in an empty environment, are identical: The latest. I actually just run them, look:
| pip | uv |
|---|---|
| aio-pika-9.4.3 | aio-pika==9.4.3 |
| aiormq-6.8.1 | aiormq==6.8.1 |
| aiida-core-2.7.0 | aiida-core==2.7.0 |
| alembic-1.16.4 | alembic==1.16.4 |
| annotated-types-0.7.0 | annotated-types==0.7.0 |
| archive-path-0.4.2 | archive-path==0.4.2 |
| asttokens-3.0.0 | asttokens==3.0.0 |
| asyncssh-2.19.0 | asyncssh==2.19.0 |
| bcrypt-4.3.0 | bcrypt==4.3.0 |
| certifi-2025.8.3 | certifi==2025.8.3 |
| cffi-1.17.1 | cffi==1.17.1 |
| charset_normalizer-3.4.3 | charset-normalizer==3.4.3 |
| circus-0.19.0 | circus==0.19.0 |
| click-8.1.8 | click==8.1.8 |
| click-spinner-0.1.10 | click-spinner==0.1.10 |
| cryptography-45.0.6 | cryptography==45.0.6 |
| decorator-5.2.1 | decorator==5.2.1 |
| deprecation-2.1.0 | deprecation==2.1.0 |
| disk-objectstore-1.3.0 | disk-objectstore==1.3.0 |
| docstring-parser-0.17.0 | docstring-parser==0.17.0 |
| executing-2.2.0 | executing==2.2.0 |
| graphviz-0.21 | graphviz==0.21 |
| greenlet-3.2.4 | greenlet==3.2.4 |
| idna-3.10 | idna==3.10 |
| importlib-metadata-6.11.0 | importlib-metadata==6.11.0 |
| ipython-9.4.0 | ipython==9.4.0 |
| ipython-pygments-lexers-1.1.1 | ipython-pygments-lexers==1.1.1 |
| jedi-0.18.2 | jedi==0.18.2 |
| jinja2-3.1.6 | jinja2==3.1.6 |
| kiwipy-0.8.5 | kiwipy==0.8.5 |
| Mako-1.3.10 | mako==1.3.10 |
| MarkupSafe-3.0.2 | markupsafe==3.0.2 |
| matplotlib-inline-0.1.7 | matplotlib-inline==0.1.7 |
| multidict-6.6.4 | multidict==6.6.4 |
| nest_asyncio-1.6.0 | nest-asyncio==1.6.0 |
| numpy-2.3.2 | numpy==2.3.2 |
| packaging-25.0 (was already satisfied!) | packaging==23.2 |
| pamqp-3.3.0 | pamqp==3.3.0 |
| paramiko-3.5.1 | paramiko==3.5.1 |
| parso-0.8.4 | parso==0.8.4 |
| pexpect-4.9.0 | pexpect==4.9.0 |
| pgsu-0.3.0 | pgsu==0.3.0 |
| plumpy-0.25.0 | plumpy==0.25.0 |
| prompt_toolkit-3.0.51 | prompt-toolkit==3.0.51 |
| propcache-0.3.2 | propcache==0.3.2 |
| psutil-5.9.8 | psutil==5.9.8 |
| psycopg-3.2.9 | psycopg==3.2.9 |
| psycopg-binary-3.2.9 | psycopg-binary==3.2.9 |
| ptyprocess-0.7.0 | ptyprocess==0.7.0 |
| pure-eval-0.2.3 | pure-eval==0.2.3 |
| pycparser-2.22 | pycparser==2.22 |
| pydantic-2.11.7 | pydantic==2.11.7 |
| pydantic-core-2.33.2 | pydantic-core==2.33.2 |
| pygments-2.19.2 | pygments==2.19.2 |
| pynacl-1.5.0 | pynacl==1.5.0 |
| pytray-0.3.4 | pytray==0.3.4 |
| pytz-2021.3 | pytz==2021.3 |
| pyyaml-6.0.2 | pyyaml==6.0.2 |
| pyzmq-27.0.2 | pyzmq==27.0.2 |
| requests-2.32.5 | requests==2.32.5 |
| shortuuid-1.0.13 | shortuuid==1.0.13 |
| sqlalchemy-2.0.43 | sqlalchemy==2.0.43 |
| stack_data-0.6.3 | stack-data==0.6.3 |
| tabulate-0.9.0 | tabulate==0.9.0 |
| tornado-6.5.2 | tornado==6.5.2 |
| tqdm-4.67.1 | tqdm==4.67.1 |
| traitlets-5.14.3 | traitlets==5.14.3 |
| typing-extensions-4.14.1 | typing-extensions==4.14.1 |
| typing-inspection-0.4.1 | typing-inspection==0.4.1 |
| upf_to_json-0.9.5 | upf-to-json==0.9.5 |
| urllib3-2.5.0 | urllib3==2.5.0 |
| wcwidth-0.2.13 | wcwidth==0.2.13 |
| wrapt-1.17.3 | wrapt==1.17.3 |
| yarl-1.20.1 | yarl==1.20.1 |
| zipp-3.23.0 | zipp==3.23.0 |
The only difference is in packaging which is irrelevant to functionality of aiida.
Meaning the entire test suite is being run redundantly for no reason.
Okay, we seem to be on the same page on these things, but:
Well, you are not actually testing a real life scenario. Where users have their own packages installed and the resolvers would yield different solutions. The two solution that uv, and pip provide based on pyproject.toml in an empty environment, are identical: The latest. I actually just run them, look:
Yes, and? How does that contradict what I said?
But the solution is not to ignore this problem by moving on, but rather suggest to develop that mechanism.
Still don't understand what you envision us to do.
(I'll stop now to let others chime in)
Actually, one thing: I am very much sympathetic to not wasting CI resources. One way to do that would be to not run test-install.yml daily, but with lesser cadence. I think weekly would be enough?
Yes, and? How does that contradict what I said?
I really don't understand how else to explain it. We are running identical tests, twice.