build: move to PEP 621 project

[Molkree]

fixes https://github.com/intel/cve-bin-tool/issues/1595

TODO list for this PR

• ~[ ] Move doc dependencies to dependency-group:~ let's make it simple and keep out of the scope of this PR. Simply moving dependencies to the main project.toml might be a bad idea because they conflict with Python version specified in it. Namely our doc dependency Sphinx==8.2.3 requires Python>=3.11 but cve-bin-tool itself only requires >=3.9. This results in unresolvable dependencies. In practice docs are completely separate project, we can add pyproject.toml for docs only later on.

  [dependency-groups]
  doc = ["doc-dep1", "doc-dep2"]
  

• [x] update test/test_requirements.py, it expects requirements.txt to be present
• ~[ ] update how we generate requirements.csv~ as far as I understand, these are handcrafted, please correct me if I'm wrong

[terriko]

Thanks for working on this! I don't have any brain power to give it this week (this is probably the only day i'm going to do any PR review) but hopefully one of the other maintainers can step in. I've at least set the checks to run for now.

[Molkree]

I have replaced toml dependency with tomli toml is abandonded, it hasn't seen any releases since 2020 tomli is a community recognized replacement. Note that this is only required for Python<3.11 (3.9 and 3.10 at the time of writing, and 3.9 will be hopefully dropped next month as well). Change is required to update one test to read project dependencies from pyproject.toml. toml is so old it can't read our (valid) pyproject.toml.

We also used toml to read cve-bin-tool config, so updating our dependency to something working will benefit there as well.

I understand that this will probably slow down approval of this PR but it's a net positive (for ~1 year until 3.10 goes out of support). tomli is already used by numerous dev tools in our dependency tree but I don't think it will help with acceptance much because it was always optional and not included in SBOM.

[Molkree]

Hm, I'm really not sure why Linux Tests (3.9) failed while building wheel 🤔 I can build it just fine locally with Python 3.9 and it succeeded in my repository.

@terriko, can you rerun the tests? I think I'm finished with this PR

[Molkree]

Rerun unfortunately didn't help. What I see is that library build succeeds on Python 3.13 ("Long tests on Python 3.13"), on Python 3.12 ("Windows long tests") but fails on Python 3.9 ("Linux tests (3.9)"). Other versions (3.10 and 3.11) are unknown because they are cancelled after 3.9 fails.

I built locally using 3.9 because I picked it as the lowest currently supported version, worked fine. Also worked fine in my fork (log).

The only thing that comes to mind now is caching, I see it's enabled for setup-python action with cache: 'pip'. I see the cache is restored in the runs on this PR, but there's no cache in my repo/locally. I'll need to check how and what is cached.

[Molkree]

Latest hypothesis: This is just an intermittent error with GAD data source. I noticed that all tests run on push to branch and on pull request and on the same commit in PR in my fork test for Python 3.11 failed on push but succeeded for PR (same code in both) which strongly suggests it's an issue with GAD. Rerun fixed on push check as well. Copying relevant part here:

      creating build/lib/cache/gad/gemnasium-db-master-pypi/pypi
      error: can't copy 'cache/gad/gemnasium-db-master-pypi/pypi/proxy.py': doesn't exist or not a regular file

This is related to this part of code base I guess: GAD data source. The error is quite obscure to be honest, maybe there's a better way to catch it during build.

@terriko, yet another rerun (only failed jobs) should fix it I hope.

Initial hypothesis: Cache key is taken from requirements.txt or pyproject.toml (docs). It's just a file hash. But I have changed dependencies as mentioned above: toml was replaced with tomli. Let alone the fact that requirements.txt was deleted altogether and pyproject.toml should be hashed now. I don't understand how the hash of deps can stay the same 🤔

@terriko, can you delete the 3.9 cache and then rerun the test on this PR, please? It can be done from this page. That should be done in quick succession so that some other PR/commit doesn't recreate the 3.9 cache with a non-PEP 517 build. Current cache has key setup-python-Linux-x64-24.04-Ubuntu-python-3.9.24-pip-c88d8bf62fa962a4911c7d077a32b08bf74fdacf9e5f68b9de5fbad88bbede2b (we might need to do the same for 3.10 and 3.11, or maybe not because 3.12 and 3.13 worked fine).

P.S. I also wonder if deleting cache and letting it be recreated from this PR will break it for all other PRs/commits... In this case, we can once again delete the cache or just merge this ASAP