Add an 'audit' goal which checks lockfiles against reported vulnerabilities.

[TansyArron]

This fetches vulnerability data from the pypi json api and prints a vulnerability report for each lockfile in the repo.

Current results from the Pants repo:

I've added a way to exclude specific vulnerabilities from the report, as sometimes they're not relevant. eg: GHSA-w596-4wvx-j9j6 only affects subversion projects.

[cburroughs]

Neat! Some prior art/discussion in #16495 & #16288

Could you give some design context for going straight to the pypi json api instead of trying to wrap an existing tool like pip-audit?

[TansyArron]

Neat! Some prior art/discussion in #16495 & #16288

Could you give some design context for going straight to the pypi json api instead of trying to wrap an existing tool like pip-audit?

Thanks for the links! Interesting to see someone elses build of this tool.

The pip-audit tool is itself a wrapper around the pypi/osv vulnerability api's, but it also does a bunch of extra work we don't need or want - eg: it performs a full resolve and potentially downloads more packages. It also goes poking through your local pip cache which doesn't play well with being sandboxed.

Because I wrote this tool to operate only over lockfiles we know we can skip all of that extra work. It works out smaller, lighter weight, and doesn't require a bunch of new 3rdparty deps.

I've tried to write it in such a way that we can add an osv backend to it if people prefer that over pypi.

[TansyArron]

Really nice stuff!

I think we may want to split the implementation into the core audit goal feature, and then have the Python part with pip_audit in the python backend. cf. how the publish goal is partitioned.

(some inconsistencies in naming as well, pip audit vs. pypi audit..)

Just to make sure I've got this straight:

• The generic audit goal moves to src/python/pants/core/goals including the generic Result/Request/Subsystem definitions.
• pip_audit_rule.py moves to src/python/pants/backend/python/goals

Do I leave anything in experimental? Where should the helper code (format_results.py, pip_audit.py) live?

The naming is a bit of a mess. If anyone's got preferences I'm all ears. pip-audit is a thing people are already familiar with as a tool. pypi-audit is slightly more accurate in that we're actually hitting the pypi api. python_3rdparty_dependency_vulnerability_audit is most accurate but crazy long. 🤷

[kaos]

Really nice stuff! I think we may want to split the implementation into the core audit goal feature, and then have the Python part with pip_audit in the python backend. cf. how the publish goal is partitioned. (some inconsistencies in naming as well, pip audit vs. pypi audit..)

Just to make sure I've got this straight:

• The generic audit goal moves to src/python/pants/core/goals including the generic Result/Request/Subsystem definitions.

Ah, right, slightly misleading phrasing on my part. This doesn't have to be a core builtin goal, but stay in a backend (where it is), being the "core" (or perhaps rather, the "common" parts of the audit feature) in the pants.backend.audit backend. (The register.py file can stay in pants.backend.experimental.audit for the experimental status.)

• pip_audit_rule.py moves to src/python/pants/backend/python/goals

Yes, and perhaps register these python specific audit rules from pants.backend.experimental.python?

Do I leave anything in experimental? Where should the helper code (format_results.py, pip_audit.py) live?

Keep audit related register.py files under experimental backends, and I think these util files for the python audit could live in src/python/pants/backend/python/audit/ (similar to other features that group related code for a feature is in that backend.)

The naming is a bit of a mess. If anyone's got preferences I'm all ears. pip-audit is a thing people are already familiar with as a tool. pypi-audit is slightly more accurate in that we're actually hitting the pypi api. python_3rdparty_dependency_vulnerability_audit is most accurate but crazy long. 🤷

I think pip-audit is misleading, as we're not using pip. The long name, although accurate, could be a bit too generic, in case alternatives turn up, they could all potentially fit that description, so I think my preference leans towards pypi-audit being the most precise.

[cburroughs]

(Narrowly on the naming, if we are not using `pip-audit the program we should not call it "pip-audit". pypi-audit is reasonable.)

[huonw]

Thanks for the contribution.

When you have a chance (and this is close to ready), please merge main (or rebase onto it) and add some release notes to docs/notes/2.22.x.md (maybe in the Python section). See https://github.com/pantsbuild/pants/discussions/20888 for more info.