feat: add support for OSV schema

[juan131]

Summary

As described at https://github.com/anchore/grype/issues/1609, Bitnami is providing vulnerability matching data for their containers via the Bitnami Vulnerability Database repository.

This database uses the Open Source Vulnerability format (AKA OSV) and it's currently part of the aggregated OSV vulnerability database.

This PR follows up https://github.com/anchore/vunnel/pull/447, adding support to grype-db to aggregate data from CVE feeds using the OSV schema.

[juan131]

cc @westonsteimel @wagoodman

[juan131]

@wagoodman sorry for the long delay on resuming this work, I've been involved in other projects these last weeks and I didn't find the time for this.

[carrodher]

Hi @wagoodman, is there anything we can help with at this time?

[carrodher]

Hi @wagoodman,

I hope you’re doing well! We wanted to check in on the status of our recent contributions. We have some open PRs, and we’re unsure about the current progress or the next steps. Could you kindly provide an update? It would help us plan our work and understand how we can continue contributing effectively.

Here are a couple of the open PRs for reference:

• https://github.com/anchore/vunnel/pull/512
• https://github.com/anchore/grype/pull/1826

We appreciate any details you can share. Thanks in advance for your help!

[willmurphyscode]

Hi @carrodher!

I wanted to give an overview of what still needs to happen to get grype supporting the bitnami vulnerability provider. The list is at least mostly in the order things need to happen. The items don't say who's doing what or when; I'm trying to just get the scope of work remaining written down, and then we can figure that out. I am happy to help with the work but I want to clarify the remaining work first.

Step 1: Syft needs to find the right packages

Syft needs to find packages that can be compared to the data bitnami publishes. The best way to do this is probably to write a Syft cataloger that parses the SPDX SBOMs that bitnami puts at paths like /opt/bitnami/redis/.spdx-redis.spdx.

This work is tracked at https://github.com/anchore/syft/issues/3065.

This issue is largely ready to pick up.

It's important that we do this first for two reasons:

1. Syft finding bitnami packages in bitnami images is a useful feature all on its own
2. When adding the data source to grype, we need to already know what we're trying to match against

Step 2: Make a test image

In order to test the new data provider end to end, there needs to be an image that grype can scan to find known vulnerabilities from the new provider.

Add a test image built from an old Bitnami (so it will have some vulns) image to our test images repo.

Step 3: Local end to end workflow working

The way to do that is to make a 3 repo local workspace with Vunnel, Grype, and Grype-DB all incorporating any necessary changes to scan a bitnami image.

To attempt and end to end run, you can go into the vunnel directory and run:

make dev provider=bitnami  # creates a subshell
make update-db 
grype "$image_from_ste_2" 

This will fail, because there are a few other things to do, but when it starts finding the known vulnerabilities that are present in the old Bitnami image, things are working and we can set up an end to end test and merge. There are a few changes that I know about, and we may discover more.

PR changes that I know about

This PR (https://github.com/anchore/grype-db/pull/217) needs to have a transformer implemented. The transformer is a function that is responsible for mapping the OSV data that vunnel downloaded from Bitnami to the vulnerability and vulnerability metadata structures that grype uses in its SQLite database.

You can see a transformer here:

https://github.com/anchore/grype-db/blob/d8d1497bd5300438ab92b42d94cb064848b2a5f8/pkg/process/v5/transformers/github/transform.go#L44

Additionally, the vunnel pr will need to be changed so that it emits records in dictionaries shaped like OSV JSON, in other words this comment needs to be addressed.

It's also possible that Grype needs to have a new matcher added for the bitnami package type, but the stock matcher might work (it tries to compare semver for packages that don't have a more specific matcher type), but some package types need a specific match. You can see all the matchers here:

https://github.com/anchore/grype/blob/09fb54ee101fde621b38aa6223c7ca6e7881baa7/grype/matcher/matchers.go#L32-L46

After these changes, I think that we will be ready to add the end to end tests.

There are also some linting fixes and missing unit tests, but I think it makes sense to get the end to end flow working. If you prefer to add unit tests as you build the blocks of the system, that's fine too.

Adding labels and test cases

Vunnel, Grype, and Grype-DB are guarded by end-to-end test cases that basically have 3 inputs:

1. a reference run of grype (latest with the last published database, for example)
2. a candidate run of grype (latest with the database created by the current branch of vunnel, for example)
3. a set of labeled test images

The test gate than asserts that for each test image, the candidate configuration is better, or at least no worse within some tolerance, than the reference configuration.

One grype in your workflow can match the test image, you can label matches grype finds against your test images.

Some details are at https://github.com/anchore/vunnel/blob/main/tests/quality/README.md#getting-started. This is a step we can definitely help with.

That ended up being a bit longer post than I thought, but I wanted to get the remaining steps written down concretely to help get your hard work over the line.

[willmurphyscode]

@carrodher and @juan131 I am pretty determined to see these changes land and will start picking up this work. I'd also be happy to sync up with you all if you'd like to provide any additional assistance - just let me know!

As an aside, we are striving to make this process easier - I realize from outside the team the process outlined in my last comment is probably a little daunting and hard to discover. I'd also be glad to sync with you on what might have made that easier.

[juan131]

@willmurphyscode thanks so much for writing down the required steps and your determination to get this done. If that's fine to you, we could set up a sync meeting (@carrodher and myself from Bitnami side) so we can plan the action items and assign who'll be leading the development

[willmurphyscode]

@juan131 a sync up meeting sounds great. Probably the easiest way to hash out the details is to DM me in our discourse (not discord) instance: https://anchorecommunity.discourse.group/ (@willmurphy). I'm traveling next week but we could plan to meet sometime 7 October or after.

We also have regularly scheduled community meetings. The next one is at Thursday, 10 October at noon Eastern.

[willmurphyscode]

@juan131 here's the branch I mentioned during the meeting today: https://github.com/anchore/syft/compare/spike-bitnami-cataloger

It should help in creating a PR to resolve https://github.com/anchore/syft/issues/3065.

Let us know if you have any questions, or when you want to set up the next meeting. Thanks!