guac icon indicating copy to clipboard operation
guac copied to clipboard

[feature] Update deps.dev collector to take advantage of the new API

Open pxp928 opened this issue 11 months ago • 6 comments

Is your feature request related to a problem? Please describe.

With the new release of the API for deps.dev -> https://blog.deps.dev/api-v3/

  • Take advantage of the batch query support
  • Utilizing purl when querying for information (no need to decompose the purl as we are currently doing)

https://github.com/guacsec/guac/blob/e2832062e88193e87c484710bcd512c48ceee3a2/pkg/handler/collector/deps_dev/deps_dev.go#L401-L419)

Describe the solution you'd like upgrade the current deps.dev collector to use the latest API features

pxp928 avatar Mar 12 '24 16:03 pxp928

@rakshitgondwal would you be interested in working on this?

pxp928 avatar Apr 29 '24 21:04 pxp928

Sure @pxp928, I can take this up after #1710

rakshitgondwal avatar Apr 30 '24 07:04 rakshitgondwal

@rakshitgondwal have you started any work on this? If not, @nathannaveen would like to take it over.

pxp928 avatar May 24 '24 19:05 pxp928

Hi @pxp928, nope I didn't get the time to do so, and sure someone else can take this over.

rakshitgondwal avatar May 24 '24 19:05 rakshitgondwal

Hey @pxp928, could I work on this?

nathannaveen avatar May 29 '24 13:05 nathannaveen

Recently I have been working on this issue, and realize that since the deps.dev v3alpha version isn't stable it won't exactly fit our needs.


For example, when doing a purl batch lookup https://docs.deps.dev/api/v3alpha/#purllookupbatch, if any of the repos passed to the batch request aren't contained in the deps.dev database, it returns nil.

So the only way to check whether a purl is contained in the deps.dev database is to manually do a purlLookup for each individual purl. Which removes our need for the purlLookupBatch.

This issues will probably disappear when the deps.dev/api/v3alpha becomes stable. But for now, this won't work for what we want.


Additionally, we directly use the osv-scanner repo. https://github.com/guacsec/guac/blob/dad65eb57ce9ffe9db2d49ba3cda097a48191eda/pkg/certifier/osv/osv.go#L27

The issue with this, is that osv-scanner calls the deps.dev resolve API, which only works with deps.dev/api/v3 not v3alpha.

Note that I didn't realize we directly use the osv-scanner repo at the time of writing this issue https://github.com/guacsec/guac/issues/1947.


So I think the best way to go about this is to wait for the version to become stable and then implement it.

If you want to run some tests, this is the batch request that will return nil: https://github.com/guacsec/guac/compare/main...nathannaveen:guac:nathan/depsdevNewAPI#diff-82819d87163ba23dd5d158849f788068d5eadd6e1818517c879ec08b0745131cR455

nathannaveen avatar Jun 21 '24 15:06 nathannaveen