osv-scanner icon indicating copy to clipboard operation
osv-scanner copied to clipboard

feat: fetch Maven metadata from specified repositories

Open cuixq opened this issue 1 year ago • 1 comments

https://github.com/google/osv-scanner/issues/1045

There are repositories defined in a Maven pom.xml. When looking for an artifact, these repositories are searched one by one until the artifact is found. Maven Central is the default registry to try at the last.

To support this behaviour, this PR:

  • makes MavenRegistryAPIClient host a list of registries besides the default registry
  • adds UpdateRegistries to DependencyClient to update the registries
  • adds a new flag to specify the default maven registry for fix
  • add new experimental options to scan to align with what we have for fix

Currently, running resolution with native data source is a bit slow and this is due to Maven resolver calling MatchingVersion for all requirements. There is work in progress to improve this and I believe this would speed the resolution.

TODO:

  • still need to update documentation for new options/flags
  • update deps.dev Maven resolver for mutil-registry resolution

cuixq avatar Sep 30 '24 02:09 cuixq

Codecov Report

Attention: Patch coverage is 67.02703% with 61 lines in your changes missing coverage. Please review.

Project coverage is 68.43%. Comparing base (a5a1e29) to head (b0295ef). Report is 169 commits behind head on main.

Files with missing lines Patch % Lines
internal/manifest/maven.go 45.45% 9 Missing and 3 partials :warning:
internal/resolution/datasource/maven_registry.go 85.07% 7 Missing and 3 partials :warning:
internal/utility/maven/maven.go 47.05% 6 Missing and 3 partials :warning:
...nternal/resolution/client/maven_registry_client.go 53.33% 5 Missing and 2 partials :warning:
cmd/osv-scanner/fix/noninteractive.go 57.14% 4 Missing and 2 partials :warning:
internal/resolution/manifest/maven.go 45.45% 4 Missing and 2 partials :warning:
cmd/osv-scanner/scan/main.go 66.66% 2 Missing and 1 partial :warning:
pkg/osvscanner/osvscanner.go 84.21% 2 Missing and 1 partial :warning:
...al/resolution/clienttest/mock_resolution_client.go 33.33% 2 Missing :warning:
cmd/osv-scanner/fix/main.go 50.00% 1 Missing :warning:
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1286      +/-   ##
==========================================
+ Coverage   68.09%   68.43%   +0.34%     
==========================================
  Files         183      183              
  Lines       17498    17606     +108     
==========================================
+ Hits        11915    12049     +134     
+ Misses       4942     4895      -47     
- Partials      641      662      +21     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar Sep 30 '24 03:09 codecov-commenter

I mark this PR to draft - the registries should not be added when importing dependencies, but that is now done by MergeParents for all cases.

cuixq avatar Oct 11 '24 05:10 cuixq