Reimplement NuGet support

[mnonnenmacher]

The current implementation of the dependency resolution in NuGetSupport has several issues which lead to a massive overreporting of dependencies. These issues are partly demonstrated in NuGetFunTest and DotNetFunTest since 1. The problem is that most of the dependency resolution rules 2 like "floating versions", "nearest wins" and "cousin dependencies" are not implemented. Also target frameworks 3 are completely ignored.

This is an issue because many libraries declare different dependencies depending on the target framework (see for example 4), and NuGet uses a "nearest framework" 5 algorithm to determine which target framework of a dependency matches the target framework of the application. An attempt has been done to fix those issues in the "nuget-rewrite" branch 6. As a workaround for the target framework issue it uses a website which makes the GetNearest function available 7, but this approach is not stable enough to be submitted to ORT.

Instead, the current approach of reimplementing the NuGet dependency resolution should be replaced with a native implementation like it is done for most other package managers. See the following links for more information about how this could be implemented:

https://github.com/joelverhagen/NuGetTools https://github.com/daveaglick/Buildalyzer https://github.com/dotnet-outdated/dotnet-outdated https://www.jerriepelser.com/blog/analyze-dotnet-project-dependencies-part-1/ https://www.jerriepelser.com/blog/analyze-dotnet-project-dependencies-part-2/ https://github.com/blackducksoftware/nuget-dotnet5-inspector

[sschuberth]

As a general reference, FOSSA has published a nice write-up about Managing Dependencies in .NET; while I haven't read through all of it yet, maybe there's some valuable information for us hidden in there.

[sschuberth]

FYI, Nexus IQ relies on https://github.com/CycloneDX/cyclonedx-dotnet to capture transitive dependencies in NuGet projects.

[sschuberth]

Another FYI, there was a remark from @maxhbr that Dependency-Check's analyzer does a better job in discovering NuGet dependencies for at least one example. Probably @maxhbr can share more details here later.

[pombredanne]

I am starting to tackle this now... I would be interested to collect as many representative project files as possible... These can be attached here... I am interested in:

1. *.*proj files such as .csproj and their *.sln parent files
2. packages.config files
3. older and newer styles of these as used in the wild.
4. .nuspec files
5. project.assets.json
6. nuget.config that may point to private NuGet feeds (it is OK to sanitize the URL of course)

In general any project files that are less common are useful. For a look at a first pass on the design please see https://github.com/nexB/nuget-inspector/blob/239e1941b2aadba1769beb622c1707c8f87739e8/docs/source/nuget-dependencies-design.rst ... comments mucho welcomed!

[pombredanne]

@sschuberth re:

FYI, Nexus IQ relies on https://github.com/CycloneDX/cyclonedx-dotnet to capture transitive dependencies in NuGet projects.

Thanks! this is useful! Of note: cyclonedx-dotnet runs as a dotnet command installed in the current dotnet environment. It seems to assume that the analyzed project development SDK must be that of tool. The tool requires the SDK as dotnet command extension. It executes a full "restore" to collect dependencies. Because of this, it will be as accurate as dotnet/nuget themselves... but it may be not be able to run in all contexts and likely assumes too much wrt. the actual runtime to be usable in practice on a large scale: this is running a full build and not just a dependency resolution.

[pombredanne]

@sschuberth re:

Another FYI, there was a remark from @maxhbr that Dependency-Check's analyzer does a better job in discovering NuGet dependencies for at least one example. Probably @maxhbr can share more details here later.

What I can see DependencyCheck does two things:

• it can parse the "legacy" "packages.config" XML file in https://github.com/jeremylong/DependencyCheck/blob/main/core/src/main/java/org/owasp/dependencycheck/analyzer/NugetconfAnalyzer.java
• it can parse .nuspec XML files in https://github.com/jeremylong/DependencyCheck/blob/main/core/src/main/java/org/owasp/dependencycheck/analyzer/NuspecAnalyzer.java
• it can parse .csproj and .vbproj projects files in https://github.com/jeremylong/DependencyCheck/blob/main/core/src/main/java/org/owasp/dependencycheck/analyzer/MSBuildProjectAnalyzer.java

As far as I can see there is no dependency resolution taking place but only static manifest parsing which would work OK for legacy nuget project with a packages.config but would be incomplete for a new style project-file based "*.*proj" project.

[pombredanne]

@sschuberth re:

As a general reference, FOSSA has published a nice write-up about Managing Dependencies in .NET; while I haven't read through all of it yet, maybe there's some valuable information for us hidden in there.

Fossa cli does essentially static parsing of nuspec, proj files,and package.config, plus the F# Paket formats . In addition it can parse the project.assets.json internal format which is a by-product of running msbuild (and would get full deps when present ) and a legacy "project.json" from older NuGet versions.

This is a static approach that will not get non listed dependencies

[pombredanne]

On a chat @tsteenbe suggested:

Maybe re-use form Microsoft SBOM tool? https://github.com/microsoft/component-detection/tree/main/test/Microsoft.ComponentDetection.VerificationTests/resources/nuget

This tool does static parsing of project.assets.json, packages.config and .nuspec files. It can also extract .nupkg archives and honors the nuget.config file for possible local paths to NuGet repos. It does only static parsing from what I saw and no dependency resolution unless you have built the project first with .NET. (It is also used elsewhere in the MSFT SBOM tool FWIW) There may be a few tidbits of code that is interesting there in anycase.

Or re-use some of GitLab test projects https://gitlab.com/gitlab-org/security-products/tests/dotnet5 or https://gitlab.com/gitlab-org/security-products/tests/dotnet6

Both are non-open source, proprietary test repos using the gitlab-ee license https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/data/licenses/gitlab-ee.LICENSE ...these are test projects not actual tools.

The actual tool (also proprietary) parses statically a subset of files: "*.csproj" and the legacy "packages.lock.json" per https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/98dbfa822bc5ed4cbcfdfee4998d5a353340c01a/finder/package_manager.go#L103 and https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/98dbfa822bc5ed4cbcfdfee4998d5a353340c01a/finder/detect_test.go#L101 and they also have a proprietary utility to process version ranges at https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/tree/master/vrange/nuget

[sschuberth]

FYI, MS & Canonical just announced .NET 6 for Ubuntu 22.04, so now that we are using Ubuntu 22.04 we could install tooling as easy as sudo apt install dotnet6.

[pombredanne]

@sschuberth re:

we could install tooling as easy as sudo apt install dotnet6.

That's very nice... note that dotnet installs alright on debian and ubuntu already (not as a base Ubuntu-provided package but as provided by Microsoft)

[bruzzlerli]

Perhaps this project is worth a look in scope of this issue:

https://github.com/tomchavakis/nuget-license

There are nice flags like --use-project-assets-json (dotnet restore required first)

[pombredanne]

I pushed at last the code in https://github.com/nexB/nuget-inspector and a first release in https://github.com/nexB/nuget-inspector/releases/tag/v0.5.0 I will push builds beyond linux in the next release

Of note:

• It supports packages.lock.json https://github.com/oss-review-toolkit/ort/issues/3943
• It should support authenticated feeds as per https://github.com/oss-review-toolkit/ort/issues/3188 but this is not tested yet.
• It may support other V3 API Nuget feeds per https://github.com/oss-review-toolkit/ort/issues/3190 but this is not tested.

This is designed to eventually be integrated in ORT.

Ping @joergmetzler @MoumitaManna @MarcelBochtler @joergmetzler

[sschuberth]

Also target frameworks 3 are completely ignored.

FYI, NuGet.org announced search by target framework. Maybe that means there's also a corresponding API now that we could use.

[sschuberth]

This has been done in https://github.com/oss-review-toolkit/ort/pull/6209.