Mitigating recently published "dependency confusion" attack vector (aka CVE-2021-24105)

[dionrhys]

Is your feature request related to a problem? Please describe.

The Background

On 9th February 2021, Alex Birsan published a Medium article disclosing successful attacks on several companies made possible by exploiting vulnerable package resolution configuration in various packaging systems: https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

The attack targets "hybrid" package feeds, where packages can be downloaded from private and public feeds at the same time. The gist of it is that your organisation could depend on a package named "MyCompany.InternalLibrary" that's only expected to be served from your own private feed, but a malicious entity uploads a package of the same ID to the public feed, thus presenting the possibility of having this malicious package be downloaded instead of the expected one.

Microsoft has published a white paper "3 Ways to Mitigate Risk When Using Private Package Feeds" to counter this attack here: https://aka.ms/pkg-sec-wp

They've also published CVE-2021-24105 about it here: https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-24105

The Mitigation

One of the methods they detail to prevent this attack is to "Reference one private feed, not multiple". For NuGet—as it is a package manager that does not support prioritizing feeds—they suggest using a single private feed, and "This may require pushing public packages to your private feed manually or configuring the private feed to pull them automatically".

However in order for the single private feed with upstream configuration to be secure, there must be limitations on how the feed determines whether it's acceptable to download a package from the upstream or not. i.e. If "MyCompany.InternalLibrary" is not expected to come from the public upstream, the feed should not source it from the public upstream.

As of this month, Azure Artifacts has implemented a user-togglable flag per package ID that specifies whether a package may be pulled from an external source or not (see https://aka.ms/upstreamBehaviorBlog and https://aka.ms/configBehavior). This allows internal libraries to be marked "don't allow externally-sourced versions", which ensures these packages may only be "internally-sourced", aka local.

The Problem

BaGet doesn't currently have this kind of protection against conflicting package IDs on an upstream mirror, so at the moment it would happily download "MyCompany.InternalLibrary 1.2.0" from nuget.org (for example) even if "MyCompany.InternalLibrary 1.1.0" is a locally-uploaded package. If any package is missing locally, it will try to fetch it from the upstream mirror.

nuget.org allows ID Prefix Reservation, which can somewhat mitigate this problem (and it's in Microsoft's white paper under "Protect your packages using controlled scopes") provided that you've prefixed all internal packages consistently. However, it would be nice-to-have as a security-in-depth measure for BaGet to be able to provide its own layer of protection so that it won't (or can be configured not to) consult the upstream feed for "internal" packages.

Describe the solution you'd like

Right now I have two ideas of how this could be implemented in BaGet:

1. Replicate the Azure Artifacts behaviour of having a per package ID flag that determines whether a package can be externally-sourced or not

• This would likely involve adding a new entity that stores the package IDs, and any attributes that apply for that package ID (as opposed to a package in general - package ID and version)
  • This is equivalent to NuGet.Gallery's PackageRegistration entity
• Unfortunately I can imagine adding a per-package flag to be infeasible at the moment because BaGet.UI doesn't expose any read/write functionality right now (I think?), so adding togglable settings for packages onto the UI seems like a big departure from its current behaviour

2. Add in a new configuration section to reserve ID prefixes that must not be mirrored from any upstream

• This is a lot less drastic of an option than 1 (doesn't involve any admin UI)
• This would essentially be a "blocklist" of prefixes that will not be retrieved from the upstream
  • Would there be any use case for an "allowlist"?
• Prefixes could be specified just as literal strings in the config (i.e. "MyCompany" to reserve packageId.StartsWith("MyCompany", StringComparison.OrdinalIgnoreCase))
• One major pitfall of this approach: what if the org has its own prefixed packages that it has deliberately put on the upstream feed, e.g. it publishes internal packages on BaGet and public packages on nuget.org, with the same ID prefixes
  • Would the configuration need to be made more granular to allow/disallow certain package IDs, overriding the prefix? (this doesn't seem ideal if the org has many packages published)

When I started writing this issue I had only option 1 in mind, but on thinking about the simplicity of BaGet I think option 2 could fit in better. I'm struggling to reconcile the problem of the same ID prefix being used on internal and public packages deliberately though. Perhaps there's a better option that I haven't thought about.

As an aside, replicating the NuGet.Gallery behaviour of showing the "visual indicator" next to a reserved package (this is the verified property in the search API) could be a nice addition in tandem with these approaches, though maybe it doesn't reflect the same purpose.

Describe alternatives you've considered

While nuget.org provides the service of reserving ID prefixes, this is subject to their manual approval and discretion and they may have legitimate reasons to reject a request, and thus prevent an organisation from being able to rely on that particular method to mitigate the security problem.

The "dependency confusion" problem isn't something that can be easily solved on the client-side with NuGet, because its ordering of HTTP package sources is non-deterministic:

For projects using the PackageReference format, NuGet uses local sources first, then sources on network shares, then HTTP sources, regardless of the order in the configuration files. NuGet always ignores the order of sources with restore operations. — https://docs.microsoft.com/en-us/nuget/consume-packages/configuring-nuget-behavior#nugetdefaultsconfig-settings

The last mitigation step detailed by Microsoft is "Utilize client-side verification features". In NuGet land, this means making use of the packages.lock.json feature and performing restores in "locked mode". Unfortunately this does not prevent the scenario of a developer using the NuGet Package Manager UI to pick a new package version, and then updating to a malicious package (that was fetched from the upstream mirror).

I've found the packages.lock.json locked-mode restore to also not work at all in a solution with a mix of packages.config and <PackageReference> projects, which is unfortunately the case for solutions containing old ASP.NET Web Applications and WCF Services that don't support <PackageReference>. This kind of setup is probably not expected these days though.

---

Thoughts? 🙂

[loic-sharma]

Hi,

Thank you for bringing this problem up! This is a tricky problem that requires multiple solutions.

Current solutions

Azure Artifacts' solution to this problem is making restore across multiple package sources deterministic:

1. They recommend making your Azure Artifacts feed your only package source solves this problem by allowing multiple upstreams, and searching upstreams in order. They also recommend having your Azure Artifacts feed as your only package source. See: https://docs.microsoft.com/en-us/azure/devops/artifacts/concepts/upstream-sources?view=azure-devops#use-a-single-feed-on-the-client
2. They support multiple upstreams, which they query in a deterministic order: https://docs.microsoft.com/en-us/azure/devops/artifacts/concepts/upstream-sources?view=azure-devops#search-order

As for client-side verification, I recommend considering trusted signers if you are in an enterprise environment that can author sign packages (see this Twitter thread). The NuGet team is considering additional features in this space: lock file improvements, trusted signers improvements, and package source pinning.

What could we add to BaGet?

In my mind BaGet should add a few features to mitigate this problem:

1. Support multiple package upstreams, with a deterministic search order. Recommend customers to use BaGet as their only package source. This should be the short term solution.
2. For your "externally sourced per package ID flag" feature idea, I think we could make this work using BaGet's appsettings.json configuration file. This would improve the usability of feature 1.
3. Add trusted signer policies to BaGet. Reject package uploads that aren't trusted, and, ignore packages from upstreams that aren't trusted. This seems like a reasonable mid-term solution.
4. Add support for .NET Foundation's Signing Service to enable BaGet to repository sign package uploads. This seems like a long-term solution that needs more thought.

I believe option 1 would be enough for customers to be secure. Does that seem reasonable? Please let me know your feedback!

[dionrhys]

Thanks for such a quick response!

I think your feature 1 suggestion will be a good first step. It still means the package manager would list and source packages from an upstream where their versions don't conflict with the packages already sourced locally though. So it'll be ideal in CI (like it is with the current behaviour using one upstream) but doesn't guard against a developer updating to a package version that doesn't exist locally.

But yes, having multiple upstream support in BaGet will be ideal to help users that do currently rely on multiple feeds, so that they can configure those feeds in a deterministic order in BaGet instead. It's a definite feature to add!

For your feature 2, do you think we could maybe wildcard this so users don't have to enter all internal package IDs in the config? So either based on a prefix, or some other kind of wildcarding like regular expressions (i.e. "MyCompany.*"). I personally like regular expressions but I know it can be marmite. I think that's very similar to my option 2 then but are you suggesting this would be a per-upstream configuration as the relation to your feature 1?

Regarding trusted signers I don't have much of an opinion yet and I've not read up in depth about it. On the face of it, I'd likely forego configuring that setup personally if it involves paying for a code-signing cert, managing an internal PKI, or dealing with cert expirations. I could be way off the mark with what's required to get this working though, and I'm going to spend some time researching this approach.

It's very clear that ensuring a package would only be trusted if signed with a private key is more robust than just relying on not sourcing said packages from an upstream. It could guard against compromise of the internal feed for example - a malicious entity's package that made its way to the internal feed would not have the correct signature to be trusted.

---

My interest in this does come up from our org setup right now - our NuGet clients source from one internal (old NuGet.Server) feed and from the public nuget.org feed. We have reserved prefixes on nuget.org and have begun renaming all our misnamed internal packages to match these prefixes. We'll be switching to a single feed setup and I'm trying to determine whether we go in on BaGet or Azure Artifacts. The ability to prevent internal packages from being sourced from an upstream I think is something we would need because the risk of human error in choosing a version of a package that shouldn't exist publically (but does) is something that should be prevented.

So my own opinion here is that it would be great to have your features 1 and 2 in the short term.

[dschuermans]

@loic-sharma Any progress on defining multiple upstreams? If I understand the current docs correctly, it's only possible to define a single upstream?