rust-lang
Add the ability to replace any source in .cargo/config.toml file
Problem
When deploying a crate with a registry dependency, the client tries to resolve the dependency from the specified URL. In JFrog we want to support cargo sparse index for a virtual repository as well - which means that we are resolving files without really knowing their source. The problem is that the client will fetch the dependencies from the real source, instead of the registry in the config.toml file. In order to resolve this issue we need the option to tell the cargo client to replace any source (maybe [source.any]), and not only specific sources like [source.crates-io]
Proposed Solution
Provide another known source that will be relevant for any source. or an option to direct all cargo requests to a specific registry (including URLs of dependencies)
Notes
No response
Just try to scratch the surface of syntax. Probably [source."*"] is a choice? Cargo already has the config key [profile.dev.package."*"] in profile overrides.
[source."*"] may not be the best choice, since it's already a valid source name today. Even .cargo/config.toml of every package in the world were migrated to the new semantic of [source."*"], old Cargo still doesn't understand the new meaning and understands it in the old way, which may lead to a complete different source replacement. It looks like a compatibility hazard.
For instance, this config setting also valid and works perfectly fine with cargo vendor.
[source.'🦀']
directory = "vendor"
[source.crates-io]
replace-with = "🦀"
I don't have an alternative at this moment 😞.
What is a virtual repository in this context?
If you don't want your registry to allow crates that have arbitrary sources, why can't you check that at publish time?
I believe virtual repository is an Artifactory concept, which provides one repository which combines multiple repositories: e.g. one repository which delegates to either a cache of crates.io, or a private internal repository (or multiple repositories).
Registries that mix crates from multiple sources are going to hit this problem. The current workaround is to use source replacement (specifically at published time) so that all crates claim that their dependencies come from crates.io. This recommendation is despite the documentation on source replacement. (Sorry for that.)
The fact that cargo keeps track of what source crates come from and does not silently interpose the same name from different sources is an important protection against "dependency confusion" attacks. The fact that it's going haywire here is because a virtual repository is "dependency confusion" as a service.
Some understanding I think I've gleaned based on this zulip chat.
I believe the issue is that you need N repository replacements for a virtual repo which provides access to N sub repos, and in the fully general case, there's no reason for the every client to have to know exactly which real repos are part of the virtual repo (and the set of repos should be able to be changed without updating all clients).
The overriding of sources seems limited? How would the virtual repository know which source should actually be used when there are multiple registries?
Maybe a implementation needs to be done as proxy. It could potentially be as easy as configuring CARGO_HTTP_PROXY to target the artifactory virtual repository or cargo can implement define it's own proxy e.g. (GOPROXY protocol, jfrog documentation)
Ensuring links bidirectionally for the community:
https://jfrog.com/help/r/jfrog-artifactory-documentation/community-assistance-to-support-cargo-virtual-repositories
What is the problem?
Cargo packages have a configuration file embedded into the package which lists the package's dependencies. Each dependency contains a URL field that is supposed to point to the same location of the package but more than often the URLs point to a source in a different location than the package itself.
Specifically in the Jfrog Artifactory context the ability to modify the source URLs in a scalable manner is crucial for us to provide the Virtual Repository capability since it each dependency can point once to a Local repository and another to a Remote repository.
With this being said, Cargo have provided developers with a solution to override dependencies as documented in Overriding Dependencies - The Cargo Book . Still, unfortunately, it is mostly for development use cases where a single user would set up their specific environment to point to local resources, and it does not provide a scalable method to override dependencies for all users consuming Cargo packages from Artifactory.
For further details please see the full discussions here: Add the ability to replace any source in .cargo/config.toml file
See the most updated discussion on Zulip: https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Possibility.20to.20replace.20any.20source.20in.20config.2Etoml/near/509207948