dream2nix
dream2nix copied to clipboard
nix-community
cargo2nix integration
I am making this issue to find aspects of similarity in our work. Currently I'm planning cargo2nix architecture. My goal is to implement independent derivation building in a way suitable for nixpkgs long-term, maximizing cache granularity.
cargo2nix in dream2nix
- cargo2nix binary is a translator, generating a Cargo.nix with some feature & platform logic baked in
- mkcrate.nix and associated mkcrate-utils.sh are the builder
- each crate's derivation is independent
Will look at the structure of dream2nix overrides before updating cargo2nix's overrides. Same for fetching.
cargo2nix could be a builder in dream2nix that uses IFD, using cargo2nix to generate a Cargo.nix and then building that. Another way would be making a translator in dream2nix that encodes information needed to use mkcrate into the dream-lock and make a builder using mkcrate. This would also require IFD, but the IFD would be moved into the translation phase. (EDIT: though if the Cargo.lock file had the required information for building the crates as you said in the cargo2nix architecture issue, then it would be possible to have it build on-the-fly with no IFD.)
I would be glad to help with any of the questions etc. you might have in regards to the Rust subsystem, or if you want to implement stuff in dream2nix using cargo2nix code.
Hi @psionic-k
My goal is to implement independent derivation building in a way suitable for nixpkgs long-term, maximizing cache granularity.
Independent (or granular) derivation building is a goal of dream2nix as well and already implemented by the nodejs subsystem for example.
Use in nixpkgs
The long term goal of dream2nix is to bring the whole dream2nix framework into nixpkgs. This prevents an IFD import of the framework.
As this only makes sense once dream2nix reaches a stable release, I plan to add a standalone package export feature where only the dream-lock.json and builder code required for a specific package is dumped into nixpkgs.
Regarding parsing Cargo.toml/Cargo.lock files via nix.
This is already implemented in dream2nix (see the cargo-lock translator). You can basically forget about the parsing and only add the granular builder. I just updated the builder template. Take a look here
I also just polished the contributor guide a bit.
Will look at the structure of dream2nix overrides before updating cargo2nix's overrides. Same for fetching.
You don't have to use the dream2nix' own overrides format. dream2nix is also compatible to the well known nixpkgs override style. I think @yusdacra uses that in nix-cargo-integration as well, if I'm not mistaken.
You don't have to use the dream2nix' own overrides format. dream2nix is also compatible to the well known nixpkgs override style. I think @yusdacra uses that in nix-cargo-integration as well, if I'm not mistaken.
Yes, nix-cargo-integration uses them. It used to use only crate2nix in it's infancy, so I haven't changed the overrides stuff since then.
We had some private registry support in cargo2nix and I had been meaning to at least figure out what it needed to work and get it documented, but fetching is very non-specific to cargo2nix. Are Rust private registries supported here?
We had some private registry support in cargo2nix and I had been meaning to at least figure out what it needed to work and get it documented, but fetching is very non-specific to cargo2nix. Are Rust private registries supported here?
No, haven't worked on that yet. But I would like to hear how it's done in cargo2nix.
This is already implemented in dream2nix (see the cargo-lock translator). You can basically forget about the parsing and only add the granular builder.
I'm re-iterating what other replies also show awareness of. The above statement is not quite true yet. We lose too much feature and platform information from the top-level when building crates independently. The lock is just not sufficient info yet. Cargo will forget feature activation if only using the lock in a deep dependency.
Making cargo generate enough lockfile info to avoid expression generation is one of the end goals currently identified in cargo2nix. It will require eventual upstreaming into cargo in order to live. Cargo2nix will just be the POC / prototype at that point.
No, haven't worked on that yet. But I would like to hear how it's done in cargo2nix.
I don't think it was good. Someone pretty inexperienced worked on it, and it was just for a small internal use case. AFAIK crane looks the most robust for external registry support.
https://github.com/cargo2nix/cargo2nix/blob/release-0.11.0/overlay/lib/fetch.nix#L13-L39
You don't have to use the dream2nix' own overrides format. dream2nix is also compatible to the well known nixpkgs override style.
cargo2nix derivations are as well. I think I'm just improving some plumbing for the cargo2nix user interface, so no heavy lifting will be needed anywhere.
Overall it seems like mkcrate.nix and the overrides, applied via makePackageSetInternal, are the big pieces that will be useful to make roughly compatible with dream2nix. While I don't think that mkcrate.nix is actually good yet, it's building a lot of things in CI and has enough stability to go in a healthy direction. The main crimes I know of:
- it clobbers the .cargo/config.toml, building a new one with string concatenation, getting in the way of a lot of asked-for linking and other configuration
- remarshal + jq usage to facilitate querying and writing tomls, some querying to recover linking information necessary for downstream cargo invocations when building dependents
- using cargo may limit workflows that invoke rustc directly, although I'm unaware of these cases
- "bare metal" linking not really supported
One of the main architectural choices I'm figuring out now is if cargo2nix really needs an overlay or to expose the combined overlay. While applying overrides to nixpkgs or crates is necessary, I'm not sure that this requires giving a modified nixpkgs back to the user when a lib attribute could probably do all of the same. How nixpkgs gets configured for cross compile support is another aspect that might affect the right choice.
There's some wasm code I'm looking at now to get a better idea of what will be necessary in these cases.
Is there still any intention of integrating cargo2nix with dream2nix? Otherwise we should close this issue.
I don't have any intentions to work on this right now, so I think we can close it and if anyone else wants later we can reopen it.
I was asleep when this conversation happened. My priority is definitely cargo2nix before attempting to package pieces of it for inclusion in dream2nix, so I'm not saying anything will happen soon, but this isn't a dead issue at all. There is a zero chance that the granular derivation-for-each-crate style isn't the better solution for most use cases, so the project has an inevitable future unless one of the other solutions offers up a granular caching implementation first.
OK, thanks for the update. Then let's re-open the issue in the future once there will be collaboration.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/rust-cargo-cargo2nix-vs-crane-vs-dream2nix-vs/25001/1
