Add crate2nix to nixpkgs + documentation

[Mic92]

This project looks awesome. What do you think about adding it to nixpkgs itself? We could also replace carnix with it and add documentation to doc/languages-frameworks/rust.section.md

[kolloch]

I would love to add Crate2nix to nixpkgs and the docs. I was just a bit shy ;)

I haven't followed carnix lately so I wouldn't like to make the judgement call to remove it from the docs.

[Mic92]

@andir would be probably happy to review it.

[andir]

Given all the current contenders in the field I'd probably be against calling one way the true way from within nixpkgs. Adding crate2nix is certainly one thing and adding parity (in terms of mentioning as alternative in the docs) would be a good first step.

In the long term I wish to get the time/energy to write about all the different approaches and what the ups and downs of them are. It probably depends on your situation which kind of tooling you prefer.

But yes I would be very happy to carry crate2nix within nixpkgs :-) Ping me for reviews.

[Mic92]

I added naersk and crate2nix here: https://nixos.wiki/wiki/Language-specific_package_helpers

Which would you propose as the current recommended one?

[kolloch]

Generallly, there are multiple contenders. The current documentation in nixpkgs is outdated. That can obviously be fixed but a "recommended" solution should be properly documented. @pmeunier

buildRustPackage while seemingly simple, is problematic because it has a complicated fixed output derivation that has the rust tooling as buildInput. That means that updating the rust tooling does NOT result in a rebuilt when it should as far as I understood it. @alyssais

I think it ultimately comes down to goals and the quality of implementation:

• crate2nixes goals are building all dependencies in separate derivations and being user friendly. I also have a goal of supporting all widely used crates. My approach would be to either extend buildRustCrate or crate2nix accordingly or, if crates do something crazy, convince them to change their code so that it works here (I did that before). A workflow without manual build file regeneration is mostly working with "import from derivation".
• carnix is what I modeled crate2nix after. I was just hitting some road blocks and wasn't able to fix them easily. @pmeunier can elaborate on the current state and strengths.
• naersk utilizes cargo as far as possible. For now, therefore it builds a crate in two derivations, one for the dependencies, one for the crate itself. Thus "incremental" rebuilds might be slower but it also needs to reimplement less of cargo and therefore it is probably easier to keep compatible in edge cases. naersk also is pure nix and does not need "import derivation" for a workflow without manual build file regeneration. @nmattia

https://github.com/tenx-tech/cargo2nix is also a contender, maybe @ebkalderon can comment on the particular strengths. I think, cross-plattform builds are one goal.

https://github.com/williamyaoh/cratesio2nix, https://github.com/fractalide/nixcrates, https://github.com/fractalide/nix-crates-index, https://github.com/williamyaoh/manilla are all deprecated/outdated and probably should be removed from the Wiki

For the good of the ecosystem, we should converge on one or two recommended solutions. I think we should seeks solutions that support all widely used crates. What the best approach is, is less clear.

We could also try to share as much as possible and converge that way. E.g. split vendoring from Cargo.lock into one nix library, And maybe incorporate good ideas and/or code from tenx-tech/cargo2nix into buildRustCrate.

Maybe we should set up a virtual conference where all the maintainers and interested parties meet and share their thoughts on this. I have time this weekend or March 8th onwards.

[nmattia]

My 2¢:

I agree 100% with @kolloch 's analysis above. I'm away this weekend and back on Monday evening. @andir you live nearby, don't you? Any of the other "interested parties" in or near Zurich besides @kolloch and I?

Unrelated: I didn't know of https://nixos.wiki/wiki/Language-specific_package_helpers, any chance I can plug https://github.com/nmattia/snack in there?

[andir]

@andir you live nearby, don't you? Any of the other "interested parties" in or near Zurich besides @kolloch and I?

I am a 4.5h train ride away. Not terribly far away. I'd be willing to do that journey if that means we can improve the ecosystem.

[Mic92]

@nmattia I added snack to the wiki.

[nmattia]

@Mic92 thanks! while we're at it, should we add https://github.com/nmattia/napalm ?

[kolloch]

@nmattia BTW, you can log in with your github credentials and edit the wiki yourself.

[kolloch]

I am not 100% percent sure about the rules for nixpkgs, but I think that "import from derivation" is banned, correct? "naersk" currently depends on that, correct? @nmattia

crate2nix usually also assumes that your generated Cargo.nix is generated in your project directory or used in "import from derivation mode" (see tools.nix).

Probably something like this would be nice for packages in nixpkgs: https://github.com/kolloch/crate2nix/issues/102 Tell me what you think! @Mic92 @andir

[nmattia]

@kolloch naersk works with --restrict-eval true, --sandbox and --option allow-import-from-derivation false (for the last one: provided the source code doesn't come from a derivation).

[kolloch]

@nmattia so it wouldn't be a good match for providing packages in nixpkgs itself, right?

[nmattia]

That is correct!

[kolloch]

@nmattia maybe you could add support for out of tree sources as long as the Cargo.lock/toml files are checked in.

[ebkalderon]

Thanks for pinging me, @kolloch! I can give some context on the goals and design choices of the cargo2nix project as it currently stands.

cargo2nix isn't a personal project of mine, but is rather a shared effort, developed and endorsed by my current employer, TenX. Both my colleague, Anh Trinh (@trha), and I are actively developing and maintaining this project at the moment.

Our goals for cargo2nix are similar in some ways to crate2nix in that we want to build all nodes in the dependency graph in separate derivations, provide a reasonably user-friendly experience, and support building all widely used crates out of the box. However, we opted not to use buildRustCrate in favor of leveraging Cargo directly instead (both at codegen and at build time), basing our design on these blog posts written by James Kay (part 1, part 2). We want to be incredibly fast, both in terms of evaluation and build times, in the presence of very large Cargo.lock files, and we also want to be correct, in that we want to be capable of building any and all Cargo projects successfully without downstream modification.

Additional requirements for us include full support for cross-compilation, support for alternative registries, direct integration with the nixpkgs-mozilla overlay for selecting the Rust toolchain version to be used, and the ability to drop into a local nix-shell with a pinned Rust toolchain and all necessary system dependencies available in the environment in order to build a crate conventionally with cargo build or cargo test or iterate locally with rls or rust-analyzer.

At this stage, cargo2nix has been pretty heavily battle-tested in production and works very reliably for us, building thousands of public and private crates a day in CI/CD. But it isn't completely there in terms of user experience, particularly when compared to crate2nix. We don't currently support import-from-derivation, we require the use of a Nixpkgs overlay (though we could potentially embed most of that code inside the Cargo.nix itself in the future), and the Rust codebase needs a refactor and some polish. Also, we really need help in terms of high-quality documentation and sample projects.

CC @trha @TylerRichie @dingxiangfei2009 in case they have additional details to provide.

[ebkalderon]

Also, in regards to the ecosystem meeting, I don't think @trha and I have any immediate plans to fly out to Europe from Singapore, unfortunately. While that could potentially be arranged, we would both be equally happy to hop on a video conference call if the need be!

[kolloch]

@Mic92 how can I contact you? (e.g. Send me a direct message on Twitter (@pkolloch) or send the contact details to Andir or ... something you propose)

[P-E-Meunier]

I haven't tested Crate2nix, but I believe the main advantage of Carnix over it is the name. AFAICT, the design is much better and less hacky. Also, it's hard for me to find time to maintain Carnix.

[P-E-Meunier]

(I'm totally willing to donate the name, and review some code, btw)

[kolloch]

Thank you so much, @P-E-Meunier.

I'd say that I would not change the name to avoid confusion. But a note in the carnix repository would be appreciated. I'll work on updated nixpkgs documentation.

[573]

I haven't tested Crate2nix, but I believe the main advantage of Carnix over it is the name. AFAIC, the design is much better and less hacky. Also, it's hard for me to find time to maintain Carnix.

@P-E-Meunier, @kolloch how about car2nix ?

[P-E-Meunier]

Carnix 2? Car2nix sounds good too. I find that having multiple tools (some deprecated) actually brings more confusion, but I guess if the solution is well documented maybe not.

[P-E-Meunier]

@nmattia : why can't naersk build one derivation per crate? I don't know the internals, but that sounds like a fairly maintenance-free solution. One of the main difficulties I had with Carnix was following the "frequent" moves by Cargo and Rustc ("frequent" in my timescale, i.e. more than once a year).

[nmattia]

@P-E-Meunier two reasons:

1. When I tried it Nix segfaulted, most likely due to memory consumption. There were ~300 crates. I might have written something silly though.
2. I haven't had the time to look into it since, and I started questioning whether it is really a good idea. I doubt there are that many shared dependencies across projects (exact crate version + features + environment variables + exact rust version + exact cargo version). And within one project I don't understand what the benefit is. Since then I've been trying to make time to work on a different approach where rustc is replaced with a nix-build, to get more incrementality.

[P-E-Meunier]

1. That happened to me as well in early versions of Carnix/buildRustCrate. There are a few tricks, I don't remember what they were, but I could give it a try if there are still plans to maintain naersk.
2. There are two arguments for it:
   1. I guess it depends on how many projects you're working on at the same time. I am probably working at the moment on ~10 projects that I compile weekly (or so). Most of them have the same version of regex, compiled with the same rustc. Some are interdependent, and therefore have the same version of all dependencies, yet they're not in a workspace (both for technical reasons, and to make contributing easier).
   2. 50% of my hard drive (or so) is target directories, many of them for projects I started a few years back, and have abandoned since. If we had a fast enough alternative to compile with Nix (even if that means not doing the final step with Nix, but with a proxy to rustc/cargo that knows about Nix), just one run of nix-collect-garbage would collect it all. Else I'd have to spend a few hours sorting them out, but then it's cheaper to buy a new hard drive.

[573]

@P-E-Meunier there is even a logo idea in my head, what about the rust crab with nix's lambdas as scissors cartoon-style (hence car2... as in two or to ...nix).

[ebkalderon]

@nmattia

When I tried it Nix segfaulted, most likely due to memory consumption. There were ~300 crates. I might have written something silly though.

Interesting anecdote from my side: the Nix GC seems to have some severe memory consumption issues when applying deeply nested recursive overrides, which may have been the wall you might've hit. Early versions of cargo2nix, which builds each crate in a separate derivation, used to consume many gigabytes of memory and occasionally segfault on single projects with sufficiently large lockfiles (in one severely pathological case, max memory consumption by the evaluator was a whopping 20GB, IIRC), which made it completely unusable for everyday use.

However, some re-engineering of the design to avoid making deeply nested override calls (https://github.com/tenx-tech/cargo2nix/pull/14 and https://github.com/tenx-tech/cargo2nix/pull/27) reduced memory usage back to normal levels and eliminated the segfaults, and now, builds are very much manageable on a typical consumer laptop.

I'm not familiar with the precise conditions which caused these GC-related segfaults to arise, unfortunately. You might have to ask @dingxiangfei2009 for details, if you'd like to know more. I'm pretty sure this is symptomatic of an underlying bug in the evaluator, though.

[Mic92]

Afaik Eelco mentioned in one Nixcon talk that when doing an override nix cannot garbage collect its parameters leading to the massive memory usage.

[ebkalderon]

@Mic92 Ooh, nice! Any idea which Nixcon talk that was? It sounds interesting.

[Mic92]

It might have been the Roadmap one, I don't remember exactly.