rfcs
rfcs copied to clipboard
NixOS
[RFC 0017] Intensional Store
This RFC builds on the implementation of RFC #62 to maximize its benefits.
- Decouple metadata (Trust DB) from the Store, keep per-user
- Move the Store to
/var/lib/nixfor better compatibility with other platforms - Store can be shared read-write on a network share
-
nix-daemonbecomes optional - Store paths provide dependencies without database
I wonder if this can already be (partly) implemented as a wrapper applied in mkDerivation by using https://github.com/NixOS/nix/pull/1502 to define __toString.
Even if we only have a manual way of transforming a drv to CAS, it would already be useful and we don't have to make it work for corner-cases like when $out has references to $out.
It would allow to break unnecessary coupling, for example here: https://github.com/moretea/yarn2nix/blob/ebde7ffe0f515900d591a9786d29898d5cea63bb/default.nix#L102-L104
A src is given which is written to the /nix/store. Then helpfully we deduce the package.json from that src. But because the src store path will change if any of the files have changed, the package.json reference will also necessarily change and create unnecessarily rebuild of the node_modules.
Sadly I don't have time to work on this right now, but I still definitely want to do it. I already prototyped it a bit when I originally posted it, and it didn't make a huge difference but it does make a difference. My approach will be to find where in Nix the checksum calculations are done, implement a version that ignores the own path, and then see where that leads me.
On Sun, Nov 5, 2017 at 7:33 PM zimbatm [email protected] wrote:
It would allow to break unnecessary coupling, for example here:
https://github.com/moretea/yarn2nix/blob/ebde7ffe0f515900d591a9786d29898d5cea63bb/default.nix#L102-L104
A src is given which is written to the /nix/store. Then helpfully we deduce the package.json from that src. But because the src store path will change if any of the files have changed, the package.json reference will also necessarily change and create unnecessarily rebuild of the node_modules.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/NixOS/rfcs/pull/17#issuecomment-341994455, or mute the thread https://github.com/notifications/unsubscribe-auth/AADWloETYRSpLbOafvayeMebm7cOepKnks5szf94gaJpZM4O1LHU .
Do you think that there is a way to implement a prototype that doesn't require a new nix builtin?
Hmm sure, to get an idea, yes. It won't actually help builds or hydra but it will give an idea of what the store will look like. Although the prototyping I did on nix-dev already gave a good insight into that…
On Sun, Nov 5, 2017, 10:00 PM zimbatm [email protected] wrote:
Do you think that there is a way to implement a prototype that doesn't require a new nix builtin?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/NixOS/rfcs/pull/17#issuecomment-342005180, or mute the thread https://github.com/notifications/unsubscribe-auth/AADWlvgmr-En2f8MjEcbcfEJfMePqTrlks5sziIAgaJpZM4O1LHU .
@tomberek iirc @edolstra has started adding intensional features to nix, not sure whether those conform to the ideas in this RFC.
@wmertens We're doing triage on the backlog of open RFCs. Are you still interested in pursuing this RFC?
BTW, I made a bit of progress towards content-addressable stores: https://github.com/edolstra/nix/commit/a4d0a1162c55614257f944d058c1f6293cb5f459. It doesn't support automatically moving derivation outputs to content-addressable locations yet, though.
Yes, I still think it will be a valuable addition. What is the next step?
Wout.
On Thu, Jan 17, 2019 at 12:05 PM Eelco Dolstra [email protected] wrote:
@wmertens https://github.com/wmertens We're doing triage on the backlog of open RFCs. Are you still interested in pursuing this RFC?
BTW, I made a bit of progress towards content-addressable stores: edolstra/nix@a4d0a11 https://github.com/edolstra/nix/commit/a4d0a1162c55614257f944d058c1f6293cb5f459. It doesn't support automatically moving derivation outputs to content-addressable locations yet, though.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NixOS/rfcs/pull/17#issuecomment-455133030, or mute the thread https://github.com/notifications/unsubscribe-auth/AADWlplnljIm0VW5S-aVD-hSznsXVUEbks5vEFjwgaJpZM4O1LHU .
The next step is to get an RFC shepherd team. @globin I think you said you would open it for nominations?
This pull request has been mentioned on Nix community. There might be relevant details there:
https://discourse.nixos.org/t/rfc-17-open-for-shepherd-nominations-intensional-store/1934/1
I'd like to nominate @vcunat and @fpletz for https://github.com/NixOS/nix/issues/859 and https://github.com/orgs/NixIPFS/people
Yes, I'd like to review the design for (future) "intensional store", regardless of getting into the team.
Before reviewing the details, I do have one high level concern.
To quote from the README:
Furthermore, the fact that a given RFC has been accepted implies nothing about what priority is assigned to its implementation, nor does it imply anything about whether a Nix/nixpkgs developer has been assigned the task of implementing the feature. While it is not necessary that the author of the RFC also write the implementation, it is by far the most effective way to see an RFC through to completion: authors should not expect that other project developers will take on responsibility for implementing their accepted feature.
Assuming this does get accepted, do we have someone ready to implement it? Changes this deep to the store model involve a significant amount of work, I'm a bit worried about this RFC getting approved but no one implementing it for a long time, at which point many of the considerations that go into the current design may have changed drastically.
@shlevy I wonder if it would be a good idea to try making a survey of what parts are currently available (even before answering your question, maybe)
Yes, definitely. I may even scrape some time to do the actual implementation.
Per meeting of the @NixOS/rfc-steering-committee: Shepherding Team is @shlevy as leader, @vcunat, @edolstra and @nbp (as long as he agrees otherwise we will follow up with further candidates).
EDIT: Also shepherds, please be reminded that it could make sense to have a chat on IRC or by videoconferece, to discuss your opinions and with the author to move this forward in an orderly and constructive way!
This pull request has been mentioned on Nix community. There might be relevant details there:
https://discourse.nixos.org/t/shepherds-chosen-for-rfc-13-17-and-39/1964/1
@wmertens There seems to be quite a bit of unaddressed commentary here. Do you want to go through it on your own time, or we can set up a call/chat in IRC to walk through with the shepherd team?
@shlevy walking through would be great! How about this wednesday evening, 8:30PM CET/2:30PM EST? We could use https://talky.io/
@wmertens I'm booked from 2:30-5:30 ET on Wednesday, I can do any time before or after then though.
I will update this RFC with the following over the next few days:
- names should not be in cas paths
- derivations should use cas inputs + the implications of not knowing intermediate cas results while deciding what to build (loop after realization?)
- impacts section describing how the nix ecosystem will change
- motivations: build shortcutting, forcing tooling improvements (no names means requiring better query tools), self-contained packages with "blockchain" secured import trees (bear with me, this needs better descriptions), rewarding binary reproducibility, crowd-sourced ihash-to-cas mapping, shortcutting builds from cross tools…
- minimal store/db changes required + migration path + how to allow other store changes in the future