[RFC 0123] Flake names

[schuelermine]

[RFC 0123] Flake names

Rendered

[edolstra]

Note that flakes had a name field but it was removed (80c36d4562af71a90c67b3adb886a1003834890e) because there was no clear use case / semantics. It also raises questions like whether names should be globally unique, and what should happen if they're not.

The RFC mentions discoverability but doesn't really explain how a name attribute helps here. Concretely, what does the name attribute provide to https://search.nixos.org/flakes that the repo URL doesn't already?

See also https://github.com/NixOS/nix/issues/5879 - we thought about bringing back name to be used for default flake output attributes (i.e. packages.<system>.<name> instead of packages.<system>.default).

The derivation that contains the flake’s content is called flake-source-${name} or, if a source control tag (e.g. git tag such as V3) is available, flake-source-${name}-${tag}.

This has the problem described here, namely that it reduces content-addressability by duplicating identical source trees when fetched from different flakes/tags.

[schuelermine]

1. In this case, the clear use case is usage in the derivation name for better UX. In my opinion the name should be entirely decorative or only used for default behaviors.
2. I find non-unique names are a non-issue, they should be disregarded, and are unlikely to be common. Currently, all derivation names are non-unique. Granted, it could confuse people into believing two derivations are of the same flake, but this is still an improvement over not knowing the source at all.
3. I thought the name could be used in the future for a more user-friendly UI. Discoverability of flakes isn’t necessarily helped as the global registry provides it, but it could make flake command usage easier and therefore more discoverable. However, I agree in hindsight that this is a mistaken inclusion, I will strike it from the RFC.
4. I see, the tag is possibly a bad idea. But why is changing the name bad for content addressability? It’s all about coercing derivations with different source expressions but the same result, right?

[JJJollyjim]

Are flakes not still an experimental feature that we're waiting for an RFC to formalise? It seems odd for this to come first.

[schuelermine]

I don’t know how else it would be possible to suggest this feature.

[edolstra]

This RFC is now open for shepherd nominations!

[lheckemann]

@schuelermine maybe you have some people in mind who would be suitable as shepherds? Feel free to nominate them, if so :)

[schuelermine]

@lheckemann I think I’m not familiar enough with the Nix development team & community to confidently nominate anyone in particular. The only people I know are ones I’ve more “seen” than “met”, and I don’t know of their specialties. How would I go about nominating someone?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-discussion-making-flakes-discoverable/18834/2

[rehno-lindeque]

Minor use case: flake-help

I use flake.description to print a help message. With this change I could also print flake.name (instead of passing name as another argument).

[jeff-hykin]

Coming from the related nix-discourse (making flakes indexable/discoverable), I'd like to propose simple change to make this forward-compatible and modular.

inputs and outputs are wonderfully elegant/generic names. So tacking on name, description, etc one after another seems like it could easily devolve into a name/pname/version issue, as a bunch of hardcoded edgecases that can never be removed without breaking things.

So, I'd like to propose a static field (name can be anything, I'm just going to call it "static" here) This would fit in nicely with inputs/outputs and be easy to explain as "flakes have only three top-level keys"

{
    static = {
        name = "anything";
        description = "blah blah blah";
        # arbitrary additional data
        license = "GPLv3";
        keywords = [ "image" "photo" "app" "image-editing" "image-viewing"  "python" ];
        code-source = "https://github.com/some_user/some_repo";
        cve-reports = "";
        bug-reports = "";
        homepage = "";
        icon = "https://";
        latest-sha256 = "";
        bin-names = [ "pip" "python" "python3" ]; # not guaranteed to match outputs, but for searching for a bin-name when package name is unknown
    };
    inputs = {
        nixpkgs = { url = "github:nixos/nixpkgs/nixos-unstable"; };
    };
    outputs = inputs : { };
}

The important aspects being;

1. static contains non-functional data (read "indexable/searchable data" )
2. The structure can be expanded as-needed in a forward-compatible way
3. Unlike inputs or outputs, static can be automated (ex: generated from json) for version bumps and hash changes, which helps solve the unmaintainability problem
4. Rather than must-have requirements, the structure can be flexibly checked against standards. For example:

• nix-search could have its own "baseV1" standard that requires name, description, and license.
  e.g. "Nix Search supports all flakes that comply with the baseV1 standard"
• Lazamar's version search could require having a version string
• NixOS could have an "ApplicationV1" standard that requires additional static fields to be considered an "app" flake
• New standards can build-upon or fork from this without breaking core nix functionality, since there's no guarantee a flake will even have a static field.
• Npm packages could have a requirement for is-npm-package, could require npm-version that follows semver pattern, and could have more strict requirements on name
• Even singular packages could have their own standard such as marking whether something was a LTS release or not

5. This allows systems outside of nix to leverage/understand packages without a full nix evaluation engine

[thufschmitt]

So, I'd like to propose a static field (name can be anything, I'm just going to call it "static" here)

Just bike-shedding on the name (because it’s too tempting), this kinda matches the meta attributes for packages, and I think it could be a good name here too.

(And regardless of the color of the bike, I agree that keeping this things scoped is a good idea, if only to keep the schema tidy)

[jeff-hykin]

So, I'd like to propose a static field (name can be anything, I'm just going to call it "static" here)

Just bike-shedding on the name (because it’s too tempting), this kinda matches the meta attributes for packages, and I think it could be a good name here too.

(And regardless of the color of the bike, I agree that keeping this things scoped is a good idea, if only to keep the schema tidy)

Haha yeah, I actually wanted to call it metadata (and would still like to). I avoided it since nix flake metadata is already a command that returns more just the static field. Maybe this can be meta, and nix flake metadata can be changed ~~since technically what's returned by nix flake metadata is the truly all the static data (I think), including metadata~~

The idea of static was both to make "non-changing" clear, and from thinking about the whole flake as a function with input args, output value, and static data. Maybe static could even be something passed to the output function.

[schuelermine]

I think static would conflict with the existing description field. Of course, it could be integrated, but that’d seriously expand the scope of this RFC. Maybe make your own RFC about a meta or static flake attribute?

[milahu]

• static contains non-functional data (read "indexable/searchable data" )

thats sounds overly limiting. scrapers must parse the .nix file anyway, and evaluation should be cheap enough

what if my flake.nix file has a million outputs, and each output has a different name/description/...?

[schuelermine]

what if my flake.nix file has a million outputs, and each output has a different name/description/...?

You wouldn’t need to evaluate outputs, though, right? Or is the no-thunks rule also active inside outputs?

[jeff-hykin]

Maybe make your own RFC about a meta or static flake attribute?

That's the plan 👍 I asked for related discussions on the discourse and this was the only one that popped up so far.

I think static would conflict with the existing description field. Of course, it could be integrated, but that’d seriously expand the scope of this RFC

I was hoping a set-attribute of static would be comparable in effort to string-attribute of name, but I think you're right since there's more to consider.

[jeff-hykin]

• static contains non-functional data (read "indexable/searchable data" )

thats sounds overly limiting.

Why? If you need functional/dynamic behavior, use outputs. Otherwise use static. It's meant as an additional capability, not a restriction.

On the flip side static is absolutely needed to index; Starting in 2020 I spent months trying to index nixpkgs and make them searchable. But the absolute bane of that endeavor was dynamically generated package names, versions, and other metadata. Some packages changed versions depending on what host-OS was performing the query, some were using fetch just for their version number. That makes it impossible to index what's available because we can't even list of all of what's available in an automated way. I also feel it's a security risk to be running random fetch calls from random packages just to find a package name/version.

Things like name cannot both be scrape-able and generated on-demand using stdenv, fetch, file read, etc.

scrapers must parse the .nix file anyway, and evaluation should be cheap enough

It is SO much easier to parse than evaluate. Even if evaluation is efficient, it is so so much slower. There's a reason nix-env -i is so slow that we can ripgrep the result faster. I don't even need a file system to parse nix; we can parse nix files even client side in a browser thanks to the tree sitter WASM module. Evaluation requires a specific environment setup, impure data injection, and backwards chaining of lazily evaluated expressions. None of those are friendly to scraping.

what if my flake.nix file has a million outputs, and each output has a different name/description/...?

Perfect! The old system (just a hard coded top-level description field) wouldn't be able to handle that case. And while this static system technically could handle that case by having a "sub-packages" field and putting them all in there, I would question if flakes were ever intended to represent a grouping of a million packages rather than one package per flake.

[milahu]

But the absolute bane of that endeavor was dynamically generated package names, versions, and other metadata.

we could cache these strings in flake.lock, which already has the "no eval" format (json) that you want otherwise, just exclude packages from your search-index that are too expensive to eval? (and notify maintainers)

when we introduce a new standard, then we get 3 types of packages: 1. the "good" packages that follow the new standard 2. the "neutral" packages that use the old standard, but metadata is fast to eval 3. the "bad" packages that use the old standard, but metadata is slow to eval

[jeff-hykin]

@milahu I think you're right that would be a lot better. That would allow arbitrarily complex functions to generate the metadata, and then the output would be easy to index 👍

[lheckemann]

@schuelermine sorry for the slow response --- just say that you nominate them in a comment here :)

[kevincox]

This RFC needs more shepherds! Please nominate those who you think could help pull this forward.

[schuelermine]

I nominate @alyssais and @edolstra for shepherd for this NFC

[schuelermine]

Are there any other contexts at the moment where the name of a store path depends on the contents of the store path instead of the expression that generates it? I’m not sure if flakes work that way right now. Can anybody confirm if the store path of a flake is determined before or after fetching it?

[schuelermine]

@thufschmitt I feel like this wouldn’t cause many expensive rebuilds, since the things defined in flakes don’t depend on the flake itself at all.

[thufschmitt]

Are there any other contexts at the moment where the name of a store path depends on the contents of the store path instead of the expression that generates it? I’m not sure if flakes work that way right now

On top of my head, this can happen in four situations:

1. Paths explicitly added to the store with nix-store --add/nix store add-path or equivalent
2. Paths added in the store as part of the evaluation ("${./foo}", the result of the builtins.fetch* family of functions, flake inputs)
3. Fixed-output derivations (all the nixpkgs.fetch* functions in particular)
4. The experimental content-addressed derivations (which aren’t really used in the wild yet)

Can anybody confirm if the store path of a flake is determined before or after fetching it?

Technically it is determined after. The logic of the fetcher is roughly:

1. Fetch the input in a temporary location
2. Serialize that as a NAR archive (more-or-less a tar, but normalizing all the filesystem metadata)
3. Hash that archive, and use that hash to compute the final store path
4. Extract it in the final store path

I feel like this wouldn’t cause many expensive rebuilds, since the things defined in flakes don’t depend on the flake itself at all.

You’re right, that probably wouldn’t change a lot

[schuelermine]

I have rewritten the RFC slightly to incorporate some of the drawbacks mentioned and for better formatting

[schuelermine]

After reading this

The RFC mentions discoverability but doesn't really explain how a name attribute helps here. Concretely, what does the name attribute provide to https://search.nixos.org/flakes that the repo URL doesn't already?

again, I’ve come up with some things: It makes the purpose of the flake clearer, as a flake could be mirrored from a different URL that doesn’t suggest the flake’s name already. It could also make it faster to make out the relevant part of the URL in flakes that have it in there. In general, the URL can be seen as an entirely functional device used for fetching the flake, which only happens to usually contain the flake name or suggest its purpose.

[schuelermine]

@lheckemann Are RFC shepherds supposed to be people who are interested in the RFC getting accepted, or just people familiar with the matter? My previous nomination was a bit uncreative and not fruitful, and sadly, I’m still not clear on how to choose good shepherd candidates…

[alyssais]

I'm afraid I'll have to decline the nomination. I don't use flakes, so I don't think I would be able to do a good job making sure this RFC is the best it can be.

Additionally, I don't think we should be doing RFCs for changes to flakes, until flakes themselves have gone through the RFC process and are no longer considered an experimental feature. Until then I think changes to flakes should be discussed on the Nix issue tracker. (I appreciate that the Nix issue tracker is a difficult place to get a response. :/)

[Ma27]

IMHO the proper solution would be to actually do flakes via RFCs rather than having a few blessed people doing what they want :shrug: