nix.dev

nix.dev copied to clipboard

based on @tazjin's tazjin/nix-1p and @zimbatm's NixCon 2019 talk Reading The Nix Language

As discussed in https://github.com/NixOS/nix.dev/issues/288

To render a preview:

git clone https://github.com/fricklerhandwerk/nix.dev
cd nix.dev
git checkout nix-language-tutorial
nix-shell
./live

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/attribute-set-or-set/14253/7

See https://github.com/NixOS/nix.dev/pull/286 for my WIP

• @fricklerhandwerk every attempt to write an introduction so far degenerated to what the manual should look like
  • idea: focus on only on the parts people will see most of the time
  • @infinisil omit obscure details such as dynamic attribute names
  • @domenkozar people will build simple configs, we should focus on most common work loads
    • @fricklerhandwerk trying to find out what the important 80% are, cannot do it on my own
      • @infinisil maybe just look at common Nix files and explain their components?
        • @fricklerhandwerk the thesis and manual already do this, but it's bumpy
          • the tutorial-style part is derived from the thesis
            • a long time ago was the best we had
              • only one of my ten study subjects found it at all, and had severe trouble
            • it should just go away
        • @domenkozar we should instead use concrete examples to test learning success
  • @infinisil saying Nix is JSON+functions is not enough
    • still need things like attribute acccess operators
  • @Mic92 it took me a while to understand that NixOS modules are functions
    • what should people learn from this, write packages or NixOS modules?
      • @fricklerhandwerk only read Nix code without fear
      • @infinisil there should be a specific tutorial for writing packages (and modules)
    • wonder if an online REPL would be a good way to help people learn
      • used one when learning programming with Ruby
      • @fricklerhandwerk if it shows to be helpful with actual learners we should invest the effort
      • definitely helped me with understanding how NixOS works
  • @Mic92 introduce unusual constructs that are used commonly in Nix
    • @fricklerhandwerk let us collect some
      • function application without parantheses
      • attribute set arguments
      • import
      • inherit
      • derivations
        • @infinisil this would introduce a lot of complexity and necessarily pull in nixpkgs
          • @fricklerhandwerk no, we just have to say that it is one of the two side effects without going into detail
            • so far we have a huge gap between pure language tutorials and what to do with them, so we need to introduce at least the principle of side effects
          • @domenkozar just show the practically usable bits, no one actually writes a derivation directly
      • string contexts
      • space-separated lists
        • with space-separated function application
          • @fricklerhandwerk the reference-style part has all of these things
            • it is already good in terms of contents, but fairly hard to read due to lacking structure and dense wording
  • @olaf start from the use case instead of listing all possibilities
    • attribute sets are a core feature, the material we build with, and
      • explain how to use them
    • @fricklerhandwerk agree, should start from expected knowledge (such as JSON) and build up sophistication guided by need
    • @Mic92 best books I learned from gave me exercises to test what I learned
      • @infinisil the draft has examples
        • @fricklerhandwerk I would like them to be interactive eventually
          • @domenkozar we can add that later
  • @Ericson2314 should go with this more experimentally
    • @fricklerhandwerk more user study sessions planned, first have to have a workable draft

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-07-28-nix-documentation-team-meeting-6/20627/1

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-34/20901/1

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-08-25-documentation-team-meeting-notes-8/21241/1

@domenkozar @infinisil this is ready for review.

From the CI:

Warning, treated as error:
/home/runner/work/nix.dev/nix.dev/source/tutorials/nix-language.md:301:broken link: https://nixos.org/manual/nix/stable/expressions/language-values.html#lists 

I'll look into getting cloudflare pages back again.

~This is fixed already, just re-run CI.~

I'll figure this out on Monday.

It seems like anchor changes from lists to list

• @fricklerhandwerk how to proceed with reviews?
  • it's a very long article
  • already tested successfully with two absolute beginners
  • just merge and fix forward?
• @infinisil would still like to review it in full
  • @domenkozar me too, will try to find time next week
  • is there a rendered version
    • @domenkozar we had netlify, but it appears to have broken when moving to the NixOS GitHub org
      • will fix this
      • if you move the repo to a different org you have to start over with Cloudflare
      • unfortunately nix.dev will be down until it is set up again (a few minutes)
• @fricklerhandwerk we can add more notes in <details> tag to adjust level of detail for readers
  • @infinisil have to strike a balance what is learning material and what should go to reference
  • can also split out details into separate articles if necessary

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-09-08-documentation-team-meeting-notes-9/21546/1

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/usability-study-session-1/21398/1

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/interpret-nix-expresssion-files/21570/2

@infinisil Thanks for the review so far.

I would like to quote @thufschmitt here:

Once agreed on a general direction – merge first, bikeshed later.

I love your rigorous feedback regarding being correct about PLT terminology. I'm mostly deliberately and only sometimes involuntarily a bit loose on some concepts to form a coherent narrative. If you can propose something that is more precise and doesn't break the overall structure it would be just great. Changing pieces in isolation would break that symmetry between overview and chapters, but we absolutely want to be predictable for readers and keep a pattern.

So I'd prefer we either merge it as is and work on a PR to get things right later; or you make a larger suggestion (or standalone PR) against this PR, which, if I understand correctly, may entail reordering sections or completely rewriting some passages to match the new structure.

@fricklerhandwerk if you rebase on top of master, we should (hopefully) get a preview link.

Once agreed on a general direction – merge first, bikeshed later.

While there's some parts of my review which are bike-shedding, the thing about advertising import as a side-effect is definitely not imo. That's just wrong and should be corrected before being merged. Yes it might require restructuring the tutorial, but that's what reviews are for. If I have time and motivation for this I'm gonna help out myself (I don't right now).

As a side note, I have a feeling that the Nix community in general tends to label everything "bike-shedding" even when it's not. Having discussions about details is not bike-shedding when those details are important.

Yeah import, builtins.readFile, etc. are cached in ways that only makes sense if we assume the underlying data doesn't change. And pure eval also affects what can be read in the first place. So at the very least, any impurity here is an unimportant detail.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-35/21701/1

@infinisil Sorry, I absolutely didn't want to sound dismissive of your comments. Proliferating the note on bike-shedding is of course driven by some frustrating experiences in the past, and it's also a sloppy and inconsiderate way of expressing that frustration. There is a recurrent pattern I notice where people (including myself) appear to jump between levels of detail, and often that's hard to follow due to how GitHub reviews are presented. That's all.

But that doesn't have anything to do with your work specifically. I agree on the side effects topic and will think it over. It has to be correct in the big picture.

Large reviews take a toll on everyone. I'm not happy this thing got so big, and the more I'm thankful you and @matta are taking time to help improve it.

I'm not sure how much you'll appreciate this, but I did a little exploration of how I'd organize such a tutorial. I focused on it being very incremental in terms of complexity and not having any prior knowledge. I like the categorization you did in https://github.com/NixOS/nix.dev/pull/267#discussion_r971337777, so I'm leaning onto that here. Note that I also listed some things yet missing from this tutorial (notably or, if then else, elemAt, ++ and more). I don't think we need to go in detail about all the operators, but pointing out that they exist would be good, and I think it would flow very nicely to introduce operators along the types they operate on. Let me know what you think of this

• Simple expressions (everything except functions declaration and function application)
  • Primitive data types and their operations: null, booleans (and !, &&, ||, ->, if then else), ints and floats (and +, - (incuding as negation), *, / and all the comparison operators), literal strings (without antiquotation, but with multiline, also + for string concatenation), paths (absolute, relative, and + specifically for path concatenation), mention that these evaluate to themselves
  • Basic variables: let in along with inherit syntax (don't mention attribute sets yet)
  • Composed data types and their operations: attribute sets (and . access, including or, // and ? also rec and with and inherit), lists (and ++, {builtins,lib}.elemAt), mention that these can contain any Nix expression, and that they evaluate to themselves too, mention equality comparison, that it works on both primitive and composed data types, == and !=. Here the note about it being similar to JSON up to this point (ignoring operators and variables) can be put
  • String antiquotation: Include basic escaping, how it interacts with paths, mention toString
• Declaring and applying functions: Single and multiple arguments (currying), attribute set arguments, ellipsis, with @, attribute, defaults, how to call these
• Builtins and library functions
  • Importing Nix expressions from files: import, mention that it's common to apply arguments to the imported expression
  • Tracing and errors: builtins.trace, throw, assert, also mention/link to lib.debug
  • Link to builtins and lib manuals
  • Derivation primitive, mention and link to stdenv.mkDerivation
  • Fetchers, builtins but also mention nixpkgs fetchers
• Impurities (especially relevant for pure evaluation/flakes): <NIX_PATH>, builtins.currentSystem, fetchers without hashes

@infinisil Thank you, I appreciate your input and its kind a lot!

Its great that you have written out in detail the orthogonal direction of what I intended to achieve, because so far I did not motivate my approach in depth. It helps me question, re-evaluate, and defend or give up the things I got convinced about in the process. Thank you.

The strategy you describe would probably be suitable for people who have barely any programming experience at all, and this is deliberately not the audience we want to address right now. I set out to explicitly follow the philosophy behind tazjin/nix-1p: you know how programming languages work in general, here are just the things unique to Nix that you will see every day. The diff to such a brief rundown is really just the level of detail and care, because we can't expect readers to fill the gaps, left by lecture notes such as nix-1p, on their own.

Specifically, this is why I would not even deal with booleans and conditionals, they are boring and unsurprising. They work like in any other language. I would also not deal with -> because, while surprising, it's rarely used in the wild. You'll deal with it when you're deep enough down the rabbit hole. Operators are mentioned alongside builtins, that should be enough. Read the reference manual.

One primary reason to go down that route is to save people's time. nix.dev motto is "getting things done". I see no reason teaching people basic programming language concepts here. The overview in terms of "elements of programming" is just to draw an outline, make a reminder of what it's all about, and encourage people not to feel overwhelmed early on.

This of course bears a risk of losing people when they are lacking prerequisites that we implicitly assume. We try to avoid that by stating them (if ever so vaguely), and at least set expectations for everyone involved.

I think your proposed outline would be very suitable for the natural follow-up on actually writing Nix language code. This is where I see elemAt, if ... then ... else, when to put parentheses around things, what to watch out for when merging attribute sets, and how to find and use common library functions.

There should also be a companion piece that ports Nix Language Quirks to nix.dev.

None of this is relevant for understanding tutorials here on nix.dev or reading existing code, though.

BTW: This is also why I introduce attribute sets first, it's the most prominent data type, and for some reason I think we should start with data although usually function declarations is literally the first thing you'd see in examples. The rest is also ordered by frequency of occurrence multiplied by weirdness from a non-Nixers perspective. That's why I'd rather had left paths and antiquotation for the end, after functions, but it would make a strange order, disconnected from the other simple expressions, and feel like an appendix for something that is still very frequently seen, even if it's not that peculiar.

A few details on your notes:

Basic variables: let in along with inherit syntax (don't mention attribute sets yet)

TIL that inherit also works for let ... in, i.e. the following is valid

let a = 1; in let inherit a; in a

1

I would absolutely definitely not include this here though, because I haven't consciously seen this in the wild in 4 years of using Nix.

Single and multiple arguments (currying)

According to Wikipedia's definition of currying, Nix doesn't have it. There are no multivariate functions, there is nothing to curry or uncurry. I stumbled over this multiple times when writing and rewriting the respective section. I've seen the term being used colloquially, and probably what we'd mean here is "curried functions" as used in the Wikipedia article (as that doesn't refer to the process of currying, but the form of representation), but I doubt it will make things clearer. I still think it may be worthwhile to link to that article for people who aren't familiar with that concept at all, if we find a way to express this gently. Maybe in the detailed explanation...

Also I thought about "simple expressions" vs "functions" again, and I'm still not convinced this is a suitable dichotomy. After taking a look at the introduction to Racket for a change of perspective, which use the word "simple" way too often for my taste, I'm not even sure what "simple" is supposed to mean. What makes (def name value) "simpler" than (fun arg) or (if cond then else)? Not to be dismissive, but that specific Racket guide fails to guide me as to what is most important to understand about Racket except "well, everything".

What would "simple" mean in our context here?

I agree that expressions denoting primitive values evaluate to themselves, okay. (Although that's not true for relative paths.) As said, I would like to structure the guide according to what each language constructs means when dealing with Nix language code, and its relative importance in expressing things we care about in the Nix ecosystem. The guide is to help people understand what we're trying to express with that code. A purely technical distinction at the PLT level simply doesn't seem fit for that purpose.

• @infinisil if then else should be mentioned, as it is different in every language
  • @fricklerhandwerk if then else is not even something you see regularly, it's usually wrapped
    • @infinisil let me open some random file in Nixpkgs... well, it's not that many instances indeed
      • it's still worth having it mentioned
    • sure, we should include it, just don't know where to put it
    • @infinisil can add a section on operators and accommodate if-then-else there
    • good idea. can we merge it with this structure already, and add the new section later?
• (lively review of structure)
  • No need to move up let in
  • Make rec less prominent
  • Side effect -> Impurity
  • guide -> tutorial
• @infinisil can live with that structure. not how I would write a tutorial, but it seems to get the job done.
• @Ericson2314: Is there another tutorial somewhere? Could we point out impurities better there?
  • @fricklerhandwerk a treatise on purity vs. impurity would be something that may actually grow best on top of the Nix Pills!
    • proposal for Nix Pills subtitle: Building the Nix ecosystem from first principles
    • idea:
      • reference = manuals
      • explanation = Nix Pills (grown to have more structure and cover more ground)
      • tutorials, guides ~ still have to figure that out
        • would like to talk again with @jonringer and @nrdxp on what we should focus on each
        • nix.dev says "getting things done", so probably that should be guides
        • doesn't matter much where things live or how they are called, as long as they are easy to find and meet user's needs
• @Ericson2314: Should the tutorial support random access?
  • @fricklerhandwerk: No, this would be for the reference manual
• Is import an impurity?
  • No, it's not relevant for reading the code anyway
  • @fricklerhandwerk the only thing I insist on is the distinction between stuff we see immediately in the code and stuff that is on the outside, such as environment variables and files on the network.
    • these may not be side effects, but they are also not under our full control
    • @LucPerkins how about "ambiguity"?
      • @fricklerhandwerk is that a common term? @matta already pointed out an instance where I (inadvertently) made up a term, and that was confusing to readers

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-09-22-documentation-team-meeting-notes-10/21906/1

The strategy you describe would probably be suitable for people who have barely any programming experience at all, and this is deliberately not the audience we want to address right now.

One or both of you may be interested in looking at a book: "COMMON LISP: A Gentle Introduction to Symbolic Computation." There is a PDF at the author's website here: https://www.cs.cmu.edu/~dst/LispBook/book.pdf

...not as an example of what this tutorial should be, but more of an example of just how much it takes to thoroughly explain things clearly.

The book is one of the best examples I have seen of slowly and thoroughly introducing a programming language to a reader who may not yet know how to program or even what programming really is. To give you a quick sense of how thoroughly the book treats the subject:

1. It takes 30 pages before the idea of a list is introduced...on a book teaching LISP!
2. It takes 80 pages before function definitions are shown in lisp syntax (parentheses, etc.). Before that it is all box flow diagrams.
3. It takes 100 pages before the idea of using the LISP repl is introduced.

I think it is quite reasonable to expect a short tutorial to take some short cuts. :-)

Case in point. And TBH I'd rather not spend years of my (or anyone's) life teaching someone the Nix language, of all things.

Something important I'd like to mention is that calling file paths and fetchers an impurity is not really accurate:

• Paths are considered pure if they are in the same "project scope" or "external", e.g. the same Git repository. In practice probably more than 99% of paths follow that pattern, and nobody considers these an impurity.
  • Also paths themselves aren't an impurity, it's reading those paths. Notably you can convert a path to a string without reading it using toString, though that's not very common.
  • Notably, absolute paths should always be considered an impurity.
  • Relative paths can also be impure, but only if they escape the project scope, e.g. with ../../../home/myUser/some/file
• Similarly, builtin eval-time fetchers like builtins.fetchurl are only considered an impurity if they don't provide a result hash. If they have a hash, Nix and the Nix community considers it pure. This is comparable to build-time fetchers (fixed-output derivations).
• In general, flakes (or more specifically pure evaluation mode) should be the gold standard for what is an impurity or not. We shouldn't call something an impurity if pure evaluation allows it

@tazjin Maybe try to not go into too much detail. We discussed this in the documentation team a bunch already, the goal of this tutorial is to get beginners to be able to read typical Nix code. The goal is not to document every feature and particularity of Nix (the reference can be a place for that, or we can have another Nix tutorial more like I described here), so things like dynamic keys, details about scoping and laziness should probably not be in scope.

Also, @fricklerhandwerk would like to get this PR merged sooner rather than later, mainly to pin the general direction of this tutorial. Minor adjustments can easily be made at a later point.

So I encourage you to especially review the structure and correctness of this PR and less details :smile:

Something important I'd like to mention is that calling file paths and fetchers an impurity is not really accurate:

@infinisil, I think there a difference between language level purity and build level purity that could be distinguished here.

Personally, I think language level purity and build level purity are completely different and separable concepts.

The current nix-language.md says "Purity is the key to reproducible builds." in a section that I think conflates these two kinds of "purities" without explanation. Language level purity is not necessary for reproducible builds, though it may be a useful programmer aid. Reproducible builds are achievable with shell scripts! :-)