nix.dev

nix.dev copied to clipboard

I have a conundrum! I would really appreciate some words of wisdom from nix gurus.

I am writing some data-analysis utilities that are glued together with nix and I am wondering about the best way to provide simple user-interfaces for them (e.g. from the unix shell.)

For example, I have a utility that processes profiler data files to produce textual reports (similar to perf report), and the current user-interface is like this:

$ nix-build -E '(import studio/tools/vmprofiler {}).analyze_dir(/var/run/snabb/12345/engine/vmprofile)'

but that's quite verbose and users would probably much prefer a usage like:

$ vmprofiler -i /var/run/snabb/12345/engine/vmprofile -o .

and now I am wondering how to at least partially bridge that gap so that the command-line interface won't be a barrier to adoption of the tools.

I suppose that I have at least three potential ways to make a shell script wrapper:

• Make it use the #!nix-shell shebang to run the tools directly with the right env.
• Make it call nix-build directly in the script (wrapper to current usage.)
• Make it into a nix package that is installed with nix-env.

I am somewhat drawn to calling nix-build because I like the idea of copying the input data into the nix store. These tools can potentially operate on large amounts of data that requires expensive preprocessing and I'd love for nix to transparently cache this intermediate processing. So while I want to have a convenient user-interface I am also happy to have nix as a runtime dependency and to use it as effectively as possible.

Does anybody see the dilemma and have an idea for how to sort it out?

Here are a couple of the scripts I am talking about:

• timeliner/default.nix
• vmprofiler/default.nix

"The problem" is to provide command-line usage instructions that won't make people go "ugh" and also won't give up nice benefits of deep nix integration.

For styx I went for nix-build wrapped in a script and packaged so it can also be easily installed.

It has the advantage to make possible to run tools from the script that access the generated store product without polluting the current directory with results dir and generating unwanted gc roots (granted --no-out-link is used).

Also, a thing that can really improve performance in case the utility generate many store products is to set preferLocalBuild = true; and allowSubstitutes = false; to every runCommand / mkDerivation call.

@ericsagnes Thanks, looks like you have blazed the trail ahead of me! I really like your installation steps.

I marked this as stale due to inactivity. → More info

Can this be closed or should be turned into a topic?

It would be great to have a conventional way to do these things. Building lightweight scripts is easy, just use writeShellApplication and add them to your shell environment. But once you want something that's more integrated with your Nix project, it quickly gets hairy. A typical problem you run into is calling nix-build on something from your source, which is brittle if source files move around. If you bake the sources into the expression, your evaluation time and closure size may explode. I've built a bit of convenience for myself, and while it does the job, it's still cumbersome to use and needs user feedback.

I'm a bit wary of trying to write tutorials for use cases that aren't well-established, because you'll spend a lot of time on research and testing, and may end up write the actual tooling, at which point it may become a contribution to Nixpkgs instead. But I'll leave a label, maybe there's more interest in that.