[PoC] Package up `binary` targets & dependencies in a nix package or docker container

[ryanrasti]

Hello! I stumbled on nix and this repo a few days ago and really love the idea. Thanks so much for creating it. I can see so much potential here.

Anyways, I am trying to create a rule to package up a binary target into a docker container (with all of its nix dependencies) with bazel. e.g., in a rule.bzl file I have a command to generate a docker tar file:

load("@io_tweag_rules_nixpkgs//nixpkgs:nixpkgs.bzl", "nixpkgs_package")

def nix_image(name, target, data, visibility=None):
  # TODO: still need to get runfiles in!
  native.genrule(
    name = 'gen_image_%s' % name,
    outs = [name + ".tar.gz"],
    cmd = r'''
# TODO: hacky -- want to take in the transitive deps explicitly!

# find the *derivers* for the outputs paths:
f=$$(find | grep bazel-support/nix-out-link.drv)
echo "Found derivers: $$f"
lines=$$(echo $$f | xargs realpath | paste -sd "," -)
echo nix-build $$(realpath $(location //nix:docker.nix)) --arg lines \"$$lines\" >&2
build_out=$$(nix-build $$(realpath $(location //nix:docker.nix)) --arg lines \"$$lines\")
echo out: $$build_out >&2
ln -s $$build_out $(OUTS)
''',
    tools=["//nix:docker.nix"] + data + [target],
    visibility = visibility,
  )

Notice the above (hackily) looks for any bazel-support/nix-out-link.drv in the build context and then passes them all to this derivation (in //nix:docker.nix):

{ pkgs ? import <nixpkgs> {}, lines ? "" }:
let
  x = break;
  imports = map (l: import l) (pkgs.lib.strings.splitString "," lines);
in
pkgs.dockerTools.buildImage {
    name = "test-nix";
    tag = "latest";

    copyToRoot = pkgs.buildEnv {
      name = "image-root";
      paths = [pkgs.coreutils pkgs.bashInteractive] ++ imports;
      pathsToLink = [ "/bin" "/lib" ];
    };
}

But to get the .drv files which are needed as inputs (I couldn't figure out how to import the built derivations as inputs), I had to create the changes in this PR.

I am wondering if there is a way to build an image like this without having to modify rules_nixpkgs? If not, what'd be the best way to support it?

[aherrmann]

@ryanrasti Thanks for raising this, it's an interesting question!

A common way to create Docker images with Nix+Bazel is to use Nix to generate a base image, and then use Bazel's rules_docker to generate the final image that includes Bazel generated artifacts. See here for a description of this approach.

In that case you'd need to make sure that the Nix generated base image contains all the Nix store paths that are required in the final image.

Something worth pointing about the approach in this PR compared to how rules_nixpkgs usually works: rules_nixpkgs uses repository rules for Nix integration instead of regular build rules. But, this approach uses a regular build rule to invoke nix-build, generate the docker image, and link it to the output. The motivation for using repository rules is that Nix modifies files outside of Bazel's control, i.e. in /nix/store. Bazel doesn't track such changes and calling nix-build in a regular build rule could lead to dangling symlinks in a distributed use-case, e.g. a remote cache populated by another node.

[ryanrasti]

A very common case for me is that I want to rely on some binary being in PATH so that when I depend on a nix target in bazel I can easily invoke it via a subprocess. Do you currently have a recommendation for this?

Another solution I used is to have a wrapper script for my py_runtime that looks for all of the nix packages in runfiles and then dynamically forms PATH from all of their bin directories. But I am not sure if that is generally supposed to work.

[aherrmann]

A very common case for me is that I want to rely on some binary being in PATH so that when I depend on a nix target in bazel I can easily invoke it via a subprocess. Do you currently have a recommendation for this?

I'm not sure how this relates to Docker, could you expand on that?

Nonetheless, coming from Nix, this is understandable, as Nix manages PATH automatically in such a way. Bazel, OTOH, doesn't do that. Instead, it's usually on the rule author to manually extend PATH as needed. That said, a recent addition to rules_sh, namely sh_binaries, may be of interest to you. With that you could create a Nix import of the following form:

# WORKSPACE
nixpkgs_package(
    name = "hello",
    attribute_path = "hello",
    build_file_content = """\
load("@rules_sh//sh:sh.bzl", "sh_binaries")
sh_binaries(
    name = "hello",
    srcs = ["bin/hello"],
    visibility = ["//visibility:public"],
)
""",
)

And then use it as follows

genrule(
    name = "some-genrule",
    toolchains = ["@hello"],
    cmd = """\
PATH="$(_HELLO_PATH):$$PATH"
hello >$@
""",
    outs = ["output"],
)

[benradf]

Closing as it's been nearly a year with no changes now. Feel free to re-open if this is something you're still interested in getting merged @ryanrasti.