Improve compiling containers on non-Linux systems

[jmgilman]

This issue documents a usability issue but does not currently offer a solution.

As it stands, executing copy2DockerDaemon will build the container locally and then push it to whatever Docker registry is configured on the local system. In non-Linux systems, this presents a problem because the Docker daemon is running inside a Linux VM. The result is that the container is typically built for macOS/Windows and then uploaded to a Docker daemon running under Linux (this is true for any runtime, including podman, containerd, etc.).

I'm not sure if this problem can be solved entirely within nix2container or if some documentation can be generated with a workaround for users in this context. The most obvious solution is to just run everything in a Linux VM and forward the Docker socket to the local machine (this is what I'm currently doing using lima).

Another potential solution that I think is worth exploring is using remote builders to build the OCI image. The primary problem I see here is that the image isn't actually stored in the nix store (which I understand is a good thing), so figuring out how to actually get the image built remotely is the primary challenge.

There may be other solutions that I'm not aware of. I figured this issue is a good starting place to discuss potential solutions and their pros/cons.

[nlewo]

Another potential solution that I think is worth exploring is using remote builders to build the OCI image.

I think this is what @nktpro is doing. Currently, there are however still some issues because of the "nix case hack" required on aufs (see #48 and https://github.com/nlewo/nix2container/issues/37), but i started to work on these issues.

[jmgilman]

I'd be curious to learn how this is being implemented. I don't think a typical nix remote builder can work since the image isn't put in the store. There may be some workarounds I'm not aware of.

[nktpro]

@jmgilman Currently the workaround is to output the image in skopeo dir format as another derivation, for example:

image = nix2container.buildImage {
 ...
};
image-as-dir = runCommand "image-as-dir" { } "${image.copyTo}/bin/copy-to dir:$out";

We can then build image-as-dir remotely on any aarch64-linux or x86_64-linux builders from macOS/Windows, then simply copy the result to any registry, e.g:

skopeo copy --insecure-policy dir:./result ...

The obvious drawback of this approach is that it defeats the strength of nix2container altogether, as it shares all the inefficiencies of dockerTools in nixpkgs.

[nlewo]

@jmgilman The remote builder builds the image.copyTo script (with all of its dependencies, ie the image itself) and your local deamon asks the remote store for the script image.copyTo which depends on all store paths required to push the image. Your local deamon just has to copy all store paths used by the image and builds the Skopeo binary which is then used to push the image: the image has been built by the remote builder and Skopeo is built and executed locally.

However, this is not fully working yet (see above mentioned issues).