Image tag differs depending on OS

[YorikSar]

trafficstars

Currently image tag depends on which OS nix2container is executed on.

For example, with flake.nix:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    nix2container.url = "github:nlewo/nix2container";
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, nix2container, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system: {
      packages.image = nix2container.packages.${system}.nix2container.buildImage {
        name = "registry.example.com";
        contents = [ nixpkgs.legacyPackages.x86_64-linux.bashInteractive ];
      };
    });
}

We get different tags for different systems:

 % nix eval .#packages.x86_64-linux.image.imageTag
"35gn4cdzaky8lvp576msc101djl8xz51"
 % nix eval .#packages.x86_64-darwin.image.imageTag
"bjnx63baqkxwb9317s1908yjvvg1r3l8"

This is caused by nix2container using outPath of pkgs.runCommand that depends on the system for which scripts are generated, not on the actual image inputs.

My use case is: I want to be able to push images from macOS and from Linux in the same way and get the same results.

[nlewo]

The image also depends on the nix2container binary since nix2container could change they way it generate images. When an image (with same inputs) is generated with 2 different nix2container versions, i don't think we want to get the same output hash. This means we would need to consider the hash of nix2container binary derivation to generate the image. I imagine the nix2container binary derivation hash differ between macOS and Linux. So, it doesn't seem to be possible to get the same output hash on macOS and Linux.

Note I'm really not familiar with cross system builds, so i could be totally wrong :confused:

Do you know if this is possible with dockerTools.buildImage?

Also, I'm wondering if the intentional store (no idea about its status) could be used since using the generated image content hash could do the job.

Finally, we could use the image digest but you would need to build the image to get its digest, to modify your deployment files at runtime (and it's pretty hard to get the digest without pushing the image to a registry)...

[YorikSar]

The image also depends on the nix2container binary since nix2container could change they way it generate images. When an image (with same inputs) is generated with 2 different nix2container versions, i don't think we want to get the same output hash. This means we would need to consider the hash of nix2container binary derivation to generate the image. I imagine the nix2container binary derivation hash differ between macOS and Linux. So, it doesn't seem to be possible to get the same output hash on macOS and Linux.

We don't have to hash the binary itself. We could in theory use hash of the sources and assume that the binary produces the same result on all systems (which is hopefully the fact now).

Do you know if this is possible with dockerTools.buildImage?

I didn't check it, but I think that it doesn't generate any tag, only provides the image tarball. Docker/OCI don't really care about tag names - they are just for user convenience.

Also, I'm wondering if the intentional store (no idea about its status) could be used since using the generated image content hash could do the job.

Do you mean Content Addressable derivations? Yes, if the output is the same for all systems (as it should be), we could mark this buildImage derivation as content addressable, but then to know its hash we need to build it.

Finally, we could use the image digest but you would need to build the image to get its digest, to modify your deployment files at runtime (and it's pretty hard to get the digest without pushing the image to a registry)...

Well, you don't have to build the image itself, you just need to create and hash its manifest. You'd need hashes of all layers for that, of course, but that can be done in memory.

In fact, it is recommended to use image digests for deployment because it pins images and cannot be changed like tags. Maybe we should provide a way for user to ask for the image digest in Nix?..

[nlewo]

We don't have to hash the binary itself.

I didn't mean hashing the nix2container binary, i was only talking about the hash of the derivation which builds the nix2container derivation. Sorry if it was not clear enough.

We could in theory use hash of the sources and assume that the binary produces the same result on all systems (which is hopefully the fact now).

Hm, this looks fragile and i would prefer to avoid doing such kind of hacks.

Do you know if this is possible with dockerTools.buildImage?

I didn't check it, but I think that it doesn't generate any tag, only provides the image tarball. Docker/OCI don't really care about tag names - they are just for user convenience.

fyi, the dockerTools.buildImage produces a Docker archive which contains a tag (in the image manifest IIRC).

Do you mean Content Addressable derivations? Yes, if the output is the same for all systems (as it should be), we could mark this buildImage derivation as content addressable, but then to know its hash we need to build it.

Yes, i meant Content Addressable derivations.

Finally, we could use the image digest but you would need to build the image to get its digest, to modify your deployment files at runtime (and it's pretty hard to get the digest without pushing the image to a registry)...

Well, you don't have to build the image itself, you just need to create and hash its manifest. You'd need hashes of all layers for that, of course, but that can be done in memory.

In fact, it is recommended to use image digests for deployment because it pins images and cannot be changed like tags. Maybe we should provide a way for user to ask for the image digest in Nix?..

Yep, i think it could be pretty nice to be able to get the digest at build time! The drawback by using digests is that we would need to read all layers of an image each time one of these layer is updated (i don't know how the digest is computed and if we could cache layer digests to compute the image digest...).