nlewo
Layers contains duplicate dependencies
When using layers results in layers containing multiple identical packages
Example:
{
inputs.nix2container.url = "github:nlewo/nix2container";
inputs.nixpkgs.follows = "nix2container/nixpkgs";
outputs = { self, nixpkgs, nix2container }:
let
pkgs = import nixpkgs { system = "x86_64-linux"; };
n2c = nix2container.packages.x86_64-linux.nix2container;
in
{
packages.x86_64-linux.default = n2c.buildImage {
maxLayers = 100;
name = "hello";
config = {
entrypoint = [ "${pkgs.hello}/bin/hello" ];
};
layers = [
(n2c.buildLayer {
deps = [ pkgs.bash ];
})
(n2c.buildLayer {
deps = [ pkgs.zsh ];
})
];
};
};
}
Result: glibc (and few other deps) exist in both bash and zsh layers
I noticed there is ignore option but seems to accept only a single store path. Is there a way to deduplicate derivations?
Actually, I think I'm wrong, I think I was misreading dive output. Can somebody confirm this?
No, it does seem to contain duplicate files. I cleared docker directory and checked /var/lib/docker/overlay2 after copying single image to the daemon
@gytis-ivaskevicius since these layers are built in non-dependent derivations, we can't remove duplicated files.
There are however several way to fix this.
- put these packages in the same layer
- you can create a dependency between these layers. Something such as:
layers = [(n2c.buildLayer {
deps = [ pkgs.bash ];
layers = [
(n2c.buildLayer {
deps = [ pkgs.zsh ];
})]
- (not sure this work in practice but could theorically work): create a layer containing the glibc and add this layer to both layers:
layers = [
(n2c.buildLayer {
deps = [ pkgs.bash ];
layers = [
(n2c.buildLayer {
deps = [ pkgs.glibc ];
})
(n2c.buildLayer {
deps = [ pkgs.zsh ];
layers = [
(n2c.buildLayer {
deps = [ pkgs.glibc ];
})
Can you explain how the second option works (creating a dependency)? I'm running into the same issue and haven't been able to wrap my head around why sometimes I get duplicate packages and sometimes I don't.
Here is another example:
{
inputs.nix2container.url = "github:nlewo/nix2container";
outputs = { self, nixpkgs, nix2container }:
let
pkgs = import nixpkgs { system = "aarch64-linux"; };
nix2containerPkgs = nix2container.packages.aarch64-linux;
n2c = nix2containerPkgs.nix2container;
test1 = pkgs.writeScriptBin "test1" ''
#!${pkgs.runtimeShell}
echo "Test1"
'';
test2 = pkgs.writeScriptBin "test2" ''
#!${pkgs.runtimeShell}
echo "Test2"
'';
in
{
packages.aarch64-linux.hello = n2c.buildImage {
name = "hello";
config = {
entrypoint = [ "${pkgs.hello}/bin/hello" ];
};
layers = [
(n2c.buildLayer {
deps = [ test1 ];
})
(n2c.buildLayer {
deps = [ test2 ];
})
];
};
};
}
The same issue of glibc getting put into both layers exists. One other oddity I've noticed is that if you do make dependent layers, for some reason everything gets squashed to a single layer in the resulting image. For example:
{
inputs.nix2container.url = "github:nlewo/nix2container";
outputs = { self, nixpkgs, nix2container }:
let
pkgs = import nixpkgs { system = "aarch64-linux"; };
nix2containerPkgs = nix2container.packages.aarch64-linux;
n2c = nix2containerPkgs.nix2container;
test1 = pkgs.writeScriptBin "test1" ''
#!${pkgs.runtimeShell}
echo "Test1"
'';
test2 = pkgs.writeScriptBin "test2" ''
#!${pkgs.runtimeShell}
echo "Test2"
'';
in
{
packages.aarch64-linux.hello = n2c.buildImage {
name = "hello";
config = {
entrypoint = [ "${pkgs.hello}/bin/hello" ];
};
layers = [
(n2c.buildLayer {
deps = [ pkgs.glibc ];
layers = [
(n2c.buildLayer {
deps = [ test1 ];
})
(n2c.buildLayer {
deps = [ test2 ];
})
];
})
];
};
};
}
Produces a single layer that contains everything, including the entrypoint. Is there a way to avoid this?
Running into the same issue, this makes the layers feature a bit pointless I'm afraid.
@jmgilman Sorry, i forgot to answer your questions:/
Can you explain how the second option works (creating a dependency)?
Considering the following example:
layers = [(n2c.buildLayer {
deps = [ pkgs.bash ];
layers = [
(n2c.buildLayer {
deps = [ pkgs.zsh ];
})]
We are building a layer (lets call it layerA) containing pkgs.bash having another layer as a dependency. This second layer (layerB) contains pkgs.zsh. These two layers are two Nix derivations where the Nix derivation to build layerA depends on the layerB Nix derivation. nix2container starts to build layerB and when building layerA, it can remove all store paths already existing in the layerB since their is a relation ship between these two layers: it is then guarantees store paths removed from layerA are actually provided into the image by layerB. Then all common store paths between layerA and layerB are removed from layerA.
Produces a single layer that contains everything, including the entrypoint. Is there a way to avoid this?
Hm, i'm surprise since i would expected at least 3 layers. I need to reproduce to understand what is happening.
However, this is not the way it is supposed to work;) I think you would instead want something such as:
layers = let glibcLayer = n2c.buildLayer {
deps = [ pkgs.glibc ];};
in [
(n2c.buildLayer {
deps = [ test1 ];
layers = [glibcLayer];
})
(n2c.buildLayer {
deps = [ test2 ];
layers = [glibcLayer];
})
];
I never tried such kind of things, but i would expect to produce an image with the glibcLayer specified two times in the image OCI manifest (but Skopeo will only have to push a single time).
@niklaskorz if my above comment doesn't help you, do not hesitate to share your issue/usecase.
@niklaskorz if my above comment doesn't help you, do not hesitate to share your issue/usecase.
I'll try to come up with a "smallest breaking example" case and report back :)
@nlewo so here's my example:
[click here to view] flake.nix
{
description = "A very basic flake";
inputs = {
nix2container.url = "github:nlewo/nix2container";
nix2container.inputs.nixpkgs.follows = "nixpkgs";
};
outputs = { self, nixpkgs, nix2container }: let
build_system = "x86_64-darwin"; # update to the system you're building on
host_system = "x86_64-linux";
build_pkgs = nixpkgs.legacyPackages.${build_system};
host_pkgs = nixpkgs.legacyPackages.${host_system};
n2c = nix2container.packages.${build_system}.nix2container;
in {
packages.${build_system}.default = let
find_pkg = name: list: build_pkgs.lib.lists.findFirst (x: (x.pname or "") == name) null list;
package = host_pkgs.salt;
python = find_pkg "python3" package.nativeBuildInputs;
in n2c.buildImage {
name = "saltstack";
copyToRoot = [
(build_pkgs.buildEnv {
name = "root";
paths = [ package ];
})
];
layers = [
# (n2c.buildLayer {
# deps = [ python ];
# })
# (n2c.buildLayer {
# deps = package.propagatedBuildInputs;
# })
(n2c.buildLayer {
deps = [ python ];
layers = [
(n2c.buildLayer {
deps = package.propagatedBuildInputs;
})
];
})
];
};
devShells.${build_system}.default = with build_pkgs; mkShell {
packages = [
dive
];
};
};
}
[click here to view] flake.lock (probably not needed)
{
"nodes": {
"flake-utils": {
"locked": {
"lastModified": 1653893745,
"narHash": "sha256-0jntwV3Z8//YwuOjzhV2sgJJPt+HY6KhU7VZUL0fKZQ=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "1ed9fb1935d260de5fe1c2f7ee0ebaae17ed2fa1",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
},
"nix2container": {
"inputs": {
"flake-utils": "flake-utils",
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1688922987,
"narHash": "sha256-RnQwrCD5anqWfyDAVbfFIeU+Ha6cwt5QcIwIkaGRzQw=",
"owner": "nlewo",
"repo": "nix2container",
"rev": "ab381a7d714ebf96a83882264245dbd34f0a7ec8",
"type": "github"
},
"original": {
"owner": "nlewo",
"repo": "nix2container",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1690441914,
"narHash": "sha256-Ac+kJQ5z9MDAMyzSc0i0zJDx2i3qi9NjlW5Lz285G/I=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "db8672b8d0a2593c2405aed0c1dfa64b2a2f428f",
"type": "github"
},
"original": {
"id": "nixpkgs",
"type": "indirect"
}
},
"root": {
"inputs": {
"nix2container": "nix2container",
"nixpkgs": "nixpkgs"
}
}
},
"root": "root",
"version": 7
}
So to summarize:
container with single layers entry:
layers = [
(n2c.buildLayer {
deps = [ python ];
})
];
Things work great, that dependency is placed in separated layer and excluded from the final one.
container with two entries in layers
After I add another layer, I'm getting the issue that was originally reported, and that you mentioned it is happening because one layer is unaware of another one. nix develop -c dive <docker image id> shows duplicates.
containers with nested layers
An approach mentioned here:
layers = [
(n2c.buildLayer {
deps = [ python ];
layers = [
(n2c.buildLayer {
deps = package.propagatedBuildInputs;
})
];
})
];
This supposed to fix the issue, but I'm getting a single layer. For example output from nix build, the file ./result looks like this:
[click here to view] ./result
{
"version": 1,
"image-config": {},
"layers": [
{
"digest": "sha256:ed20f1a86223da2bcc9125bb240e562de4d176cfc1331130b33c469c62046b0e",
"size": 272749056,
"diff_ids": "sha256:ed20f1a86223da2bcc9125bb240e562de4d176cfc1331130b33c469c62046b0e",
"paths": [
{
"path": "/nix/store/ic9wnagwh22yhqh3lcdlnv5m178w6f0f-libunistring-1.1"
},
{
"path": "/nix/store/a9mxddm4a5p4kp84vys4n2nrmwqgk7kr-libidn2-2.3.4"
},
{
"path": "/nix/store/2y9zl8ky5ac28ali6ly1zfz11d4fq48b-xgcc-12.3.0-libgcc"
},
{
"path": "/nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8"
},
{
"path": "/nix/store/9gyhjdryy7g26frwsxsbs0gdffjbaxr4-ncurses-6.4"
},
{
"path": "/nix/store/5p62fc7h9ij36fqsxlbq73mbwdhnmbkv-zlib-1.2.13"
},
{
"path": "/nix/store/m6kphh9rh424vbw5wq84jcg0r4is4sc5-gcc-12.3.0-libgcc"
},
{
"path": "/nix/store/2fpmbk0g0ggm9zq89af7phvvvv8dnm7n-gcc-12.3.0-lib"
},
{
"path": "/nix/store/p6dlr3skfhxpyphipg2bqnj52999banh-bash-5.2-p15"
},
{
"path": "/nix/store/9w5faa08asvglqhywg43nrz2vh0asfn0-libffi-3.4.4"
},
{
"path": "/nix/store/q3lnkq87ckh490dplz1rg8l90xnp3h25-mailcap-2.1.53"
},
{
"path": "/nix/store/nm53mzyb0ndl4mmfjzrwb8vwmgjik4ik-bzip2-1.0.8"
},
{
"path": "/nix/store/nhnxylvmaisx7pw8pyyx3a8cq93xanpx-libxcrypt-4.4.36"
},
{
"path": "/nix/store/k0r87f1hh9bxf6s2vkkkrrlwfkxsn1jx-expat-2.5.0"
},
{
"path": "/nix/store/iddgks1c4nd16vm50pfxj0ivlm4awl1k-readline-8.2p1"
},
{
"path": "/nix/store/hwdckdgh3y73qravhmx7dmkv825zbipi-tzdata-2023c"
},
{
"path": "/nix/store/fvzc0n5c4r13mpqv5jrq0vkjw7r5jndn-openssl-3.0.9"
},
{
"path": "/nix/store/fr2hlcmhwwbrxyzidbmgsam8b6nd7lr1-sqlite-3.42.0"
},
{
"path": "/nix/store/d00zy8wpgbpi3d87b6r7n8zbcgs6maqh-xz-5.4.3"
},
{
"path": "/nix/store/ad8ivr9pkzw1w01mvq88q3ra601zpd6c-gdbm-1.23"
},
{
"path": "/nix/store/jhflvwr40xbb0xr6jx4311icp9cym1fp-python3-3.10.12"
},
{
"path": "/nix/store/vczvk2zwf4v6fg3v332qsvavpmsfjpc7-python3.10-pycparser-2.21"
},
{
"path": "/nix/store/ricjxi3qg9hgk3p1x4qm0gr68nimj9cm-python3.10-pysocks-1.7.1"
},
{
"path": "/nix/store/ddjn9h5fyhc1b07b32y4mxvf8p4vfc63-python3.10-brotli-1.0.9"
},
{
"path": "/nix/store/9a4a95d0bna4zq7jk0sjbwih5k1z2igd-python3.10-cffi-1.15.1"
},
{
"path": "/nix/store/j2l0z6m40jncm80i33nk0aw2zs2vp5ss-brotli-1.0.9-lib"
},
{
"path": "/nix/store/zgs6qcxngpa7jcv766yn08g9bwnz29dh-nss-cacert-3.90"
},
{
"path": "/nix/store/ib2a54mc11gal5mra851w8bvbidaiar4-python3.10-py-1.11.0"
},
{
"path": "/nix/store/w5fv5fabcayxzd219mp6msgd3pzn93xl-python3.10-markupsafe-2.1.3"
},
{
"path": "/nix/store/r9g98kkxx9909678vnsj7kj5dz58clhf-python3.10-urllib3-1.26.14"
},
{
"path": "/nix/store/j19v9wiwp094gz0q3rf89vyvia5yl1rr-python3.10-idna-3.4"
},
{
"path": "/nix/store/bkk9dwn22mnnv79hqz1vn42jydhmp9dv-python3.10-charset-normalizer-3.0.1"
},
{
"path": "/nix/store/9sgbb4kiwdazjmrnnqxkshc2j82gbs2k-python3.10-brotlicffi-1.0.9.2"
},
{
"path": "/nix/store/4x5pywdrb17mjhpm54xmdjz5dqllp96r-python3.10-babel-2.12.1"
},
{
"path": "/nix/store/2ry2jni7wly794ivzp7jh0pxbynw3lj5-python3.10-certifi-2023.05.07"
},
{
"path": "/nix/store/wzr7vmr65fns7f0c5vg4cz9hmbd35lp7-libsodium-1.0.18"
},
{
"path": "/nix/store/ip6fhd8yn2w79ls7qdxm7dwj38r9nlnv-zeromq-4.3.4"
},
{
"path": "/nix/store/fw3ih3j78n0pil093p57bsc6dhvd9vxj-libyaml-0.2.5"
},
{
"path": "/nix/store/yqsqmykd6k9xqxx3f5y4qc8viwg33wxy-python3.10-pyzmq-24.0.1"
},
{
"path": "/nix/store/xal21vd4d9nfwjkcvw0fyq6ivsbxg1pz-openssl-3.0.9"
},
{
"path": "/nix/store/vxd7chj69jkgggynv81rn327xccmdar3-python3.10-psutil-5.9.5"
},
{
"path": "/nix/store/nn3h0bl8h1p604v9h2d950nqp0jr2s9a-python3.10-msgpack-1.0.5"
},
{
"path": "/nix/store/k8kplbjyc8cnngpg4ib6nkl16v3mqfhw-python3.10-pycryptodome-3.17.0"
},
{
"path": "/nix/store/isvrjf7mxi1y2hphh18zg6wyhyf8685f-python3.10-distro-1.8.0"
},
{
"path": "/nix/store/h4w0cfacw1sidw564gkkc51xcwb4a2h2-python3.10-looseversion-1.3.0"
},
{
"path": "/nix/store/g5p3zln7bk1gcq5p17mlywqrkrk5lj58-python3.10-jmespath-1.0.1"
},
{
"path": "/nix/store/g4b3sss35jbybsb5hnpdbic8wb9hy0ri-python3.10-packaging-23.0"
},
{
"path": "/nix/store/d66wysv5w90klmvknfyp6h1mpl374xd5-python3.10-Jinja2-3.1.2"
},
{
"path": "/nix/store/ci6sxn8vfm9sw4wd6kqpyw9hq59gf8kn-python3.10-pyyaml-6.0"
},
{
"path": "/nix/store/9xnrc5n8z6sjzc98azz5cd19i6wm92m1-python3.10-requests-2.31.0"
},
{
"path": "/nix/store/m5c8f99vpqdn3gyv7vllql9ggi1vpw87-salt-3006.1"
},
{
"path": "/nix/store/ifw35xqmx8fi2kiy0ydg2p6lxkcfmxn0-root",
"options": {
"rewrite": {
"regex": "^/nix/store/ifw35xqmx8fi2kiy0ydg2p6lxkcfmxn0-root",
"repl": ""
}
}
}
],
"mediatype": "application/vnd.oci.image.layer.v1.tar"
}
],
"arch": "amd64"
}
So my question is how can I modify the above flake to get:
- first layer: just the python and its dependencies
- second layer: salt package's dependencies (excluding python)
- third layer just the salt (and no dependencies)
Also, I think more clarification is needed in the README.md, the nested layers are not explained well, I was actually confused what they are for until I found this ticket.
I'm also wondering if instead of having to use buildLayer function to create a layer, it wouldn't be simpler for the user to have list of attribute sets. For example:
layers = [
{
deps = [ python ];
}
{
deps = package.propagatedBuildInputs;
}
{
copyToRoot = pkgs.buildEnv {
name = "root";
paths = [ pkgs.bashInteractive pkgs.coreutils ];
pathsToLink = [ "/bin" ];
};
}
Then buildImage would evaluate each and be able to remove duplicates without additional hacking like nested layers.
Anyway, great tool, and I hope it will replace existing docker build in nixpkgs. For now I'll just combine python and deps into single layer, as that seem to work, but I would love to be able to eventually do 3 layers (python, project deps, project).
@nlewo is there anything I could provide to help identify the problem? Or if I'm using things incorrectly, how the call should be modified to get expected results?
I think this feature should be implemented as part of nix2container. I don't believe that there is much of a point in hacking solution on top of it due to complexity reasons
@takeda I have been able to reproduce and i confirm there is an issue. I however didn't find time yet to dig more (i should be able to in the next 2 weeks). Thx for your example, that's really useful!
@takeda Your expression is producing an empty layer because the layer containing the propagetedBuildInputs also contains python3. So, when nix2container builds the python3 layer, it doesn't include any storepaths because they are all part of the dependency layer. This leads to an empty python3 layer and nix2container fails to remove already existing storepaths when there is an empty layer on the path.
In your usecase, i don't think you want this empty layer so you should try to remove it.
nix2container should either support empty layers or explicitly fails on empty layers. I don't know yet what to do...
Sorry, but I'm confused I assumed that buildLayer would automatically take care of duplicates, perhaps I'm doing them in reverse?
How can I correctly do to achieve what I want, basically:
- first layer contains just python and its dependencies
- second layer would contain all application dependencies (excluding python)
- third layer would just contain my application
I tried reverse, i.e.:
layers = [
(n2c.buildLayer {
deps = package.propagatedBuildInputs;
layers = [
(n2c.buildLayer {
deps = [ python ];
})
];
})
];
This results in just two layers:
- first (base layer) are the dependencies except python (that should be the 2nd layer)
- second layer seems to contain python and its dependencies + application (salt)
So that still doesn't look right.
After thinking about what I got in previous comment I tried something once more:
layers = [
(n2c.buildLayer {
deps = [ python ];
})
(n2c.buildLayer {
deps = package.propagatedBuildInputs;
layers = [
(n2c.buildLayer {
deps = [ python ];
})
];
})
];
I think this works, I do see there are 2 openssl packages with different hashes, but have a suspicion that it is problem with the salt package.
So it looks like I'm getting:
- first layer (base) python and its dependencies
- second layer application dependencies excluding stuff from the first layer
- third layer the application + openssl (but I suspect dangling openssl is artifact of salt)
So I think the layers feature, while very powerful it is probably the most difficult to get feature of containers2nix I think more documentation how it works is needed. I also think that since the layers parameter inside of buildLayer work more like exclude operation, so perhaps it should be named exclude to be less confusing?
But this still can get complex, especially when somebody needs more layers. One has to be very cautious to exclude everything that's in previous layers.
I think what I suggested in previous comment would greatly simplify the work and make it much more intuitive e.g.:
layers = [
{
deps = [ python ];
}
{
deps = package.propagatedBuildInputs;
}
{
copyToRoot = pkgs.buildEnv {
name = "root";
paths = [ pkgs.bashInteractive pkgs.coreutils ];
pathsToLink = [ "/bin" ];
};
}
The code then would go, and:
- create layer with
python - create next layer with
package.propagatedBuildInputsbut excluding everything from the previous layer - create next layer but excluding everything from two previous layers
- and so on.
I believe that would be much simpler for the user and much more robust. What do you think?
I think I got hang of it, and probably the best way is to use it with let.
This is how I'm using it in my project and looks like it works as expected and dive shows 100% efficiency:
# organize the container into 3 layers:
# - base layer with python and busybox
# - layer with application dependencies
# - our application
layers = with nix2container; let
layer-1 = buildLayer {
deps = [
pkgs.busybox
config.out_lean_python
];
};
layer-2 = buildLayer {
deps = config.out_python.propagatedBuildInputs;
layers = [
layer-1
];
};
in [
layer-1
layer-2
];
I believe if I needed to add layer-3, then inside of its layers section I would list layer-1 and layer-2 unless I'm certain layer-3 doesn't have any overlap with prior layers.
I stumbled upon this problem myself and wrote a little blog post about my fix:
https://blog.eigenvalue.net/2023-nix2container-everything-once/
Maybe it is useful to you or other people discovering this issue!
@kolloch Thank you very much for this post!
(I think it could be linked in this nix2container readme section)
I independently came to the same solution as @takeda, though it would be great if this could be handled more automatically by nix2container— one option could be shipping and recommending the layer-folding function that @kolloch suggested, another could be having some kind of post-processing filter step applied at- or after JSON generation that automatically removes dupes (assuming stable layer order).
Another thing that would be nice to have is a way of defining a layer based on the intersection of two other layers. This could dovetail with auto-deduping, where you basically have a layer that's Django and everything below it, a separate layer that's Flask and everything below it, and then a Python-Common layer that's everything in both of those. Then no matter whether you're making a container that has Flask or Django or both, you only ever have those three layers, and the common stuff is shared— but all that only really works with automatic deduping (and it isn't really expressable at all today, short of manually discovering and enumerating the common stuff).
I also proposed an idea for making a smarter automatic layering system able to detect these types of relationships between multiple containers over time, please see https://github.com/nlewo/nix2container/issues/113 if you're interested to weigh in on that.
@mikepurvis you might want to look into this PR in nixpkgs. The example layer algo\pipeline linked from PR description works perfectly for python or nodejs applications and doesn’t suffer from any duplication in layers. This could possibly be ported to here.
