Mic92
Feature request: update system-dependent hashes
I maintain a package that uses a fixed-output derivation submodule which stores platform-specific hashes as an attribute set in outputHash (example: https://github.com/NixOS/nixpkgs/blob/029b4824cbfc2007f1c5834fc717d4ff3b1d6636/pkgs/by-name/op/opencode/package.nix#L106-L113).
Currently, to update a hash I have to run nix-build <package>.<submodule> --system <platform> and then replace the hashes manually. I saw that many other packages with system-dependent hashes provide a custom update script.
Wouldn't it be possible to detect when the output hash is an attribute set. If so, read the attribute keys (the systems) and update the hashes for all defined systems automatically.
Questions / notes:
- I looked through the code but didn’t find an obvious way to add this. Any pointers on how you’d implement it?
- Is this a niche problem, or something others encounter often?
Thanks for considering — happy to help prototype if you point me to the right place in the code.
We sort of already support this:
https://github.com/NixOS/nixpkgs/blob/2b78ecc45e163d655c08c9a4cffb4a91c66d0493/pkgs/applications/networking/instant-messengers/ferdium/update.sh#L11
An extension to this might be to have a --systems aarch64-linux,x86_64-linux flag, but the semantics are still a bit unclear to me.
I saw that flag. However, when I update with that flag none of the output hashes in my submodule get updated. (As an example, I'm using nix-update -s tui -s node_modules --system aarch64-darwin opencode to update the opencode package. In this case the node_modules submodule contains system specific hashes.)
outputHash =
{
x86_64-linux = "";
aarch64-linux = "";
x86_64-darwin = "";
aarch64-darwin = "";
}
So, I'm not sure how the --system flag is supposed to work. I expected it to update the system-specific output hash for the selected system.
I encountered the same issue building PR https://github.com/NixOS/nixpkgs/pull/449660. Before working on that PR I looked into the system flag (from https://github.com/Mic92/nix-update/pull/130) but either it doesn't work for me or I'm holding it wrong.
Using a straightforward update script like below attempted to execute bash-aarch64 on my buildhost-amd64 (localhost). I have a builder for aarch64 configured in my system nix config, but it didn't use that configuration.
passthru.updateScript = nix-update-script {
extraArgs = ["--system" "aarch64-linux"];
};
click for run/error log
[bert-proesmans@development:~/nixpkgs]$ nix-shell maintainers/scripts/update.nix --argstr package netbootxyz-efi
Going to be running update for following packages:
- netboot.xyz-efi-2.0.86
Press Enter key to continue...
Running update for:
Enqueuing group of 1 packages
- netboot.xyz-efi-2.0.86: UPDATING ...
- netboot.xyz-efi-2.0.86: ERROR
--- SHOWING ERROR LOG FOR netboot.xyz-efi-2.0.86 ----------------------
Traceback (most recent call last):
File "/nix/store/ydl85jvifpzxnx5apca6y2f1yvr5x3b0-nix-update-1.13.1/bin/.nix-update-wrapped", line 9, in <module>
sys.exit(main())
~~~~^^
File "/nix/store/ydl85jvifpzxnx5apca6y2f1yvr5x3b0-nix-update-1.13.1/lib/python3.13/site-packages/nix_update/__init__.py", line 469, in main
package = update(options)
File "/nix/store/ydl85jvifpzxnx5apca6y2f1yvr5x3b0-nix-update-1.13.1/lib/python3.13/site-packages/nix_update/update.py", line 218, in update
update_src_hash(opts, package.filename, package.hash)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ydl85jvifpzxnx5apca6y2f1yvr5x3b0-nix-update-1.13.1/lib/python3.13/site-packages/nix_update/dependency_hashes.py", line 97, in update_hash_with_prefetch
target_hash = nix_prefetch(opts, attr_name)
File "/nix/store/ydl85jvifpzxnx5apca6y2f1yvr5x3b0-nix-update-1.13.1/lib/python3.13/site-packages/nix_update/dependency_hashes.py", line 86, in nix_prefetch
raise UpdateError(msg)
nix_update.errors.UpdateError: failed to retrieve hash when trying to update netbootxyz-efi.src
--- nix stderr (last 20 lines) ---
warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
this derivation will be built:
/nix/store/xj83fxsscvcxc1x58iz63fdyl1fg62q4-netboot.xyz.efi.drv
building '/nix/store/xj83fxsscvcxc1x58iz63fdyl1fg62q4-netboot.xyz.efi.drv'...
error: executing '/nix/store/pq80mryqppk9a2z0i36hg112pbvfjr2p-bash-5.3p3/bin/bash': Exec format error
error: Cannot build '/nix/store/xj83fxsscvcxc1x58iz63fdyl1fg62q4-netboot.xyz.efi.drv'.
Reason: builder failed with exit code 1.
Output paths:
/nix/store/iay4pl2acj9v5lr611bc39q2s4gz2z6g-netboot.xyz.efi
Last 1 log lines:
> error: executing '/nix/store/pq80mryqppk9a2z0i36hg112pbvfjr2p-bash-5.3p3/bin/bash': Exec format error
For full logs, run:
nix log /nix/store/xj83fxsscvcxc1x58iz63fdyl1fg62q4-netboot.xyz.efi.drv
--- SHOWING ERROR LOG FOR netboot.xyz-efi-2.0.86 ----------------------
The update script for netboot.xyz-efi-2.0.86 failed with exit code 1
[bert-proesmans@development:~/nixpkgs]$ file /nix/store/pq80mryqppk9a2z0i36hg112pbvfjr2p-bash-5.3p3/bin/bash
/nix/store/pq80mryqppk9a2z0i36hg112pbvfjr2p-bash-5.3p3/bin/bash: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/cnw00vg7w28qsn4jc6vxbwlignc35w4n-glibc-2.40-66/lib/ld-linux-aarch64.so.1, BuildID[sha1]=3273ee0a5620a3015616dfe4ff3df3c401e564c8, for GNU/Linux 3.10.0, not stripped
I naively expected the following derivation to "just update" as following;
- Fixes version number to latest on first run, ignores version match on second run
- Fixes hash for x64 on first run
- Fixes hash for aarch64 on second run
{
lib,
stdenv,
fetchurl,
_experimental-update-script-combinators,
nix-update-script,
}:
let
version = "2.0.88";
sourceMap = {
aarch64-linux = fetchurl {
url = "https://github.com/netbootxyz/netboot.xyz/releases/download/${version}/netboot.xyz-arm64.efi";
sha256 = "064iz2xwghkdqjvq6a7hpssnzcngw87ziqk3514pbr9sapcbvr81";
};
x86_64-linux = fetchurl {
url = "https://github.com/netbootxyz/netboot.xyz/releases/download/${version}/netboot.xyz.efi";
sha256 = "1c8vfq0241k1sh4n0a5cqhwrcfq5sisvbhzldwryq2lg94kxk5la";
};
};
in
stdenv.mkDerivation (finalAttrs: {
pname = "netboot.xyz-efi";
inherit version;
src =
sourceMap.${stdenv.hostPlatform.system}
or (throw "Unsupported system: ${stdenv.hostPlatform.system}");
# ...
# THE MAGIX HERE ---
passthru.updateScript = _experimental-update-script-combinators.sequence [
(nix-update-script { })
(nix-update-script { extraArgs = ["--system" "aarch64-linux"]})
];
# ---
meta = {
homepage = "https://netboot.xyz/";
# ...
};
})
error: executing '/nix/store/pq80mryqppk9a2z0i36hg112pbvfjr2p-bash-5.3p3/bin/bash': Exec format error
I think we should nix-update setting --eval-system instead of --system in nix. Than remote builders are used correctly.
I think we should nix-update setting --eval-system instead of --system in nix. Than remote builders are used correctly.
I'm trying to piece together what your response means.
When I run the linked Ferdium update script I get the same behaviour (attempt to run bash-aarch64) as reported in my previous post. This is an error because my localhost system explicitly does not emulate other platforms than amd64.
My naïve understanding of nix-update is that it uses nix evaluation mode to find the necessary data (hashes, version literals) to update a package. Is realising/running binaries from the system under evaluation (aarch64 in this case) a requisite for the evaluation mode to complete?
This thread is specifically about hash-updates for blobs with binary native code, which is a niche in updating nix package recipes. nix-update doesn't do this automatically so there is now manual scripting in all cases, with the ferdium package you linked as an example. I always approached this type of updates as "record hashes and push" without effort to test other platforms than amd64.
Is there a principle that I must always have access to all the supported platforms while updating nix packages? With obvious consequence that nix-update just works in that context.
Is there consideration for downstream effects? For example bot r-ryantm doesn't work for packages opted into nix-update (hypothetically) evaluating for multiple architectures, or drive-by contributors likely have no access to cross-emulation or remote builders for every (package supported) platform.
This reply is not a judgement, I'm trying to understand and try to be better next time. It's probably not immediately possible to have nix-update make use of pure evaluation mode and/or restricted to build-platform binaries. On the other hand how rigorous should updating hashes for referenced binary blobs be?
I think we should nix-update setting --eval-system instead of --system in nix. Than remote builders are used correctly.
I'm trying to piece together what your response means.
When I run the linked Ferdium update script I get the same behaviour (attempt to run bash-aarch64) as reported in my previous post. This is an error because my localhost system explicitly does not emulate other platforms than amd64.
My naïve understanding of nix-update is that it uses nix evaluation mode to find the necessary data (hashes, version literals) to update a package. Is realising/running binaries from the system under evaluation (aarch64 in this case) a requisite for the evaluation mode to complete?
This thread is specifically about hash-updates for blobs with binary native code, which is a niche in updating nix package recipes. nix-update doesn't do this automatically so there is now manual scripting in all cases, with the ferdium package you linked as an example. I always approached this type of updates as "record hashes and push" without effort to test other platforms than amd64.
Even FOD hashes require to build the FOD with a platform curl, so this doesn't work for anything but builtin fixed-output derivation fetcher.
Is there a principle that I must always have access to all the supported platforms while updating nix packages? With obvious consequence that nix-update just works in that context.
For this feature to work, you would need to have access to all these architectures yes.
Is there consideration for downstream effects? For example bot r-ryantm doesn't work for packages opted into nix-update (hypothetically) evaluating for multiple architectures, or drive-by contributors likely have no access to cross-emulation or remote builders for every (package supported) platform.
r-ryantm could potentially get access to other platforms. For contributors, github actions could potentially provide this.
This reply is not a judgement, I'm trying to understand and try to be better next time. It's probably not immediately possible to have nix-update make use of pure evaluation mode and/or restricted to build-platform binaries. On the other hand how rigorous should updating hashes for referenced binary blobs be?
Sometimes we can take advantage of cross-compilation. For example, use nix-update pkgsCross.aarch64-multiplatform.netbootxyz-efi
