Option in nix-build, to force rebuild, even if package exists in store

[avnik]

I know about --check, but it require root, and overwrite path in store (if I understand right). Would be nice, if we will have --test-build or something named likewise, to do rebuilds in separate temporary paths.

It would be useful for bisecting nixpkgs. And also can be reused later, to compare results in temporary with matched store paths, for finding impurities.

[copumpkin]

An alternative could just be teaching the daemon to understand --check

[shlevy]

Because --check is not completely atomic, we can't let arbitrary users run it.

[copumpkin]

Could it be made atomic? I guess it could get ugly for self-referential outputs

On Thursday, March 5, 2015, Shea Levy [email protected] wrote:

Because --check is not completely atomic, we can't let arbitrary users run it.

— Reply to this email directly or view it on GitHub https://github.com/NixOS/nix/issues/493#issuecomment-77377978.

[shlevy]

Would require the intensional model to be atomic. Otherwise we want to replace the existing path with the newly built one, and in the case where the existing path is a directory (that is, almost always), we have to move the old directory away first then move the new directory into place.

[shlevy]

(of course, Linux's new renameat2 flag RENAME_EXCHANGE would help where supported)

[copumpkin]

Well, I wouldn't expect check to overwrite the existing output. In a world with chroot and bind mounts, it seems doable, right? Convince the package to build against an output that's a bind mount with the same path as the actual $out, check it produces the same thing?

On Thursday, March 5, 2015, Shea Levy [email protected] wrote:

Would require the intensional model to be atomic. Otherwise we want to replace the existing path with the newly built one, and in the case where the existing path is a directory (that is, almost always), we have to move the old directory away first then move the new directory into place.

— Reply to this email directly or view it on GitHub https://github.com/NixOS/nix/issues/493#issuecomment-77378644.

[shlevy]

And if it doesn't produce the same thing, what should we do? We don't have bit-perfect builds in many cases. If you're --checking, the presumption is that you think something might have gone wrong with the current valid path.

[copumpkin]

Ah, I thought it existed purely for the sake of checking bit-perfectness (so it could tell us we suck!)

[copumpkin]

If you want to repair the current store path, it seems like it shouldn't be called check

[shlevy]

Oops, I've been talking about --repair this whole time :grin: Ignore me, --check does exactly what you're saying. 1aa19b24b27c6bbf4d46cdd7f6d06b534dd67c19

[vcunat]

:-) I also fail to see why --check should need root privileges, in principle, being "read-only".

[copumpkin]

I don't think it does. It just doesn't work with nix daemon and is undocumented

[vcunat]

By "need root privileges" I meant "not being supported by nix daemon", as I assumed the setting when nix store isn't directly writable by the user.

[atondwal]

Found this issue when I was looking for this exact option, and --check does the trick for me. close?

[nh2]

I'm trying to force rebuilding on the following example package:

% cat mypackage.nix
with (import <nixpkgs> {});

derivation {
  name = "simple";
  builder = "${pkgs.bash}/bin/bash";
  args = [
    "-c"
    "echo hello > $out"
  ];
  src = ./.;
  system = builtins.currentSystem;
}

I get an error when using --check:

% nix-build mypackage.nix --check
these derivations will be built:
  /nix/store/dxlm486347zy1v9jprjwvz0ncg58vdy2-simple.drv
error: some outputs of ‘/nix/store/dxlm486347zy1v9jprjwvz0ncg58vdy2-simple.drv’ are not valid, so checking is not possible

Without --check, it works fine. Why does --check fail here, or how otherwise could I force a rebuild?

[Gabriella439]

@nh2: That error message means that at least one output of your derivation has not been built before. The --check flag only works with a derivation that has already been fully built because it needs a prior build to compare against

[nh2]

The --check flag only works with a derivation that has already been fully built because it needs a prior build to compare against

Ah, I misdunerstood that then. I thought --check would build it first if it doesn't exist yet, and then build again to compare.

[tg-x]

check does not seem to work here:

# nix-build --check -A mpir
...
make[1]: Leaving directory '/build/mpir-3.0.0'                                                                                                                                                                                                                                                                                
post-installation fixup                                                                                                                                                                                                                                                                                                       
shrinking RPATHs of ELF executables and libraries in /nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0
shrinking /nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0/lib/libmpir.so.23.0.3
shrinking /nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0/lib/libmpirxx.so.8.4.3
strip is /nix/store/0y7jmqnj48ikjh37n3dl9kqw9hnn68nq-binutils-2.31.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0/lib
patching script interpreter paths in /nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0
checking for references to /build/ in /nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0...
error: derivation '/nix/store/1vx59hlxzvigdb72zs8p6n6ybgkdxlni-mpir-3.0.0.drv' may not be deterministic: output '/nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0' differs

and then the store path /nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0 is not created/updated after this finishes.

also, the man page of nix-build is missing the --check option and the documented --repair option does not seem to work either:

# nix-build --repair -A mpir
error: repairing is not supported when building through the Nix daemon

[7c6f434c]

Hm, and it looks like --repair wants to operate on entire store and will consider non-corrupted paths OK, and --repair-path wants to redownload and not rebuild? Is there a way to build locally (like --check) but actually replace the path if it targets wrong subarchitecture without deleting all the rev-deps?

[tg-x]

yes, repair-path does not rebuild, only works from binary cache apparently:

# nix-store --option binary-caches '' --repair-path /nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0/ --debug
substitution of '/nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0': created
substitution of '/nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0': woken up
entered goal loop
substitution of '/nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0': init
acquiring global GC lock '/nix/var/nix/gc.lock'
acquiring read lock on '/nix/var/nix/temproots/15438'
acquiring write lock on '/nix/var/nix/temproots/15438'
downgrading to read lock on '/nix/var/nix/temproots/15438'
substitution of '/nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0': trying next substituter
path '/nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0' is required, but there is no substituter that can build it
substitution of '/nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0': done
substitution of '/nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0': goal destroyed
error: cannot repair path '/nix/store/nil03idsbz7gk4xpn424yy29v2mp0ma1-mpir-3.0.0'

[stale[bot]]

I marked this as stale due to inactivity. → More info

[NorfairKing]

Still relevant.

[bhankas]

Still relevant! :)

[ProofOfKeags]

Still relevant

[stale[bot]]

I marked this as stale due to inactivity. → More info

[NorfairKing]

I'm going to have to write a not-stale bot for the issues that I'm involved in.

[piegamesde]

How about introducing a never-stale label instead? (Also, I actually do not mind having a bot bump this every year or so. Maybe some exponential backoff would help though?)

[AleXoundOS]

When there is a project which internally describes several derivations, nix-build --check is not enough.

For example, if I build embedded device firmware with nix, each mtd partition of it being an individual nix derivation, nix-build --check rebuilds only the final derivation (that combines partitions into the firmware image file). But it's not clear how to rebuild/check all derivations introduced by my project on which the output firmware image depends on (manual recursion with nix-tree is still tedious and error-prone).

Perhaps, assurance begins from binary cache hashes and signatures, so there is no point in digging deeper, rebuilding and checking the "world" down to some trusted nix bootstrap tooling. So, it could be useful to have an option to rebuild and check everything that's not in a binary cache (or not hashed/signed by other means if any).

And it's a real case where a check for being bit-perfect makes sense - embedded device firmware. I've actually encountered this.

[imuli]

Another use case! Someone on my team had sandbox = false and picked up some host libraries in a node package that wrapped some native libraries. It'd be sort of nice to just rebuild the world instead of re-build+download the wold. :)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-does-nix-calculate-the-hash-of-a-derivation/24360/1