`nix copy` is showing unreasonably huge amounts of data copied

[oddlama]

Describe the bug

When using nix copy to copy packages between hosts, I am seeing unreasonably huge amounts of data, 3-4 orders of magnitude more than what is actually being copied:

(Shows ~7TB while actual data copied is around 600M)

This looks like dependencies are more than one time, or something else that amplifies the number. I'm sorry if this a desired behavior, but I wasn't able to find a related issue or documentation confirming this.

Steps To Reproduce

1. Use nix copy to copy a system toplevel to another host
2. See huge numbers.

Expected behavior

The numbers should reflect actual bytes transferred or at least actual bytes processed. It should definitely be bounded by the closure size.

nix-env (Nix) 2.17.0

Priorities

Add :+1: to issues you find important.

[arduano]

I just encountered this as well. I believe it's showing the wrong unit, listing kilobytes instead of megabytes.

A simple fix would probably be to divide by 1024 when logging.

[oddlama]

Maybe, but I'm not 100% sure it's that simple. It frequently feels off by far more than a factor of 1k. More like if it was a quadratic accumulation of error - it definitely grows faster the more it already shows.

But it's hard to find evidence for any of this, since I cant exactly tell how much of a closure was really uploaded and how much was already present on the target.

[arduano]

I'm not too sure, but I found that the number approached a similar amount to how many gigabytes of storage I was copying. I'm doing a lot of copying for testing some architecture at the moment.

[thufschmitt]

I suspect that some places get this wrong, others get it right, making the whole thing quite hard to track.

I tried to make sense of what was going on at some point in the past, but didn't manage to. Anyone with a bit more willpower than me is welcome to give it a try, it's probably just a matter of following the trail deeply enough to go back to the root

[nrdxp]

I assume that nix copy doesn't check the destination for already existing paths and just copies the entire closure indiscriminantly. It is rather annoying, especially when you forget to pass --no-check-sigs during a lengthly copy and have to do it all over again :unamused:

[Ericson2314]

I've noticed that ssh and ssh-ng work differently, and differently across versions. It seems like we have an ever-changing mess of attempts at getting this right.

[nrdxp]

At the very least perhaps paths could be saved in some temporary location for a brief period of time, so that canceled or failed copies (because of the aforementioned sig check failure) don't require a full restart. Just to be clear I was experiencing this behavior with ssh-ng. I didn't think to try the older protocol at the time, but next time I'll see if that's any better.

[Nomkid]

the current amount transferred seems to always exceed the total amount to transfer, from all the examples i have seen. here's a couple more: