hard links to symlink in ~/.cargo/bin installed by rustup-init in macOS Big Sur break SuperDuper! backup

[sidney]

rustup-init installed by Homebrew on macOS Big Sur 11.6, when run creates a number of files in ~/.cargo/bin which are hard links to a symlink. This breaks backing up using SuperDuper! with its Smart Update option, which does not support files that are hard links to symlinks. Also, Posix specification of link says "if path1 names a symbolic link, it is implementation-defined whether link() follows the symbolic link, or creates a new link to the symbolic link itself." so the action is not well defined.

Steps

1. On macOS Big Sur, with homebrew installed, use brew install rustup-init
2. Run the command rustup-init, accepting the default installation options
3. Run the command ls -il ~/.cargo/bin/

Expected result: The 12 files in ~/.cargo/bin should be symlinks all pointing to /usr/local/bin/rustup-init with the first column (inode) all being different, and the third column (reference count of the inode) all being 1.

Actual result: The 12 files all have the same inode number and all show it having a reference count of 12.

Also, trying to make a backup of the disk using the Smart Update option of SuperDuper! 3.5-beta3 (the beta is required for compatibility with Big Sur, but the lack of support for hard links to symlinks is not a beta thing) results in an error when it tries to copy one of the link files in ~/.cargo/bin.

Possible Solution(s) Whatever method rustup-init uses to create the links in ~/.cargo/bin to /usr/local/bin/rustup-init, it should create individual symlinks instead of what it is now doing that creates hard links to the same symlink inode.

Deleting the links and recreating them myself using ln -s commands worked around my problem with SuperDuper!.

Notes

Output of ls -il ~/.cargo/bin/ showing same inode 43077450 and reference count 12

total 0
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 cargo -> /usr/local/bin/rustup-init
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 cargo-clippy -> /usr/local/bin/rustup-init
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 cargo-fmt -> /usr/local/bin/rustup-init
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 cargo-miri -> /usr/local/bin/rustup-init
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 clippy-driver -> /usr/local/bin/rustup-init
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 rls -> /usr/local/bin/rustup-init
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 rust-gdb -> /usr/local/bin/rustup-init
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 rust-lldb -> /usr/local/bin/rustup-init
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 rustc -> /usr/local/bin/rustup-init
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 rustdoc -> /usr/local/bin/rustup-init
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 rustfmt -> /usr/local/bin/rustup-init
43077450 lrwxr-xr-x  12 sidney  staff  26  3 Oct 11:38 rustup -> /usr/local/bin/rustup-init

Output of rustup --version:

rustup 1.24.3 (2021-05-31)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.55.0 (c8dfcfe04 2021-09-06)`

Output of rustup show:

Default host: x86_64-apple-darwin
rustup home:  /Users/sidney/.rustup

stable-x86_64-apple-darwin (default)
rustc 1.55.0 (c8dfcfe04 2021-09-06)

[kinnison]

This is fascinating, We build the hard links by linking to rustup in that directory, so the question comes, how did that come to be a symlink instead of a copy of rustup ?

[sidney]

That' s a hint. Homebrew installs into a versioned directory, such as /usr/local/Cellar/rustup-init/1.24.3/bin/rustup-init and creates a symlink /usr/local/bin/rustup-init pointing to it. That's how all programs are installed in Homebrew so that /usr/local/bin can be in PATH and version upgrades can be handled seamlessly.

If I run rustup-init so it runs from /usr/local/bin/ it looks like it creates ~/.cargo/bin/rustup as a symlink to /usr/local/bin/rustup-init and then the other 11 files are hard links to~/.cargo/bin/rustup

If instead of the command rustup-init I explicitly run /usr/local/Cellar/rustup-init/1.24.3/bin/rustup-init then ~/.cargo/bin/rustup ends up with as a copy of /usr/local/Cellar/rustup-init/1.24.3/bin/rustup-init and the other files in ~/.cargo/bin/ are hard links to ~/.cargo/bin/rustup

However you get it to work, in a Homebrew installation the files in ~/.cargo/bin should end up as symlinks to /usr/local/bin/rustup-init not the hard-coded versioned file or else it will break when there is a version upgrade. And they can't be hard links to a symlink in ~/.cargo/bin/rustup because of the problems with hard links to symlinks.

[stouset]

This has bitten me as well.

The "obvious" fix is to change the logic to link to rustup with symlinks rather than hardlinks, since symlinks-to-symlinks have well-defined semantics (unlike hardlinks-to-symlinks). Do we know the reason that rustup's authors chose to use hardlinks here?

[stouset]

Reading back through the history, it looks like different operating systems have different requirements for linking. For Windows, you need Administrator privileges to symlink. Other operating systems like Android don't support hard linking. So it looks like rustup defaults to trying to hardlink, and falls back to symlinking if that fails.

It might be possible to simply switch the order of attempts? Hardlinks seem to have more caveats around them in general from my own experience.

[kinnison]

Homebrew installing rustup-init seems very confusing to me - is it providing a no-self-update binary? if so realistically it should be providing rustup and the proxy set rustc, cargo etc. NOT rustup-init.

[stouset]

That's correct, it's a no-self-update binary.

❯ rustup self update
error: self-update is disabled for this build of rustup
error: you should probably use your system package manager to update rustup

[kinnison]

Right, so realistically if homebrew is expecting to be in control of updating rustup then instead of providing rustup-init to users, it should be providing rustup and all the proxies in the path instead.

[stouset]

Well, or it should just provide the rustup-init parts and install rustup normally, allowing self-updating.

[stouset]

Looks like the bug is still in rustup here. Homebrew just drops the upstream rustup-init binary into the user's PATH and expects the user to invoke it.

Ideally rustup would create a copy of itself in ~/.cargo the way that it works when curl-piping and symlink everything into there.

[kinnison]

OK, so let's see if I'm understanding the context fully.

1. Homebrew is providing rustup-init which it expects the user to run
2. It does this via some set of symbolic links, rather than a direct executable on the PATH
3. This results in rustup-init making $CARGO_HOME/bin/rustup link to that link
4. Rustup then proceeds to make hard-links to that symlink.

This appears to stem from a fix made to rustup where utils::copy_file() will do the symlinking if appropriate so that lldb-preview doesn't break. But I'm unconvinced it's doing it right anyway:

https://github.com/rust-lang/rustup/blob/master/src/utils/utils.rs#L343-L361

In theory if we wanted to preserve symbolic links in a copy, we should be copying the symlink's destination, not the source location of the copy. Regardless, for installing rustup we shouldn't really be making that symbolic link.

As such it's probably https://github.com/rust-lang/rustup/blob/master/src/cli/self_update.rs#L683 which is needing a change. We could perhaps offer a variant of utils::copy_file() which resolves symlinks instead.

[hube12]

Hello, this is really a fascinating issue, because before I tried to override the cargo component today (and thus since all the binary share the same inodes, override all the other binaries in .cargo/bin) I never noticed that all of those have a symbolic link count of 12.

Btw this issues arise on any linux distribution:

debian@DESKTOP-SA4B8HR:~/.cargo/bin$ ls -lahi
total 191M
 35161 drwxr-xr-x  2 debian debian 4.0K May  5 00:01 .
 35160 drwxr-xr-x  5 debian debian 4.0K Jan 19 00:31 ..
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 cargo
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 cargo-clippy
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 cargo-fmt
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 cargo-miri
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 clippy-driver
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 rls
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 rust-gdb
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 rust-lldb
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 rustc
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 rustdoc
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 rustfmt
 33609 -rwxr-xr-x 12 debian debian  15M May  4 23:54 rustup

This is mind boggling how this is something that went under the radar for so long.

[kinnison]

@hube12 This is quite deliberate and not at all unexpected. When you go manually fiddling with stuff installed by rustup you get what you pay for as it were. Any issues you're experiencing on Linux are unrelated to this issue which is about 'SuperDuper! backup' breaking because homebrew is not providing rustup properly to users coupled with an "optimisation/correction" in our copy code which isn't quite right.

I really would like someone from homebrew to chime in here, because if we can fix this by homebrew properly providing rustup rather than rustup-init then we can fix this in the way package managers are meant to work.

[stouset]

Frankly I think that's misstating the situation a little bit. Nothing is "manually fiddling with stuff installed by rustup".

The short form of this is that if you invoke rustup-init through a symlink, rustup-init will create hardlinks to that symlink. Hardlinks to symlinks have poorly-defined semantics which differ from operating system to operating system. No tool should be creating hardlinks to symlinks.

[stouset]

Here's an example:

❯ mkdir tmp; cd tmp

~/tmp
❯ curl -sL https://github.com/rust-lang/rustup/archive/refs/tags/1.24.3.tar.gz | tar -xz

~/tmp
❯ cd rustup-1.24.3; cargo build --release; cd ..
   Compiling libc v0.2.93
   Compiling autocfg v1.0.1
   ...
   Finished release [optimized] target(s) in 1m 17s

~/tmp took 1m17s 
❯ ln -s ./rustup-1.24.3/target/release/rustup-init ./rustup-init

~/tmp
❯ env RUSTUP_HOME=.rustup CARGO_HOME=.cargo ./rustup-init -y --no-modify-path --profile minimal
info: profile set to 'minimal'
info: default host triple is aarch64-apple-darwin
info: syncing channel updates for 'stable-aarch64-apple-darwin'
info: latest update on 2022-05-19, rust version 1.61.0 (fe5b13d68 2022-05-18)

...

Rust is installed now. Great!

~/tmp took 20s 
❯ ls -il .cargo/bin
total 0
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 cargo -> /Users/stephen/tmp/rustup-init
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 cargo-clippy -> /Users/stephen/tmp/rustup-init
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 cargo-fmt -> /Users/stephen/tmp/rustup-init
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 cargo-miri -> /Users/stephen/tmp/rustup-init
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 clippy-driver -> /Users/stephen/tmp/rustup-init
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 rls -> /Users/stephen/tmp/rustup-init
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 rust-gdb -> /Users/stephen/tmp/rustup-init
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 rust-lldb -> /Users/stephen/tmp/rustup-init
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 rustc -> /Users/stephen/tmp/rustup-init
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 rustdoc -> /Users/stephen/tmp/rustup-init
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 rustfmt -> /Users/stephen/tmp/rustup-init
24584940 lrwxr-xr-x  12 stephen  staff  30 May 20 17:02 rustup -> /Users/stephen/tmp/rustup-init

Voilà, hardlinks to a symlink, and I did not "go manually fiddling with stuff installed by rustup".

[kinnison]

@stouset you appear to have erroneously taken my statement to another person as referring to the original meaning of this issue. The "manual fiddling" was about hube12's attempt to "override the cargo binary".

I did not mean to insinuate that the hardlink-to-symlink issue was because someone was manually fiddling; I've stated quite clearly already that I believe this is a combination of a slight misbehaviour in rustup's copy_file code combined with homebrew incorrectly packaging rustup-init.

[stouset]

My apologies if I misunderstood your reply.