Unix shell support for nerves_ssh

[SteffenDE]

Hello there,

this is a continuation of a discussion from #83. I think it would be a great feature to allow connecting to nerves devices using SSH into a unix shell, separate to the existing IEx / Erlang shells. Therefore, I've written :nerves_ssh_shell and this issue discusses possible ways to maybe integrate this feature into :nerves_ssh in the future.

---

Some thoughts:

The only problem I see is that currently nerves_ssh_shell depends on erlexec, as this is the only way I found to actually run a process inside a pseudoterminal from the BEAM world and it currently does not compile for musl based systems.

I've seen https://github.com/jjcarstens/extty/pull/8, but this does not create a "real" pseudoterminal for running interactive shell commands. nerves.system.shell works around this by wrapping commands with the script utility, but this is not a good solution as there seems to be no way to handle resizing terminals properly. Also, depending on tools like script and stty on nerves would require changing the base systems.

So with all that, using erlexec works great and I've already contributed a change that makes using interactive terminals much easier. So maybe the way forward is:

1. Find out why erlexec does not work with musl and try to fix it
2. Make erlexec an optional dependency of :nerves_ssh and clarify what needs to be configured for Unix shell support
3. Include, document and test the ssh_server_channels from :nerves_ssh_shell for using the shells, either as SSH subsystem (with reduced features) or using a separate daemon

Still, I feel like I this is maybe not the best solution, as the configuration needed for erlexec is global and might interfere with users that already use erlexec with other settings. Maybe there could be a very small wrapper around Port or a nif that allows spawning a command in a pty (https://man7.org/linux/man-pages/man7/pty.7.html) and setting possible pty options (modes, window size, ...), but I'm not proficient enough in C / Rust / Zig to write anything like this.

[SteffenDE]

While working on https://github.com/saleyn/erlexec/pull/159, I found that compiling with musl works fine one alpine; so maybe this issue is already fixed by https://github.com/nerves-project/nerves_system_br/commit/fb4e00870beb05f81e5bef22562fb30bc754ac56, I'll try that out soon.

Inspired by https://github.com/jjcarstens/extty, I tried to implement https://github.com/SteffenDE/ex_pty, but looking at the resulting C code I'd very much prefer to use erlexec. As already mentioned above, my C is horrible, but if someone is inspired by this and likes to clean it up, feel free.

[fhunleth]

My feeling is that if erlexec works then that sounds like a great route. I skimmed the erlexec code and it seems fine. At least I'm not seeing anything yet that would motivate me to spend time on something new.

[SteffenDE]

So I just tried using the x86_64 base system with musl, the error is still the same iirc:

DEBUG=1 MIX_TARGET=x86_64 mix firmware
==> nerves
==> ssh_multi_daemon_example

Nerves environment
  MIX_TARGET:   x86_64
  MIX_ENV:      dev

===> Compile (apps)
===> Expanded command sequence to be run: []
===> Running provider: do
===> Expanded command sequence to be run: [app_discovery,{bare,compile}]
===> Running provider: app_discovery
===> Found top-level apps: [erlexec]
        using config: [{src_dirs,["src"]},{lib_dirs,["apps/*","lib/*","."]}]
===> Compile (apps)
===> Not adding provider pc compile from module pc_prv_compile because it already exists from module pc_prv_compile
===> Not adding provider pc clean from module pc_prv_clean because it already exists from module pc_prv_clean
===> Running provider: {bare,compile}
===> Compile (untagged)
===> Running hooks for compile in app erlexec (/Users/steffen/Desktop/MonitorNews/nerves_containers/nerves_ssh_shell/examples/ssh_multi_daemon_example/deps/erlexec) with configuration:
===>    {provider_hooks, [{pre, [{pc,compile}]}]}.
===> Running provider: {pc,compile}
===> Running hooks for {pc,compile} with configuration:
===>    {provider_hooks, []}.
===>    {pre_hooks, []}.
===> Linking /Users/steffen/Desktop/MonitorNews/nerves_containers/nerves_ssh_shell/examples/ssh_multi_daemon_example/_build/x86_64_dev/lib/erlexec/priv/x86_64-nerves-linux-musl/exec-port
===> sh(/Users/steffen/.nerves/artifacts/nerves_toolchain_x86_64_nerves_linux_musl-darwin_arm-1.6.0/bin/x86_64-nerves-linux-musl-g++ c_src/ei++.o c_src/exec.o c_src/exec_impl.o --sysroot=/Users/steffen/.nerves/artifacts/nerves_system_x86_64-portable-1.20.0/staging  -L/Users/steffen/.nerves/artifacts/nerves_system_x86_64-portable-1.20.0/staging/usr/lib/erlang/lib/erl_interface-5.3/lib -lei -o /Users/steffen/Desktop/MonitorNews/nerves_containers/nerves_ssh_shell/examples/ssh_multi_daemon_example/_build/x86_64_dev/lib/erlexec/priv/x86_64-nerves-linux-musl/exec-port)
failed with return code 1 and the following output:
/Users/steffen/.nerves/artifacts/nerves_toolchain_x86_64_nerves_linux_musl-darwin_arm-1.6.0/bin/../lib/gcc/x86_64-nerves-linux-musl/11.3.0/../../../../x86_64-nerves-linux-musl/bin/ld: warning: libc.so, needed by /Users/steffen/.nerves/artifacts/nerves_toolchain_x86_64_nerves_linux_musl-darwin_arm-1.6.0/bin/../lib/gcc/x86_64-nerves-linux-musl/11.3.0/../../../../x86_64-nerves-linux-musl/lib/../lib64/libstdc++.so, not found (try using -rpath or -rpath-link)
/Users/steffen/.nerves/artifacts/nerves_toolchain_x86_64_nerves_linux_musl-darwin_arm-1.6.0/bin/../lib/gcc/x86_64-nerves-linux-musl/11.3.0/../../../../x86_64-nerves-linux-musl/bin/ld: /Users/steffen/Desktop/MonitorNews/nerves_containers/nerves_ssh_shell/examples/ssh_multi_daemon_example/_build/x86_64_dev/lib/erlexec/priv/x86_64-nerves-linux-musl/exec-port: hidden symbol `atexit' in /Users/steffen/.nerves/artifacts/nerves_system_x86_64-portable-1.20.0/staging/usr/lib64/libc_nonshared.a(atexit.oS) is referenced by DSO
/Users/steffen/.nerves/artifacts/nerves_toolchain_x86_64_nerves_linux_musl-darwin_arm-1.6.0/bin/../lib/gcc/x86_64-nerves-linux-musl/11.3.0/../../../../x86_64-nerves-linux-musl/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status

Not sure what to make of this.

[jjcarstens]

Looks like erlexec has a hard requirement on glibc. I would need to investigate further if that can be mediated in the lib, or if we can enable something like glibcompat in the x86 system - or if we replicate the x86 system with gnu rather than musl

[SteffenDE]

Hmm I though alpine uses musl, but there compiling erlexec works fine:

$ docker run --rm -v (pwd):(pwd) -w (pwd) -it hexpm/elixir:1.13.4-erlang-25.0.2-alpine-3.16.0 sh
# apk add --update build-base
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/aarch64/APKINDEX.tar.gz
(1/17) Installing binutils (2.38-r2)
(2/17) Installing libmagic (5.41-r0)
(3/17) Installing file (5.41-r0)
(4/17) Installing libgomp (11.2.1_git20220219-r2)
(5/17) Installing libatomic (11.2.1_git20220219-r2)
(6/17) Installing gmp (6.2.1-r2)
(7/17) Installing isl22 (0.22-r0)
(8/17) Installing mpfr4 (4.1.0-r0)
(9/17) Installing mpc1 (1.2.1-r0)
(10/17) Installing gcc (11.2.1_git20220219-r2)
(11/17) Installing musl-dev (1.2.3-r0)
(12/17) Installing libc-dev (0.7.2-r3)
(13/17) Installing g++ (11.2.1_git20220219-r2)
(14/17) Installing make (4.3-r0)
(15/17) Installing fortify-headers (1.1-r1)
(16/17) Installing patch (2.7.6-r7)
(17/17) Installing build-base (0.5-r3)
Executing busybox-1.35.0-r13.trigger
OK: 199 MiB in 41 packages
# mix local.rebar --force
# export PATH=/root/.mix:$PATH
# make
===> Analyzing applications...
===> Compiling pc
===> Verifying dependencies...
===> Compiling c_src/ei++.cpp
===> Compiling c_src/exec.cpp
===> Compiling c_src/exec_impl.cpp
===> Linking /Users/steffen/Desktop/MonitorNews/nerves_containers/erlexec/priv/aarch64-unknown-linux-musl/exec-port
===> Analyzing applications...
===> Compiling erlexec
# echo $?
0
# ls -la /lib/libgcompat.so.0
ls: /lib/libgcompat.so.0: No such file or directory

So gcompat does not seem to be needed (see https://pkgs.alpinelinux.org/contents?file=&path=&name=gcompat&branch=v3.16&repo=community&arch=x86_64).

[SteffenDE]

So, quick update: Passing -rpath-link to the linker actually makes the link succeed:

/Users/steffen/.nerves/artifacts/nerves_toolchain_x86_64_nerves_linux_musl-darwin_arm-1.6.0/bin/x86_64-nerves-linux-musl-g++ \
	c_src/ei++.o c_src/exec.o c_src/exec_impl.o \
	--sysroot=/Users/steffen/.nerves/artifacts/nerves_system_x86_64-portable-1.20.0/staging \
	-Wl,-rpath-link,/Users/steffen/.nerves/artifacts/nerves_toolchain_x86_64_nerves_linux_musl-darwin_arm-1.6.0/x86_64-nerves-linux-musl/sysroot/usr/lib \
	-L/Users/steffen/.nerves/artifacts/nerves_system_x86_64-portable-1.20.0/staging/usr/lib/erlang/lib/erl_interface-5.3/lib \
	-lei -o /Users/steffen/Desktop/MonitorNews/nerves_containers/nerves_ssh_shell/examples/ssh_multi_daemon_example/_build/x86_64_dev/lib/erlexec/priv/x86_64-nerves-linux-musl/exec-port

Changing the nerves_env.exs makes it compile:

--- old 2022-07-09 15:44:37.000000000 +0200
+++ deps/nerves_system_br/nerves_env.exs        2022-07-09 15:10:16.000000000 +0200
@@ -35,6 +35,14 @@
 
     String.replace_suffix(gcc, "-gcc", "")
   end
+
+  def toolchain_sysroot(toolchain_path) do
+    path = toolchain_path |> Path.join("*/sysroot") |> Path.wildcard() |> List.first()
+
+    path || Mix.raise("""
+    Could not find sysroot of toolchain!
+    """)
+  end
 end
 
 system_path =
@@ -43,7 +51,7 @@
 
 sdk_sysroot = Path.join(system_path, "staging")
 
-{_toolchain_path, crosscompile} =
+{toolchain_path, crosscompile} =
   if File.dir?(Path.join(system_path, "host")) do
     # Grab the toolchain from a Buildroot output directory
     toolchain_path = Path.join(system_path, "host")
@@ -141,7 +149,7 @@
   "-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -pipe -O2 --sysroot #{sdk_sysroot}"
 )
 
-System.put_env("LDFLAGS", "--sysroot=#{sdk_sysroot}")
+System.put_env("LDFLAGS", "--sysroot=#{sdk_sysroot} -Wl,-rpath-link,#{Utils.toolchain_sysroot(toolchain_path)}/usr/lib")
 System.put_env("ERL_CFLAGS", "-I#{erts_dir}/include -I#{erl_interface_dir}/include")
 System.put_env("ERL_LDFLAGS", "-L#{erl_interface_dir}/lib -lei")
 
@@ -218,4 +226,3 @@
   run 'make menuconfig' and look for the Erlang options.
   """)
 end
-

I tested this using nerves_system_x86_64 and it seems to work (despite the system being very broken, see https://github.com/nerves-project/nerves_system_x86_64/issues/170).

[SteffenDE]

So it turns out that https://github.com/nerves-project/nerves_system_x86_64/issues/170 also fixed this issue. Turns out that using the musl toolchain with a gnu system leads to some strange issues 😄

As everything seems to work now, I'll try to come up with a draft PR soon to include the system shells as optional components in nerves_ssh.