ssh: ttyd crashing with SIGSEGV

[agners]

Describe the issue you are experiencing

It seems that ttyd (part of the SSH add-on) is crashing every now and then:

coredumpctl
TIME                           PID UID GID SIG     COREFILE EXE           SIZE
Fri 2022-04-01 13:57:28 UTC  12895   0   0 SIGSEGV none     /usr/bin/ttyd  n/a
Fri 2022-04-01 17:20:48 UTC  13274   0   0 SIGSEGV none     /usr/bin/ttyd  n/a
Mon 2022-04-04 11:29:38 UTC  15490   0   0 SIGSEGV none     /usr/bin/ttyd  n/a

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Which add-on are you reporting an issue with?

Terminal & SSH

What is the version of the add-on?

9.3.0

Steps to reproduce the issue

1. Install the add-on
2. Let the system run for multiple days
3. Observe SIGSEGV in coredumpctl

It seems that aarch64 systems are affected.

Anything in the Supervisor logs that might be useful for us?

No response

Anything in the add-on logs that might be useful for us?

No response

Additional information

From journalctl

Apr 04 11:29:17 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [2022/04/04 13:29:17:1154] N:  ++ [wsisrv|0|adopted] (1)
Apr 04 11:29:17 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [2022/04/04 13:29:17:1178] N: HTTP / - 172.30.32.2
Apr 04 11:29:17 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [2022/04/04 13:29:17:2805] N: HTTP /token - 172.30.32.2
Apr 04 11:29:17 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [2022/04/04 13:29:17:3412] N: WS   /ws - 172.30.32.2, clients: 1
Apr 04 11:29:17 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [2022/04/04 13:29:17:3458] N: started process, pid: 598
Apr 04 11:29:38 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [2022/04/04 13:29:38:2558] N: WS closed from 172.30.32.2, clients: 0
Apr 04 11:29:38 ha-prod audit[15490]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj==docker-default (enforce) pid=15490 comm="ttyd" exe="/usr/bin/ttyd" sig=11 res=1
Apr 04 11:29:38 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [2022/04/04 13:29:38:2562] N: killing process, pid: 598
Apr 04 11:29:38 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [2022/04/04 13:29:38:2565] N: lws_libuv_closewsi: [wsisrv|0|adopted]
Apr 04 11:29:38 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [2022/04/04 13:29:38:2567] N: lws_libuv_closewsi: thr 0: [wsisrv|0|adopted] sa left 2: dyn left: 3 (rk 0)
Apr 04 11:29:38 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [2022/04/04 13:29:38:2569] N:  -- [wsisrv|0|adopted] (0) 21.141s
Apr 04 11:29:38 ha-prod kernel: audit: type=1701 audit(1649071778.255:664): auid=4294967295 uid=0 gid=0 ses=4294967295 subj==docker-default (enforce) pid=15490 comm="ttyd" exe="/usr/bin/ttyd" sig=11 res=1
Apr 04 11:29:38 ha-prod audit: BPF prog-id=161 op=LOAD
Apr 04 11:29:38 ha-prod audit: BPF prog-id=162 op=LOAD
Apr 04 11:29:38 ha-prod systemd[1]: Started Process Core Dump (PID 47518/UID 0).
Apr 04 11:29:38 ha-prod kernel: audit: type=1334 audit(1649071778.275:665): prog-id=161 op=LOAD
Apr 04 11:29:38 ha-prod kernel: audit: type=1334 audit(1649071778.279:666): prog-id=162 op=LOAD
Apr 04 11:29:38 ha-prod systemd-coredump[47519]: Process 15490 (ttyd) of user 0 dumped core.
Apr 04 11:29:38 ha-prod systemd[1]: [email protected]: Deactivated successfully.
Apr 04 11:29:39 ha-prod audit: BPF prog-id=162 op=UNLOAD
Apr 04 11:29:39 ha-prod audit: BPF prog-id=161 op=UNLOAD
Apr 04 11:29:39 ha-prod kernel: audit: type=1334 audit(1649071779.007:667): prog-id=162 op=UNLOAD
Apr 04 11:29:39 ha-prod kernel: audit: type=1334 audit(1649071779.007:668): prog-id=161 op=UNLOAD
Apr 04 11:29:39 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [13:29:39] INFO: Starting Web Terminal...
Apr 04 11:29:39 ha-prod homeassistant/aarch64-addon-ssh:9.3.0/addon_core_ssh[429]: [2022/04/04 13:29:39:0797] N: ttyd 1.6.3-3e37e33 (libwebsockets 4.2.1-v4.2.1)

[agners]

Stacktrace of this issue:

Thread 1 "ttyd" received signal SIGSEGV, Segmentation fault.
0x000055d6b9b20dae in lws_callback_on_writable ()
(gdb) bt
#0  0x000055d6b9b20dae in lws_callback_on_writable ()
#1  0x000055d6b9b138da in read_cb ()
#2  0x00007f7845acc747 in ?? () from /usr/lib/libuv.so.1
#3  0x00007f7845acd283 in ?? () from /usr/lib/libuv.so.1
#4  0x00007f7845ad1ce0 in uv.io_poll () from /usr/lib/libuv.so.1
#5  0x00007f7845ac3f85 in uv_run () from /usr/lib/libuv.so.1
#6  0x000055d6b9b21b40 in lws_service ()
#7  0x000055d6b9b1314c in main ()

(using latest ttyd/libwebsocket version)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mdegat01]

Interesting. So I actually use Frenck's SSH & Web Terminal add-on rather then the one from here. That also uses ttyd and I don't have any coredumps even though I'm also on an aarch64 system. I wonder if there's some difference in the base images that causes this?

Also do you have to actually leave the ingress page open for multiple days for this to happen or simply run the addon?

[agners]

This still seems to be the case with latest homeassistant/aarch64-addon-ssh:9.6.0 (which comes with Alpine 3.16).

Interesting. So I actually use Frenck's SSH & Web Terminal add-on rather then the one from here. That also uses ttyd and I don't have any coredumps even though I'm also on an aarch64 system. I wonder if there's some difference in the base images that causes this?

Hm, that is an interesting hint.

Also do you have to actually leave the ingress page open for multiple days for this to happen or simply run the addon?

No, even a short time is enough. Just open the Terminal (via ingress) and then click something else or close the tab.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.