[Bug]: error: timeout waiting for connserver to register

[juergenbarth]

Current Behavior

When trying to connect a tab to a Synology NAS via ssh, I get the following error:

Disconnected from "[email protected]" error: conncontroller [email protected] start wsh connserver error: timeout waiting for connserver to register

No connection is established.

Expected Behavior

The program should connect to the server without any problems, as it does with my Ubuntu based Linux server.

Steps To Reproduce

1. Create a new tab
2. Create a terminal block
3. Click on the Laptop-Icon (Connect to Local Machine)
4. In the "Connect to" field enter either the hostname or the IP-address of the host to connect to, e.g. [email protected]
5. Watch the error message come up

Wave Version

Client Version 0.8.8 (202410012200)

OS

macOS 14.7

Architecture

arm64

Anything else?

No response

Questionnaire

• [X] I'm interested in fixing this myself but don't know where to start
• [ ] I would like to fix and I have a solution
• [ ] I don't have time to fix this right now, but maybe later

[esimkowitz]

@juergenbarth, what kind of authentication does your Synology server use for SSH?

[juergenbarth]

@esimkowitz : It uses ssh-key. This used to work under version 0.7.

[esimkowitz]

Do you have a password or a passphrase on your key? Is it stored in your SSH config or in an SSH Agent?

Also do you see anything weird in ~/.waveterm/waveapp.log when you try to connect?

[juergenbarth]

@esimkowitz There are no passwords on the key and it is stored in SSH config.

Here are a few interesting lines from ~/.waveterm/waveapp.log, taken from the latest connect attempt (changed the username 😎):

2024-10-08 18:47:23 [wavesrv] 2024/10/08 18:47:23.899730 SETMETA: block:a4a96f51-280d-4759-bb10-c53f95a17421 | map[connection:[email protected] file:] 2024-10-08 18:47:23 [wavesrv] 2024/10/08 18:47:23.910227 Connect [email protected] 2024-10-08 18:47:23 [wavesrv] 2024/10/08 18:47:23.910245 sending event: wps.WaveEvent{Event:"connchange", Scopes:[]string{"connection:[email protected]"}, Sender:"", Persist:0, Data:wshrpc.ConnStatus{Status:"connecting", Connection:"[email protected]", Connected:false, HasConnected:false, ActiveConnNum:0, Error:""}} 2024-10-08 18:47:24 [wavesrv] 2024/10/08 18:47:24.010925 remote domain socket [email protected] "" 2024-10-08 18:47:24 [wavesrv] 2024/10/08 18:47:24.103377 shell detecting using command: ~/.waveterm/bin/wsh shell 2024-10-08 18:47:24 [wavesrv] 2024/10/08 18:47:24.110502 detecting shell: /bin/sh 2024-10-08 18:47:24 [wavesrv] 2024/10/08 18:47:24.110540 starting conn controller: WAVETERM_JWT="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjb25uIjoianVlcmdlbkB3b29kcGVja2VyLm5hdmlnaW8uaW8iLCJjdHlwZSI6ImNvbm5zZXJ2ZXIiLCJleHAiOjE3NTk5NDIwNDQsImlhdCI6MTcyODQwNjA0NCwiaXNzIjoid2F2ZXRlcm0iLCJzb2NrIjoiL3RtcC93YXZldGVybS1mZjgzOGQ0ODMxYzIyYmFlLnNvY2sifQ.CFsWBbqbFQAMpcWpd8FNVYoavH9gEydw11vB4oW3XwM" ~/.waveterm/bin/wsh connserver 2024-10-08 18:47:24 [wavesrv] 2024/10/08 18:47:24.118767 [conncontroller:[email protected]:output] Error: error setting up domain socket rpc client: failed to connect to Unix domain socket: dial unix /tmp/waveterm-ff838d4831c22bae.sock: connect: permission denied 2024-10-08 18:47:24 [wavesrv] 2024/10/08 18:47:24.118763 conn controller ("[email protected]") terminated: Process exited with status 1 2024-10-08 18:47:29 [wavesrv] 2024/10/08 18:47:29.115051 error: unable to start conn server for [email protected]: timeout waiting for connserver to register 2024-10-08 18:47:29 [wavesrv] 2024/10/08 18:47:29.115226 domain socket listener shutting down

[esimkowitz]

@oneirocosm any ideas?

[esimkowitz]

@juergenbarth do you know if your NAS allows for mounting domain sockets over SSH? We use this mechanism to establish a connection between our app and your remote server.

Also, what OS does Synology use? Is it Linux?

[juergenbarth]

@esimkowitz All I can tell you is that this used to work with WaveTerm 0.7 and that I can log on this way directly from macOS Terminal via the ssh [email protected] command.

The Synology OS is a Linux derivative.

[esimkowitz]

I wonder if the OS has some safeguard that is preventing us from running our WSH helper binary on the remote. At the moment, all connections established via our connections UI require our WSH helper to be loaded onto the remote before we complete the connection

[esimkowitz]

Though we had a similar mechanism in the old app so it'd be weird if that were the regression...

[juergenbarth]

Could it be related to the shell on the Synology? We only have sh not bash or zsh...

[esimkowitz]

@juergenbarth do you see the wsh binary in ~/.waveterm/bin on your Synology NAS?

[juergenbarth]

@esimkowitz Yes it's there, the permissions are 777. I can execute it, it brings the usage info. It even gets updated to the latest release

[oneirocosm]

That's really interesting. What about the /tmp directory? Does it exist? And if it does, what permissions does it have?

[juergenbarth]

@oneirocosm /tmp does exist and has permissions drwxrwxrwt.

[oneirocosm]

I'm still not completely sure why the domain socket isn't connecting, but we found a different bug which may end up resolving this one. I'll keep you posted as we work on the fix.

[oneirocosm]

I just realized i was ambiguous about something. The /tmp dir that exists with drwxrwxrwt, was that on your host or on the NAS? I meant to have you check on the NAS. Also, if it does exist on the NAS, is it possible for you to create a file there while logged in as the user you are trying to connect to?

[juergenbarth]

@oneirocosm Yes it is on the NAS:

me@nas:~$ touch /tmp/test
me@nas:~$ ll /tmp/test
-rw------- 1 me users 0 Oct 19 09:04 /tmp/test
me@nas:~$ rm /tmp/test

[oneirocosm]

Hello again! We recently encountered something similar to this on an Alpine Linux machine. The cause there was that in the machine's sshd config file (/etc/ssh/sshd_config), there was a line AllowTcpForwarding no. This prevented the unix domain socket from being sent to the host.

If that is the case for you, one way to fix it would be to set AllowTcpForwarding yes

This will allow wsh to properly communicate across machines.

If you don't want to open tcp forwarding, or if you have a different issue, the next release should have a fix so you can connect to the machine without wsh, although in that case, you won't be able to use wsh commands with that connection.

We'd also like to make a longer term solution that keeps wsh in cases like this (we could do this by providing an alternative to unix domain sockets), but I don't know when that will be ready.

[joaberg]

I had the same problem on a RHEL server. Connection was just hanging, and never timing out. Setting "AllowTcpForwarding yes" on the server and restarting sshd, then restarted waveterm to clear the hanging connection. Now it works... thanks for the tip.

You should add this to a Q&A/Troubleshooting section in the docs.