tabby icon indicating copy to clipboard operation
tabby copied to clipboard

SSH Sessions Randomly Disconnect After 1 Hour in Versions 1.0.222–1.0.224

Open CryptoSiD opened this issue 7 months ago • 4 comments

Since updating to version 1.0.222, I'm experiencing unexpected SSH session disconnections.

Some SSH sessions consistently disconnect after exactly 1 hour, while others remain stable indefinitely. What's odd is that the same sessions always disconnect, suggesting a pattern rather than random drops.

I downgraded to version 1.0.221, and the issue no longer occurs — all sessions remain connected as expected.

To reproduce the issue, open around 10 SSH tabs to 10 different servers and leave them idle for at least 1 hour using version 1.0.222, 1.0.223, or the nightly version 1.0.224.

CryptoSiD avatar May 24 '25 15:05 CryptoSiD

I have the same issue and can confirm 1.0.221 fixes it.

hboyd2003 avatar May 28 '25 19:05 hboyd2003

+1

acumeko avatar May 29 '25 07:05 acumeko

+1

gnagnaf avatar May 30 '25 09:05 gnagnaf

+1

largefriessss avatar May 30 '25 23:05 largefriessss

I'm @ 1.0.223 and the issue is still present. All my sessions disconnect after 30-60 minutes.

When will "nightly" be released ?!

Onepamopa avatar Jul 29 '25 14:07 Onepamopa

I just started using tabby (v1.0.223) and I'm having the same issue. The disconnect seems to happen pretty consistently at around an hour. I've captured debug in case that is helpful (see below). If I can provide any other information please let me know. I'm really liking the application so far!

Tabby is running on Windows 11 (10.0.26100)

The remote server is running Rocky 10.0 Kernel: 6.12.0-55.18.1.el10_0.x86_64 #1 SMP PREEMPT_DYNAMIC sshd: OpenSSH_9.9p1, OpenSSL 3.2.2 4 Jun 2024 (RPM: 9.9p1-7.el10_0.rocky.0.1)

From Tabby (start time: 2025-07-31T18:40:42Z) [2025-07-31T19:40:42Z DEBUG russh::sshbuffer] > msg type 80, len 27 [2025-07-31T19:40:42Z DEBUG russh::client] < msg type 82, seqn 314001, len 1 [2025-07-31T19:40:42Z DEBUG russh::sshbuffer] > msg type 80, len 27 [2025-07-31T19:40:42Z DEBUG russh::client] < msg type 82, seqn 314002, len 1 [2025-07-31T19:40:42Z DEBUG russh::sshbuffer] > msg type 80, len 27 [2025-07-31T19:40:42Z DEBUG russh::client] < msg type 82, seqn 314003, len 1 [2025-07-31T19:40:42Z DEBUG russh::sshbuffer] > msg type 80, len 27 [2025-07-31T19:40:42Z DEBUG russh::client] beginning re-key [2025-07-31T19:40:42Z DEBUG russh::sshbuffer] > msg type 20, len 761 [2025-07-31T19:40:42Z DEBUG russh::sshbuffer] > msg type 80, len 27 [2025-07-31T19:40:42Z DEBUG russh::client] < msg type 82, seqn 314004, len 1 [2025-07-31T19:40:42Z DEBUG russh::client] < msg type 20, seqn 314005, len 923 [2025-07-31T19:40:42Z DEBUG russh::client::kex] negotiated algorithms: Names { kex: Name("curve25519-sha256"), key: Ecdsa { curve: NistP256 }, cipher: Name("aes128-ctr"), client_mac: Name("hmac-sha1"), server_mac: Name("hmac-sha1"), server_compression: None, client_compression: None, ignore_guessed: false, strict_kex: false } [2025-07-31T19:40:42Z DEBUG russh::sshbuffer] > msg type 30, len 37 [2025-07-31T19:40:42Z DEBUG russh::client] kex impl continues: ClientKex { cause: Rekey { strict: true, session_id: <32> }, state: "waiting for DH response" } [2025-07-31T19:40:42Z DEBUG russh::sshbuffer] > msg type 80, len 27 [2025-07-31T19:40:42Z DEBUG russh::client] < msg type 31, seqn 314006, len 248 [2025-07-31T19:40:42Z DEBUG russh::client::kex] received server host key: Ok("ecdsa-sha2-nistp256 ") [2025-07-31T19:40:42Z DEBUG russh::sshbuffer] > msg type 21, len 1 [2025-07-31T19:40:42Z DEBUG russh::client] kex impl continues: ClientKex { cause: Rekey { strict: true, session_id: <32> }, state: "waiting for NEWKEYS" } [2025-07-31T19:40:42Z DEBUG russh::client] < msg type 21, seqn 314007, len 1 [2025-07-31T19:40:42Z DEBUG russh::client] kex impl has completed [2025-07-31T19:40:42Z DEBUG russh::client] kex done [2025-07-31T19:40:42Z DEBUG russh::client] < msg type 82, seqn 1, len 1 [2025-07-31T19:40:42Z DEBUG russh::client] < msg type 82, seqn 2, len 1 [2025-07-31T19:40:42Z DEBUG russh::sshbuffer] > msg type 80, len 27 [2025-07-31T19:40:42Z DEBUG russh::sshbuffer] > msg type 80, len 27 [2025-07-31T19:40:42Z DEBUG russh::client] disconnected Russh(IO(Custom { kind: UnexpectedEof, error: "early eof" })) [2025-07-31T19:40:42Z DEBUG russh::client] drop session

Remote server log: Jul 31 15:40:43 sshd-session[6331]: Corrupted MAC on input. Jul 31 15:40:43 sshd-session[6331]: ssh_dispatch_run_fatal: Connection from user port 55881: message authentication code incorrect

jamiephill avatar Jul 31 '25 19:07 jamiephill

Unfortunately, the issue is still present in the latest release 1.0.224

CryptoSiD avatar Aug 08 '25 14:08 CryptoSiD

Yup ...... same shit again. @Eugeny any reason this not fixed yet? It's EXTREMELY annoying......

Onepamopa avatar Aug 08 '25 18:08 Onepamopa

version 1.0.225 +1 All sessions will be disconnected after about one hour, but this is not due to idle timeout — once it even happened while I was editing a file. Changing the keep-alive settings does not work.

master-genius avatar Aug 12 '25 06:08 master-genius

When I close the window, the following error often pops up. I’m not sure if it’s related to the disconnection. Here’s a screenshot of the error.

Microsoft Visual C++ Runtime Library

Assertion failed!

Program: ...ed\node_modules\node-pty\build\Release\conpty.node
File: D:\a\tabby\tabby\app\node_modules\node-pty\...\conpty.cc
Line: 110

Expression: status == napi_ok

For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts

(Press Retry to debug the application - JIT must be enabled)

master-genius avatar Aug 12 '25 06:08 master-genius

Experiencing this with the latest update on all my hosts, whether local or remote, even WSL (over ssh)

Gaibhne avatar Aug 12 '25 06:08 Gaibhne

Still present in 1.0.225 in .deb

4mig4 avatar Aug 14 '25 13:08 4mig4

Same issue.

Tabby 1.0.225 Windows 11 - 64 bits - 10.0.26100 Build 26100

[2025-08-17T10:20:01Z DEBUG russh::client] beginning re-key
[2025-08-17T10:20:01Z DEBUG russh::sshbuffer] > msg type 20, len 717
[2025-08-17T10:20:01Z DEBUG russh::client] < msg type 20, seqn 44, len 986
[2025-08-17T10:20:01Z DEBUG russh::client::kex] negotiated algorithms: Names { kex: Name("curve25519-sha256"), key: Ecdsa { curve: NistP256 }, cipher: Name("aes128-ctr"), client_mac: Name("hmac-sha1"), server_mac: Name("hmac-sha1"), server_compression: None, client_compression: None, ignore_guessed: false, strict_kex: false }
[2025-08-17T10:20:01Z DEBUG russh::sshbuffer] > msg type 30, len 37
[2025-08-17T10:20:01Z DEBUG russh::client] kex impl continues: ClientKex { cause: Rekey { strict: true, session_id: <32> }, state: "waiting for DH response" }
[2025-08-17T10:20:01Z DEBUG russh::client] < msg type 31, seqn 45, len 249
[2025-08-17T10:20:01Z DEBUG russh::client::kex] received server host key: Ok("ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDQkpZ5l+j2rEdbk+vjwjG+oxQ1W+F4i7yGEokdBYqLv9tHkUEfs94U2IxCbU0PNNS4kJaKJY60Y+XcEUt2gRp0=")
[2025-08-17T10:20:01Z DEBUG russh::sshbuffer] > msg type 21, len 1
[2025-08-17T10:20:01Z DEBUG russh::client] kex impl continues: ClientKex { cause: Rekey { strict: true, session_id: <32> }, state: "waiting for NEWKEYS" }
[2025-08-17T10:20:01Z DEBUG russh::client] < msg type 21, seqn 46, len 1
[2025-08-17T10:20:01Z DEBUG russh::client] kex impl has completed
[2025-08-17T10:20:01Z DEBUG russh::client] kex done
[2025-08-17T10:20:01Z DEBUG russh::client] < msg type 94, seqn 1, len 247
[2025-08-17T10:20:01Z DEBUG russh::client] < msg type 94, seqn 2, len 132
[2025-08-17T10:20:01Z DEBUG russh::client] < msg type 94, seqn 3, len 16
[2025-08-17T10:20:01Z DEBUG russh::client] < msg type 94, seqn 4, len 17
[2025-08-17T10:20:06Z DEBUG russh::sshbuffer] > msg type 94, len 10
[2025-08-17T10:20:06Z DEBUG russh::client] disconnected Russh(IO(Custom { kind: UnexpectedEof, error: "early eof" }))
[2025-08-17T10:20:06Z DEBUG russh::client] drop session

flocknroll avatar Aug 17 '25 10:08 flocknroll

Has version 226 fixed this issue?

guoh27 avatar Aug 18 '25 05:08 guoh27

Nop, still not fixed.

Onepamopa avatar Aug 18 '25 07:08 Onepamopa

@Eugeny are we gonna be able to work normally anytime soon? It's getting ridiculous ....

Onepamopa avatar Aug 18 '25 07:08 Onepamopa

It's very frustrating that a bug so severe that it makes this tool essentially unusable for many people has gone without so much as a comment that it has been noticed by the devs :(

Gaibhne avatar Aug 18 '25 08:08 Gaibhne

Please keep in mind that this is a free tool that I'm working on in my free time when my life allows it.

It's not that I'm purposedly ignoring you. I'm unable to reproduce it on my end despite trying different configurations. Some users reported that disabling SSH keepalive messages fixes it, so I've disabled it by default in v224 to see if it helps - apparently it doesn't.

It would be helpful if you could crank up the sshd log level and see if anything gets logged on the server side at the time of disconnect.

Eugeny avatar Aug 18 '25 08:08 Eugeny

Please keep in mind that this is a free tool that I'm working on in my free time when my life allows it.

It's not that I'm purposedly ignoring you. I'm unable to reproduce it on my end despite trying different configurations. Some users reported that disabling SSH keepalive messages fixes it, so I've disabled it by default in v224 to see if it helps - apparently it doesn't.

It would be helpful if you could crank up the sshd log level and see if anything gets logged on the server side at the time of disconnect.

Eugeny

Don't worry, most of us understand this and we are grateful.

4mig4 avatar Aug 18 '25 09:08 4mig4

Maybe try against REDHAT 7 , 8 and 9 ( or Centos 7 and Rocky 8 and 9 ?

4mig4 avatar Aug 18 '25 09:08 4mig4

Please keep in mind that this is a free tool that I'm working on in my free time when my life allows it.

It's not that I'm purposedly ignoring you. I'm unable to reproduce it on my end despite trying different configurations. Some users reported that disabling SSH keepalive messages fixes it, so I've disabled it by default in v224 to see if it helps - apparently it doesn't.

Absolutely understandable - I think everyone here understands bugs are hard and often can't be fixed quickly. The problem here, at least for me, was more that months went by without even an initial 'I'm looking at it' comment, leaving me to wonder whether anyone was investigating it at all, or maybe the whole issue fell through the cracks with nobody looking at it at all.

It would be helpful if you could crank up the sshd log level and see if anything gets logged on the server side at the time of disconnect.

Will do and report back.

Gaibhne avatar Aug 18 '25 09:08 Gaibhne

The client (I had two shells open, with connection sharing activated) only reports:

C:\Users\me\AppData\Local\Programs\Tabby\resources\builtin-plugins\tabby-core\dist\index.js:15690 [ssh-localhost-2223] Destroying
C:\Users\me\AppData\Local\Programs\Tabby\resources\builtin-plugins\tabby-core\dist\index.js:15690 [ssh-shell-localhost-2223] Destroying
C:\Users\me\AppData\Local\Programs\Tabby\resources\builtin-plugins\tabby-core\dist\index.js:15690 [ssh-localhost-2223] Destroying
C:\Users\me\AppData\Local\Programs\Tabby\resources\builtin-plugins\tabby-core\dist\index.js:15690 [ssh-shell-localhost-2223] Destroying

Followed by Unhandled Promise rejection: SendError, which I assume are not interesting (but let me know if they are and I will sanitize one and post it too).

The server, with logging at DEBUG3, reports (to my understanding) that the client attempts a new connection before the old one disconnects; as the log is very long and contains many things I don't quite understand, I am wary of posting it here in public. Any way to send it to your privately ?

Gaibhne avatar Aug 18 '25 13:08 Gaibhne

Please keep in mind that this is a free tool that I'm working on in my free time when my life allows it.

It's not that I'm purposedly ignoring you. I'm unable to reproduce it on my end despite trying different configurations. Some users reported that disabling SSH keepalive messages fixes it, so I've disabled it by default in v224 to see if it helps - apparently it doesn't.

It would be helpful if you could crank up the sshd log level and see if anything gets logged on the server side at the time of disconnect.

So I could replicate it with a CentOS 7.9 with DEBUG3 in sshd3

If you use a Virtual Machine be sure to install support for guest, otherwise the test is not valid

Looks to me like mishandled rekeying on the client side.

4mig4 avatar Aug 18 '25 17:08 4mig4

Happens on Windows as well, both old saved profiles, and newly created ones.

Onepamopa avatar Aug 18 '25 18:08 Onepamopa

This seems to be the same as https://github.com/Eugeny/tabby/issues/10651 where as a workaround disabling the compression is suggested. Would be interesting to hear if anyone experiencing the issue here is still experiencing it with compression disabled.

Gaibhne avatar Aug 19 '25 16:08 Gaibhne

This seems to be the same as #10651 where as a workaround disabling the compression is suggested. Would be interesting to hear if anyone experiencing the issue here is still experiencing it with compression disabled.

How do we do that? I don't see an option in the windows UI in the profile(s)?

Onepamopa avatar Aug 19 '25 17:08 Onepamopa

@Gaibhne you can send it to [email protected] and reference this ticket #

@Onepamopa compression negotiation is technically a part of the ssh kex, so you can disable both zlib compression algorithms under the Ciphers tab.

Eugeny avatar Aug 19 '25 18:08 Eugeny

Disabled both zlib compression algos - no change - still disconnected after 60 minutes.

Onepamopa avatar Aug 19 '25 21:08 Onepamopa

Maybe you need to compare all changes from 1.0.221 onward, to detect the culprit

Onepamopa avatar Aug 19 '25 21:08 Onepamopa

I can confirm that for me, disabling all compression algorithms other than none has allowed SOME my connections to survive through the night (~18 hours). For me, I needed a complete restart of Tabby for the changes to 'take' - maybe part of those settings are cached ? Originally, I tried disabling compression for a connection profile, closed all open connections and opened a new one and that one still timed out after an hour. Only after a reboot (due to OS updates) did the changed setting take effect. Some connections are still timing out.

Gaibhne avatar Aug 20 '25 09:08 Gaibhne