SSH tunnels stop working after some time

[f1nality]

I faced strange behavior. I'm using the following code to set up SSH tunnels to connect to PostgreSQL

    from sshtunnel import SSHTunnelForwarder
    import paramiko

    private_key_str = conf.get('ssh_private_key').replace('\\n', '\n')
    private_key = paramiko.RSAKey.from_private_key(StringIO(private_key_str))

    server = SSHTunnelForwarder(
        ssh_address_or_host=(conf.get('ssh_host'), int(conf.get('ssh_port'))),
        ssh_username=conf.get('ssh_user'),
        ssh_pkey=private_key,
        remote_bind_address=(conf.get('host'), int(conf.get('port'))),
        logger=logger
    )
    server.start()

When connection is made everything works ok, but after some time (~10 hours) i'm getting PSQL error:

psycopg2.OperationalError: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

netstat -lntu shows opened ports, but making telnet returns error.

bash-5.0# netstat -lntu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 0.0.0.0:36877           0.0.0.0:*               LISTEN       
bash-5.0# telnet localhost:36877
Connected to localhost:36877
Connection closed by foreign host

All new SSH tunnels started afterwards - also returns errors right after start (and somehow they create two open ports). Only application restart solves the problem.

Starting SSH tunnel manually through ssh -L localhost:1111:DB_HOST:DB_PORT -i KEY BASTION_USER@BASTION_HOST works fine, such tunnels work and don't return telnet error.

[lucasgherculano]

did you tried the keepalive parameter?

[f1nality]

@lucasgherculano Do you mean SSH client or server? Tried adding SSH bastion server parameters, but need time to test if it helped:

ClientAliveInterval 300
ClientAliveCountMax 2

I thought sshtunnel should implement keepalive on its side.

[f1nality]

keepalive didn't work

[f1nality]

Manually creating SSH tunnel with "ssh" command works ok

[lucaslcode]

@f1nality also encountering this. Did you ever find a solution?

[matthew-k2001]

Hi @f1nality , any idea how you fixed this?

[f1nality]

@lucaslcode @matthew-k2001 I've switched to using "ssh" command to create tunnels in the end :(

[matthew-k2001]

Thanks @f1nality for the quick reply, could you elaborate what you mean?

[f1nality]

@matthew-k2001 i'm using subprocess.Popen to run command "ssh -N ...". You can check source code here: https://github.com/jet-admin/jet-bridge/blob/master/packages/jet_bridge_base/jet_bridge_base/ssh_tunnel.py

[matthew-k2001]

Thanks @f1nality, my knowlegde might be lacking here so could you clarify why does that keep the connection. Is it similar to AutoSSH?

[f1nality]

@matthew-k2001 this library sshtunnel implements SSH using paramiko and TCP sockets, "ssh" command is a much more widely used utility in linux distributives which has its own implementation. So it is actually different code, making the same. But looks like sshtunnel or paramiko (less likely) has some bug with keeping connections alive, but "ssh" command does not have this bug.

[matthew-k2001]

Thanks @f1nality that makes sense!