Problem when sending multiple commands via jumpSSH using multithreading

[davidoiknine]

Hello,

I created a script where I send 15 000 commands to 70 switchs, so around 200 commands per switch. For each switch, I create one SSH connection via jumpSSH (via a jump server) this way:

# establish ssh connection between your local machine and the jump server
gateway_session = SSHSession('gateway.example.com',
                             'my_user', password='my_password').open()

# from jump server, establish connection with a remote server
remote_session = gateway_session.get_remote_session('remote.example.com',
                                                    password='my_password2')

Once the connection is established, I'm sending 200 commands using multithreading:

def main_loop():
   gateway_session = SSHSession('gateway.example.com',
                            'my_user', password='my_password').open()
   for switch in result.keys(): 
       remote_session = gateway_session.get_remote_session(switch["ip"],
                                                   password='my_password2')
       with ThreadPoolExecutor(max_workers=8) as executor:
           for command in result[switch]["commands"]:
               executor.submit(launch_command, command, remote_session)

def launch_command(command, remote_session):
   process = remote_session.get_cmd_output(command)           

The problem is that if I'm using 7 or 8 workers, it's not working properly (some information I get is missing) and I'm getting this error:

Secsh channel 55 open FAILED: open failed: Connect failed Secsh channel 60 open FAILED: open failed: Connect failed Secsh channel 113 open FAILED: open failed: Connect failed Secsh channel 170 open FAILED: open failed: Connect failed Secsh channel 213 open FAILED: open failed: Connect failed Secsh channel 44 open FAILED: open failed: Connect failed...

If I'm using 6 workers or less, I have no any error and everything is working correctly.

I have the impression that it's creating more than one SSH connection per leaf and so that's why there is this error, because maybe I'm reaching the number max of sessions allowed in switches, but according to my code, only one connection per switch is set up.

Do you know why I'm experiencing this issue? Thank you,