Invoking new SSHClient() never finishes

[yonatankarimish]

trafficstars

(note: I found this issue while trying to circumvent issue #612 )

I've been using the library for quite some time (version 0.27.0) for opening connections to network devices and executing commands on them. I recently encountered a problem creating new instances of SSHClient():

If I only create a single client, I encounter different problems with the channels opened by sshClient. (Will be opening a separate issue shortly), so I tried opening many ssh clients to check if that issue persists.

Surprisingly, I found out many clients never finished the invocation of their own constructor.

Minimal reproducible environment:

Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-131-generic x86_64)
JDK 11

Source code:

        for(int i = 0; i<=100; i++) {
            AtomicInteger counter = new AtomicInteger(i);
            threadingManager.submit(() -> { // submits the lambda to a central thread pool for concurrent execution
                try {
                    logger.fatal("Before creating ssh client #"+counter.get());
                    final SSHClient sshClient = new SSHClient();
                    logger.fatal("After creating ssh client #"+counter.get());
                } catch (Exception e) {
                    e.printStackTrace();
                }
            });
        }

Output:

After double checking, I found out that:

1. running the code above sequentially works using the code below, but is extremely slow (each new client takes longer than the previous one)

        for(int i = 0; i<=100; i++) {
            try {
                logger.fatal("Before creating ssh client #"+i);
                    final SSHClient sshClient = new SSHClient();
                logger.fatal("After creating ssh client #"+i);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }

2. running the original example with the "synchronized" keyword using the code below has the same results as the sequential example (which is quite logical, since it implies only one thread can create a new SSHClient at a time)

    for(int i = 0; i<=100; i++) {
        AtomicInteger counter = new AtomicInteger(i);
        threadingManager.submit(() -> {
            try {
                logger.fatal("Before creating ssh client #"+counter.get());
                synchronized (this) { //this is a singleton
                    final SSHClient sshClient = new SSHClient();
                }
                logger.fatal("After creating ssh client #"+counter.get());
            } catch (Exception e) {
                e.printStackTrace();
            }
        });
    }
   

3. After investigating the source code, I found the problematic part was in the BouncyCastleRandom constructor, line 21:

   ....= (new SecureRandom()).generateSeed(8);

Checking across the web, there were multiple issues regarding the implementation used by Java in their source code (notabely here and here).

Thanks for your help on this matter

[hierynomus]

Creating a new Random instance uses entropy (randomness) in your system. Randomness is generated by user interaction in normal computers. The problem is when you're using virtual machines or docker containers is that these have very little entropy (as there is little to no user interaction). The solution would be to use a single Config instance to create all the clients. This ensures that the Random instance is only created once.

[yonatankarimish]

Thank you for your fast reply.

I've read the Javadoc at your advice, and used a single Config instance as you suggested, but with varying levels of success. After several more exceptions, I created a simple benchmark test that generates 100 new ssh clients and connects them to the vm on which my code is running.

1. When running the code sequentially, all 100 clients are opened successfully

        final HostConfig.Host localhostConfig = sessionEngine.localhostConfig; //configuration of the os running the jvm
        final DefaultConfig sshConfig = new DefaultConfig(); //Default config for new ssh clients

        for(int i = 0; i<=100; i++) {
            logger.fatal("Before starting session #"+i);
            try {
                SSHClient sshClient = new SSHClient(sshConfig);
                sshClient.addHostKeyVerifier(new PromiscuousVerifier());
                sshClient.connect(localhostConfig.getHost(), localhostConfig.getPort());
                sshClient.authPassword(localhostConfig.getUsername(), localhostConfig.getPassword());
            }catch (Exception e) {
                logger.fatal("Failed to start session #"+i + ". Caused by: ", e);
            }
            logger.fatal("After starting session #"+i);
        }

2. When running the code in parallel, Transport exceptions are thrown:

        final HostConfig.Host localhostConfig = sessionEngine.localhostConfig; //configuration of the os running the jvm
        final DefaultConfig sshConfig = new DefaultConfig(); //Default config for new ssh clients

        Object lock = new Object();
        for(int i = 0; i<=100; i++) {
            AtomicInteger counter = new AtomicInteger(i);
            threadingManager.submit(() -> { // submits the lambda to a central thread pool for concurrent execution
                logger.fatal("Before starting session #"+counter.get());
                try {
                    SSHClient sshClient = new SSHClient(sshConfig);
                    sshClient.addHostKeyVerifier(new PromiscuousVerifier());
                    sshClient.connect(localhostConfig.getHost(), localhostConfig.getPort());
                    sshClient.authPassword(localhostConfig.getUsername(), localhostConfig.getPassword());
                }catch (Exception e) {
                    logger.fatal("Failed to start session #"+counter.get() + ". Caused by: ", e);
                }
                logger.fatal("After starting session #"+counter.get());
            });
        }

3. When synchronizing the connect + auth section, the clients open successfully final HostConfig.Host localhostConfig = sessionEngine.localhostConfig; //configuration of the os running the jvm final DefaultConfig sshConfig = new DefaultConfig(); //Default config for new ssh clients

    Object lock = new Object();
    for(int i = 0; i<=100; i++) {
        AtomicInteger counter = new AtomicInteger(i);
        threadingManager.submit(() -> { // submits the lambda to a central thread pool for concurrent execution
            logger.fatal("Before starting session #"+counter.get());
            try {
                SSHClient sshClient = new SSHClient(sshConfig);
                sshClient.addHostKeyVerifier(new PromiscuousVerifier());
                synchronized (lock) {
                    sshClient.connect(localhostConfig.getHost(), localhostConfig.getPort());
                    sshClient.authPassword(localhostConfig.getUsername(), localhostConfig.getPassword());
                }
            }catch (Exception e) {
                logger.fatal("Failed to start session #"+counter.get() + ". Caused by: ", e);
            }
            logger.fatal("After starting session #"+counter.get());
        });
    }
   

[yonatankarimish]

Which leads me to three questions:

1. Are there any restrictions on creating clients, opening connections, allocating pty's and starting shells in parallel (multi-threaded)?

2. Reading the Javadoc of SSHClient, the disconnect() method states that "[there is a] thread spawned by the transport layer". Which parts of the library spawn threads / run in thread pools?

3. Is there any way to run these threads in a pool / executor of my choice, or to get a reference (pointer) to them?

[hierynomus]

1. No
2. The TransportImpl.init() starts the Reader
3. No, these are not poolable threads.

[yonatankarimish]

Since you answered "no" to the first question (i.e. there are no multi-threading restrictions), why does the 2nd example throw TransportExceptions while the 3rd example does not?

[yonatankarimish]

@hierynomus Checking again after 4 months... Any chance you can help me out with this issue?

[hierynomus]

What kind of Transport Exceptions are thrown? Can you create a reproducing integration test case using the integration test docker container?