Unable to run commands on Extreme SLXOS

[vivekest]

Hey Ronf, How are you? Truly appreciate asyncssh and your support to community over a decade now 🥇🥇🥇 Thank you!!!

I'm able to successfully login to a Extreme SLXOS switch but unable to run any commands on it using run or process methods. Following two codes work fine on Juniper and Linux but not SLXOS, operations via Putty and Secure CRT are always fine.

CODE 1

async def get_show_version(hostname, username, password):
    # Create an SSH connection
    async with asyncssh.connect(
            hostname,
            username=username,
            password=password,
    ) as conn:
        print('logged in', type(conn))
        await asyncio.sleep(10)
        print('awake')
        # Open a session
        result = await conn.run('show version',
                                term_type='xterm',    #Also tried vt100, vt220 and ansi
                                term_size=(80, 24),
                                request_pty='force')
        print(result.stdout, result.stderr)
asyncio.run(get_show_version('1.1.1.1', 'abc', 'pass'))

CODE 2

    async with client.create_process("show version", stderr=asyncssh.STDOUT) as process:
        while True:
            recv = await process.stdout.read(512)
            print('received', recv)
            break

"Login is successful but the command always times out as a result of SSH timeout in both codes.

Debug logs for code 1 (similar logs for code 2 as well):

DEBUG:asyncssh:[conn=0] Sending version SSH-2.0-AsyncSSH_2.13.1
DEBUG:asyncssh:[conn=0] Received version SSH-2.0-OpenSSH_7.5 PKIX[10.2]
DEBUG:asyncssh:[conn=0] Requesting key exchange
DEBUG:asyncssh:[conn=0] Received key exchange request
DEBUG:asyncssh:[conn=0] Beginning key exchange
DEBUG:asyncssh:[conn=0] Completed key exchange
INFO:asyncssh:[conn=0] Beginning auth for user admin
DEBUG:asyncssh:[conn=0] Received authentication banner
DEBUG:asyncssh:[conn=0] Trying password auth
INFO:asyncssh:[conn=0] Auth for user admin succeeded
logged in <class 'asyncssh.connection.SSHClientConnection'>
DEBUG:asyncssh:[conn=0] Received unknown global request: [email protected]
awake
DEBUG:asyncssh:[conn=0, chan=0] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=0] Requesting new SSH session
DEBUG:asyncssh:[conn=0, chan=0]   Terminal type: xterm
DEBUG:asyncssh:[conn=0, chan=0]   Terminal size: 80x24
INFO:asyncssh:[conn=0, chan=0]   Command: show version
INFO:asyncssh:[conn=0, chan=0] Received exit status 3
INFO:asyncssh:[conn=0, chan=0] Received channel close
INFO:asyncssh:[conn=0, chan=0] Channel closed
INFO:asyncssh:[conn=0] Closing connection
INFO:asyncssh:[conn=0] Sending disconnect: Disconnected by application (11)
INFO:asyncssh:[conn=0] Connection closed
Last login: Wed May 17 16:15:04 2023 from 1.2.7.2
Unsuccessful attempts: 0 since last login.

SECURITY WARNING: The default password for at least
one default account (root, admin and user) have not been changed.

Welcome to the Extreme SLX-OS Software
admin connected from 1.2.7.2 using ssh on SWITCH
SWITCH# 
*** IDLE TIMEOUT ***

Doesn't matter how many times I send a command it always times out, and in all results stdout contains the SSH login banner + this "*** IDLE TIMEOUT ***" (generated by switch at ssh timeout), env: {}, command: show version, subsystem:None, exit_status: 3, exit_signal: None, returncode: 3

As we are getting *** IDLE TIMEOUT *** in stdout, we can say that there's at least one way communication happening between my system and switch. How to troubleshoot this further?

Thank you!!

[ronf]

I don't have any direct experience with SLXOS, but from what you're showing here it looks to me like maybe it doesn't support using "exec" (where you pass a command in to run, instead of asking for a "shell"). If you change the first argument to create_process() from "show version" to input="show version\r", does that help?

You could also try this same change on the version using run().

[vivekest]

Hello Ronf, thanks for your response, running create_process() and run() with input="show version\r" arg did make a difference.

Using create_process() SSH session didn't timeout and cmd executed quickly, but like prev case stdout contains the SSH login banner instead of the cmd output.

Using run() when the term_type is not set the SSH session doesn't timeout and cmd executes quickly, but like prev case stdout contains the SSH login banner instead of the cmd output.

Using run() when the term_type is not set the SSH session doesn't timeout and cmd executes quickly, but like prev case stdout contains the SSH login banner instead of the cmd output.

With run() when the term_type is set the SSH session does timeout and cmd executes quickly, but unlike prev case stdout contains the SSH login banner and also the required cmd output with the prompt at the end.

This is the code is used this time:

logging.basicConfig(level=logging.DEBUG)

async def get_show_version(hostname, username, password):
    # Create an SSH connection
    async with asyncssh.connect(hostname,username=username,password=password,known_hosts=None) as conn:
        
        # Open a session using create_process
        async with conn.create_process("show version", stderr=asyncssh.STDOUT) as process:
            start = datetime.now()
            result1 = await process.stdout.read(512)
            print(f"Result 1 stdout in {datetime.now() - start}: {result1}")

        # Open a session using run without term_type
        start = datetime.now()
        result2 = await conn.run(input="show version\r")
        print(f"Result 2 complete in {datetime.now() - start}: {result2}")

        # Open a session using run with term_type
        start = datetime.now()
        result3 = await conn.run(input="show version\r",
                                 term_type='xterm',
                                 term_size=(80, 24))
        print(f"Result 3 complete in {datetime.now() - start}: {result3}")

asyncio.run(get_show_version('1.1.1.1', 'abc', 'pass'))

DEBUG LOGS:

INFO:asyncssh:[conn=0] Auth for user admin succeeded
DEBUG:asyncssh:[conn=0, chan=0] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=0] Requesting new SSH session
DEBUG:asyncssh:[conn=0] Received unknown global request: [email protected]
INFO:asyncssh:[conn=0, chan=0]   Command: show version
INFO:asyncssh:[conn=0, chan=0] Closing channel
**Result 1 stdout in 0:00:00.031782: **SHOWS LOGIN BANNER HERE****

INFO:asyncssh:[conn=0, chan=0] Received exit status 0
INFO:asyncssh:[conn=0, chan=0] Received channel close
INFO:asyncssh:[conn=0, chan=0] Channel closed
DEBUG:asyncssh:[conn=0, chan=1] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=1] Requesting new SSH session
INFO:asyncssh:[conn=0, chan=1]   Interactive shell requested
INFO:asyncssh:[conn=0, chan=1] Received exit status 0
INFO:asyncssh:[conn=0, chan=1] Received channel close
INFO:asyncssh:[conn=0, chan=1] Channel closed
DEBUG:asyncssh:[conn=0, chan=2] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=2] Requesting new SSH session

**Result 2 complete in 0:00:00.932609: env: {}, command: None, subsystem: None, exit_status: 0, exit_signal: None, returncode: 0, stdout: **SHOWS LOGIN BANNER HERE**, stderr:**
DEBUG:asyncssh:[conn=0, chan=2]   Terminal type: xterm
DEBUG:asyncssh:[conn=0, chan=2]   Terminal size: 80x24
INFO:asyncssh:[conn=0, chan=2]   Interactive shell requested
INFO:asyncssh:[conn=0] Connection failure: [WinError 1236] The network connection was aborted by the local system
INFO:asyncssh:[conn=0, chan=2] Closing channel due to connection close
INFO:asyncssh:[conn=0, chan=2] Channel closed: [WinError 1236] The network connection was aborted by the local system
INFO:asyncssh:[conn=0] Closing connection
INFO:asyncssh:[conn=0] Sending disconnect: Disconnected by application (11)

**Result 3 complete in 0:13:46.533957: env: {}, command: None, subsystem: None, exit_status: None, exit_signal: None, returncode: None, stdout: **SHOWS LOGIN BANNER + REQUIRED CMD OUTPUT WITH PROMPT**, stderr:**

[ronf]

In the case where it worked, it is completely expected behavior that you'll also get the login banner before the desired command output and the prompt after. Unfortunately, when using "shell" style access (which seems to be your only option here), all the output is mixed together, and other than parsing the output for the prompt, there's really no way to know where the different parts of the output begin/end. If you know the exact prompt to expect, you can use readuntil() on stdout to automatically break things up, and then just delete the appropriate number of characters at the end of each returned block based on the length of the prompt which you matched. The only potential danger is that the command output could contain the prompt, but that's relatively unlikely, and depending on the command you run may not ever happen in practice.

Based on your tests, it seems like you need to make sure a PTY is allocated for things to work, along with requesting a "shell" and writing your command to stdin (which is what input is doing when you set it) rather then sending the command in as part of an "exec" request.

[vivekest]

Ronf, the working case is not properly working. It takes ~13mins for result3 to get the result as the result is obtained at the end of SSH timeout, a new conn is required to run more commands. How can I do this better?

result3 = await conn.run(input="show version\r",
                                 term_type='xterm',
                                 term_size=(80, 24))

[ronf]

Ah, sorry. I misread the part about the last test case timing out. I'm surprised that it wouldn't exit right away, as we should be sending EOF immediately after sending the command when input= is used. If your goal is to run multiple commands on the same session, though, using input= is probably not the right approach anyway. I think your only option with this target is probably to use readuntil() to read until you see a prompt. So, the process would look something like:

• Open a connection and authenticate with asyncssh.connect() with term_type set to request a PTY.
  • Create a process with create_process() without specifying a command or input=.
  • Read login banner using readuntil() on the process stdout looking for the prompt you're expecting to see.
  • While you have commands to send:
    • Send a command by using write() on the process stdin.
    • Read back the command output and next prompt using readuntil() on stdout.
    • Remove the trailing bytes from the read result to throw away the prompt which was read.

Since you are asking for a PTY here, you don't need to worry about picking up any output on stderr. When a PTY is requested, all output to stdout and stderr are combined and will appear on stdout.

[vivekest]

I don't have the exact prompt for the devices on which I'm trying to run commands but I do have a list prompt regexs. I think we can't use readuntil() in such a scenario. I need to get the auth banner, login banner and prompt after creating a SSH conn. Then if the prompt matches the regex I can run my commands. Ain't there some general code that works for all network nodes? Putty and SecureCRT does it so easily, I'm already using run() method on sshclient to run commands on Juniper and Linux, method to do the same on Extreme SLXOS is going to be different. I need to do the same for Cisco and other vendors also in future. What methods and classes should I use to keep my code short and uniform for such a requirement? Thank you!!!

[ronf]

Unfortunately, a regex can't be used here. The standard Python regex module can't really handle incremental matching, which is what would be needed here to match on the prompt as data is read a piece at a time. The readuntil() code can match on a list of possible prompts, but all of them have to be exact match, or really a "suffix" match as you don't need to match on the full prompt as long as the piece you do match on never appears in the command output.

As for a single piece of code that works against all devices, that's difficult given that different target devices support different subsets of the SSH protocol. So, just because a piece of code works on one of the targets doesn't mean it will work on the others, as you've seen.

The "least common denominator" would seem to require that you only ever request a "shell" session, and that you only ever open one such "shell" session per connection. It also probably requires you to request a PTY, since some systems seem to require that.

Once started, you can run multiple commands on that session, but only if you can figure out when the session is ready to accept a new command, and what parts of the output are banners vs. command output. The only way I can see to do that would be by matching on the prompt that the target system outputs. That means you'll need to know what that prompt string is, though, and it must be unique (not appearing in any banners or command output).

If you got something like this working, I think it would probably work on almost any device, since that would look like what the target would see when using OpenSSH, PuTTY, or SecureCRT to connect to it. The hard part is understanding the output. That's easy for a human running an interactive client, but less easy to do with a script unless you know something about the target and the expected command output.

[ronf]

Closing due to inactivity. Feel free to reopen this or open a new issue if you need anything else.