Using ansible-pylibssh produces Unable to decode JSON from response to get_capabilities(). Received 'None'. error

[netexgb]

SUMMARY

When running cisco.ios.ios_command with ansible_pylibssh, I get a JSON RPC decoding error, which I do not get when using paramiko

ISSUE TYPE

• Bug Report

COMPONENT NAME

cisco.ios

ANSIBLE VERSION

xxxx@xxxx:~$ python3 -m pip install --upgrade ansible-pylibssh
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: ansible-pylibssh in ./.local/lib/python3.8/site-packages (0.4.0)

xxxx@xxxx:~$ ansible --version
ansible [core 2.13.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/xxxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/xxxx/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/xxxx/.ansible/collections
  executable location = /home/xxxx/.local/bin/ansible
  python version = 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

xxxx@xxxx$ ansible-galaxy collection list cisco.ios

# /usr/local/lib/python3.8/dist-packages/ansible_collections
Collection Version
---------- -------
cisco.ios  3.3.0

# /home/xxxx/.ansible/collections/ansible_collections
Collection Version
---------- -------
cisco.ios  3.3.0

# /home/xxxx/.local/lib/python3.8/site-packages/ansible_collections
Collection Version
---------- -------
cisco.ios  3.3.0

CONFIGURATION

xxxx@xxxx$ ansible-config dump --only-changed
COLLECTIONS_PATHS(/etc/ansible/ansible.cfg) = ['/home/xxxx/.ansible/collections']
DEPRECATION_WARNINGS(/etc/ansible/ansible.cfg) = True
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = True
PARAMIKO_HOST_KEY_AUTO_ADD(/etc/ansible/ansible.cfg) = True
PARAMIKO_LOOK_FOR_KEYS(/etc/ansible/ansible.cfg) = True
RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False
TASK_TIMEOUT(/etc/ansible/ansible.cfg) = 0

STEPS TO REPRODUCE

ANSIBLE VERSION:
xxxx@xxxx:~/.ssh$ ansible --version
ansible [core 2.13.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/xxxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/xxxx/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/xxxx/.ansible/collections
  executable location = /home/xxxx/.local/bin/ansible
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True


PLAYBOOK:
---
- name: show configuration objects
  hosts: xxxx
  gather_facts: no

  tasks:
    - name: Verify Configuration
      cisco.ios.ios_command:
         commands:
           - sh run | s ^ro.*example
           - sh run | i ip pref.*DEF

      # when: inventory_hostname == "xxxx"
      register: item

    - debug:
         var: item.stdout_lines

EXPECTED RESULTS

xxxx@xxxx:/mnt/c/Users/xxxx/Repositories/$ ansible-playbook cisco_config_verify.yml -u xxxx --ask-pass -vvvvvv
ansible-playbook [core 2.13.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/xxxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/xxxx/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/xxxx/.ansible/collections
  executable location = /home/xxxx/.local/bin/ansible-playbook
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
SSH password:
setting up inventory plugins
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Parsed /etc/ansible/hosts inventory source with yaml plugin
Loading collection cisco.ios from /home/xxxx/.ansible/collections/ansible_collections/cisco/ios
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
Loading callback plugin default of type stdout, v2.0 from /home/xxxx/.local/lib/python3.8/site-packages/ansible/plugins/callback/default.py
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: cisco_config_verify.yml *******************************************************************************************************************Positional arguments: cisco_config_verify.yml
verbosity: 6
remote_user: xxxxx
connection: smart
timeout: 10
ask_pass: True
become_method: sudo
tags: ('all',)
inventory: ('/etc/ansible/hosts',)
forks: 5
1 plays in cisco_config_verify.yml

PLAY [show configuration objects] *******************************************************************************************************************META: ran handlers
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
Loading collection ansible.netcommon from /home/xxxx/.ansible/collections/ansible_collections/ansible/netcommon

TASK [Verify Configuration] *************************************************************************************************************************task path: /mnt/c/Users/xxxx/Repositories/xxxx/cisco_config_verify.yml:7
redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
<XXXX> attempting to start connection
<XXXX> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /home/xxxx/.local/bin/ansible-connection
<XXXX> local domain socket does not exist, starting it
<XXXX> control socket path is /home/xxxx/.ansible/pc/667f5f96cf
<XXXX> Loading collection ansible.netcommon from /home/xxxx/.ansible/collections/ansible_collections/ansible/netcommon
<XXXX> redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
<XXXX> Loading collection cisco.ios from /home/xxxx/.ansible/collections/ansible_collections/cisco/ios
<XXXX> redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
<XXXX> local domain socket listeners started successfully
<XXXX> loaded cliconf plugin ansible_collections.cisco.ios.plugins.cliconf.ios from path /home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/cliconf/ios.py for network_os ios
<XXXX> ssh type is set to auto
<XXXX> autodetecting ssh_type
[WARNING]: ansible-pylibssh not installed, falling back to paramiko
<XXXX> ssh type is now set to paramiko
<XXXX>
<XXXX> local domain socket path is /home/xxxx/.ansible/pc/667f5f96cf
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
<XXXX> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
<XXXX> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.ios.ios_command  at /home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/modules/ios_command.py
<XXXX> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.ios.ios_command
<XXXX> ANSIBLE_NETWORK_IMPORT_MODULES: complete


TASK [debug] ****************************************************************************************************************************************task path: /mnt/c/Users/xxxx/Repositories/xxxx/cisco_config_verify.yml:16
redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
<XXXX> attempting to start connection
<XXXX> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /home/xxxx/.local/bin/ansible-connection
<XXXX> found existing local domain socket, using it!
<XXXX> invoked shell using ssh_type: paramiko
<XXXX> ssh connection done, setting terminal
<XXXX> loaded terminal plugin for network_os ios
<XXXX> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<XXXX> firing event: on_open_shell()
<XXXX> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<XXXX> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<XXXX> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<XXXX> ssh connection has completed successfully
<XXXX> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<XXXX> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<XXXX> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<XXXX> updating play_context for connection
<XXXX>
<XXXX> local domain socket path is /home/xxxx/.ansible/pc/667f5f96cf

META: ran handlers
META: ran handlers

PLAY RECAP ******************************************************************************************************************************************XXXX : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

ACTUAL RESULTS

xxxx@xxxx:/mnt/c/Users/xxxx/Repositories/$ ansible-playbook cisco_config_verify.yml -u xxxx --ask-pass -vvvvvv
ansible-playbook [core 2.13.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/xxxx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/xxxx/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/xxxx/.ansible/collections
  executable location = /home/xxxx/.local/bin/ansible-playbook
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
SSH password:
setting up inventory plugins
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Parsed /etc/ansible/hosts inventory source with yaml plugin
Loading collection cisco.ios from /home/xxxx/.ansible/collections/ansible_collections/cisco/ios
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
Loading callback plugin default of type stdout, v2.0 from /home/xxxx/.local/lib/python3.8/site-packages/ansible/plugins/callback/default.py
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: cisco_config_verify.yml *******************************************************************************************************************Positional arguments: cisco_config_verify.yml
verbosity: 6
remote_user: xxxx
connection: smart
timeout: 10
ask_pass: True
become_method: sudo
tags: ('all',)
inventory: ('/etc/ansible/hosts',)
forks: 5
1 plays in cisco_config_verify.yml

PLAY [show configuration objects] *******************************************************************************************************************META: ran handlers
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
Loading collection ansible.netcommon from /home/xxxx/.ansible/collections/ansible_collections/ansible/netcommon

TASK [Verify Configuration] *************************************************************************************************************************task path: /mnt/c/Users/xxxx/Repositories/cisco_config_verify.yml:7
redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
<XXXX> attempting to start connection
<XXXX> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /home/xxxx/.local/bin/ansible-connection
<XXXX> local domain socket does not exist, starting it
<XXXX> control socket path is /home/xxxx/.ansible/pc/4a1c10f12f
<XXXX> Loading collection ansible.netcommon from /home/xxxx/.ansible/collections/ansible_collections/ansible/netcommon
<XXXX> redirecting (type: terminal) ansible.builtin.ios to cisco.ios.ios
<XXXX> Loading collection cisco.ios from /home/xxxx/.ansible/collections/ansible_collections/cisco/ios
<XXXX> redirecting (type: cliconf) ansible.builtin.ios to cisco.ios.ios
<XXXX> local domain socket listeners started successfully
<XXXX> loaded cliconf plugin ansible_collections.cisco.ios.plugins.cliconf.ios from path /home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/cliconf/ios.py for network_os ios
<XXXX> ssh type is set to auto
<XXXX> autodetecting ssh_type
<XXXX> ssh type is now set to libssh
<XXX> ![image](https://github.com/ansible-collections/cisco.ios/assets/50598498/7643b193-f00f-4705-97f8-861c756ba700)>
<XXXX> local domain socket path is /home/xxxx/.ansible/pc/4a1c10f12f
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
redirecting (type: action) cisco.ios.ios_command to cisco.ios.ios
<XXXXt> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
<XXXX> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.ios.ios_command  at /home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/modules/ios_command.py
<XXXX> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.ios.ios_command
<XXXX> ANSIBLE_NETWORK_IMPORT_MODULES: complete
<XXXX> ANSIBLE_NETWORK_IMPORT_MODULES: Result: {'failed': True, 'msg': "Unable to decode JSON from response to get_capabilities(). Received 'None'.", 'exception': '  File "/home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/module_utils/network/ios/ios.py", line 97, in get_capabilities\n    capabilities = Connection(module._socket_path).get_capabilities()\n  File "/home/xxxx/.local/lib/python3.8/site-packages/ansible/module_utils/connection.py", line 194, in __rpc__\n    response = self._exec_jsonrpc(name, *args, **kwargs)\n  File "/home/xxxx/.local/lib/python3.8/site-packages/ansible/module_utils/connection.py", line 173, in _exec_jsonrpc\n    raise ConnectionError(\n', 'invocation': {'module_args': {'commands': ['sh run | s ^ro.*example', 'sh run | i ip pref.*DEF'], 'match': 'all', 'retries': 10, 'interval': 1, 'wait_for': None, 'provider': None}}, '_ansible_parsed': True}
The full traceback is:
  File "/home/xxxx/.ansible/collections/ansible_collections/cisco/ios/plugins/module_utils/network/ios/ios.py", line 97, in get_capabilities
    capabilities = Connection(module._socket_path).get_capabilities()
  File "/home/xxxx/.local/lib/python3.8/site-packages/ansible/module_utils/connection.py", line 194, in __rpc__
    response = self._exec_jsonrpc(name, *args, **kwargs)
  File "/home/xxxx/.local/lib/python3.8/site-packages/ansible/module_utils/connection.py", line 173, in _exec_jsonrpc
    raise ConnectionError(
fatal: [XXXX]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "commands": [
                "sh run | s ^ro.*example",
                "sh run | i ip pref.*DEF"
            ],
            "interval": 1,
            "match": "all",
            "provider": null,
            "retries": 10,
            "wait_for": null
        }
    },
    "msg": "Unable to decode JSON from response to get_capabilities(). Received 'None'."
}

PLAY RECAP ******************************************************************************************************************************************XXXX : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

[webknjaz]

Ref https://github.com/ansible/pylibssh/issues/360