network_cli connections failing with "could not recover task_vars"

[craigers521]

Running ansible version 2.9.11 on macOS with python 3.8, with latest mitogen version from master, targeting cisco isr routers and other network_cli based connection devices. All playbooks fail with this error:

<10.119.122.254> attempting to start connection
<10.119.122.254> using connection plugin network_cli
<10.119.130.254> attempting to start connection
<10.119.130.254> using connection plugin network_cli
<10.119.122.254> local domain socket does not exist, starting it
<10.119.122.254> control socket path is /Users/cse015/.ansible/pc/92ada5d152
<10.119.122.254> local domain socket listeners started successfully
<10.119.122.254> loaded cliconf plugin ios from path /Users/cse015/python-virtual-environments/ansible29/lib/python3.8/site-packages/ansible/plugins/cliconf/ios.py for network_os ios
<10.119.122.254>
<10.119.122.254> local domain socket path is /Users/cse015/.ansible/pc/92ada5d152
<10.119.130.254> local domain socket does not exist, starting it
<10.119.130.254> control socket path is /Users/cse015/.ansible/pc/77c5115e9c
<10.119.130.254> local domain socket listeners started successfully
<10.119.130.254> loaded cliconf plugin ios from path /Users/cse015/python-virtual-environments/ansible29/lib/python3.8/site-packages/ansible/plugins/cliconf/ios.py for network_os ios
<10.119.130.254>
<10.119.130.254> local domain socket path is /Users/cse015/.ansible/pc/77c5115e9c
fatal: [10.119.122.254]: UNREACHABLE! => {
    "changed": false,
    "msg": "could not recover task_vars. This means some connection settings may erroneously be reset to their defaults. Please report a bug if you encounter this message.",
    "unreachable": true
}
fatal: [10.119.130.254]: UNREACHABLE! => {
    "changed": false,
    "msg": "could not recover task_vars. This means some connection settings may erroneously be reset to their defaults. Please report a bug if you encounter this message.",
    "unreachable": true

I also tried running with ansible 2.10 and the latest collectionSupport branch from the s1113950 fork and i was met with a different error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/usr/local/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 613, in _execute
    self._connection = self._get_connection(cvars, templar)
  File "/usr/local/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 908, in _get_connection
    connection, plugin_load_context = self._shared_loader_obj.connection_loader.get_with_context(
  File "/Users/cse015/.ansible/plugins/mitogen-dev/ansible_mitogen/strategy.py", line 170, in wrap_connection_loader__get
    return ansible_mitogen.loaders.connection_loader__get(name, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/loader.py", line 826, in get_with_context
    obj.__init__(instance, *args, **kwargs)
  File "/Users/cse015/.ansible/collections/ansible_collections/ansible/netcommon/plugins/connection/network_cli.py", line 324, in __init__
    super(Connection, self).__init__(
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/connection/__init__.py", line 290, in __init__
    self._update_connection_state()
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/connection/__init__.py", line 368, in _update_connection_state
    control_path = ssh._create_control_path(
  File "/Users/cse015/.ansible/plugins/mitogen-dev/ansible_mitogen/plugins/connection/mitogen_ssh.py", line 67, in _create_control_path
    return Connection.vanilla_class._create_control_path(*args, **kwargs)
AttributeError: 'get_with_context_result' object has no attribute '_create_control_path'
fatal: [10.119.130.254]: FAILED! => {
    "msg": "Unexpected failure during module execution.",
    "stdout": ""

[s1113950]

@craigers521 thanks for testing this on the #715 branch. Is there a simple way to reproduce this using another network_cli-based task? I don't have a cisco router available to test with.

[isundaylee]

I ran into this same issue AttributeError: 'get_with_context_result' object has no attribute '_create_control_path' for a vyos_config task. Ansible 2.10.6 with Mitogen 0.3.0rc1.

Not sure if this qualifies as a "simple way to reproduce": but Vyos can be installed on a virtual machine, which can be used to test network_cli tasks against.

[s1113950]

@isundaylee do you have a vagrant file or something of the like to create a vyos installation on a vm? I don't have the time to look into reproducing with Vyos at the moment unfortunately.

[lorephoenix]

Hello I am having a similar issue running Ansible 2.10 using Mitogen 0.30rc1 ansible-galaxy collection list Collection Version

---

ansible.netcommon 1.4.1 ansible.posix 1.1.1 cisco.ios 1.2.1

The issue occur when gathering facts

• name: "main | Gathering Facts!" cisco.ios.ios_facts: gather_subset: - hardware

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 158, in run res = self._execute() File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 613, in _execute self._connection = self._get_connection(cvars, templar) File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 913, in _get_connection ansible_playbook_pid=to_text(os.getppid()) File "/home/deployer/ansible/plugins/mitogen-0.3.0rc1/ansible_mitogen/strategy.py", line 168, in wrap_connection_loader__get return ansible_mitogen.loaders.connection_loader__get(name, *args, **kwargs) File "/usr/lib/python3.6/site-packages/ansible/plugins/loader.py", line 826, in get_with_context obj.init(instance, *args, **kwargs) File "/home/deployer/.ansible/collections/ansible_collections/ansible/netcommon/plugins/connection/network_cli.py", line 332, in init play_context, new_stdin, *args, **kwargs File "/usr/lib/python3.6/site-packages/ansible/plugins/connection/init.py", line 290, in init self._update_connection_state() File "/usr/lib/python3.6/site-packages/ansible/plugins/connection/init.py", line 371, in _update_connection_state self._ansible_playbook_pid File "/home/deployer/ansible/plugins/mitogen-0.3.0rc1/ansible_mitogen/plugins/connection/mitogen_ssh.py", line 67, in _create_control_path return Connection.vanilla_class._create_control_path(*args, **kwargs) AttributeError: 'get_with_context_result' object has no attribute '_create_control_path' fatal: [hostname]: FAILED! => { "msg": "Unexpected failure during module execution.", "stdout": "" }

[MrLucifer11]

Hi,

I have the same issue with ansible 2.10.3 and Mitogen 0.30rc1. Has anyone found a solution please ?

[XuNiLuS]

Hello,

I have the same issue with Ansible 3.4.0 (Based 2.10.13) and Mitogen 0.30rc1. is there a solution?

[arhue]

Same issue with Ansible 2.13.1 and Mikrotik Router OS 7.5.

[markafarrell]

I have been able to reproduce this issue with the following https://github.com/markafarrell/mitogen-issue-766-repro

[RomyxBaps]

I have you find a solution ? I have the problem for a rsync tasks...

[rawat-he]

Hi,

I am facing the same issue. Below is the traceback

The full traceback is:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/usr/local/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 549, in _execute
    self._connection = self._get_connection(cvars, templar, current_connection)
  File "/usr/local/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 945, in _get_connection
    connection, plugin_load_context = self._shared_loader_obj.connection_loader.get_with_context(
  File "/usr/local/lib/python3.10/site-packages/ansible_mitogen/strategy.py", line 125, in wrap_connection_loader__get
    return ansible_mitogen.loaders.connection_loader__get(name, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/ansible/plugins/loader.py", line 904, in get_with_context
    obj.__init__(instance, *args, **kwargs)  # pylint: disable=unnecessary-dunder-call
  File "/data/collections/ansible_collections/ansible/netcommon/plugins/connection/netconf.py", line 232, in __init__
    super(Connection, self).__init__(
  File "/usr/local/lib/python3.10/site-packages/ansible/plugins/connection/__init__.py", line 289, in __init__
    self._update_connection_state()
  File "/usr/local/lib/python3.10/site-packages/ansible/plugins/connection/__init__.py", line 367, in _update_connection_state
    control_path = ssh._create_control_path(
  File "/usr/local/lib/python3.10/site-packages/ansible_mitogen/plugins/connection/mitogen_ssh.py", line 87, in _create_control_path
    return Connection.vanilla_class._create_control_path(*args, **kwargs)
AttributeError: 'get_with_context_result' object has no attribute '_create_control_path'

msg: 'Unexpected failure during module execution: ''get_with_context_result'' object has no attribute ''_create_control_path''

Ansible: 2.14.0 Target OS: Alpine Linux v3.17 Python: 3.10.13 mitogen: 0.3.6

May I know what was the fix?

Thanks