mitogen icon indicating copy to clipboard operation
mitogen copied to clipboard
verified publisher icon mitogen-hq

mitogen 0.2.9, network_cli, Ansible 2.8: Warning: could not recover task_vars

Open Yannis100 opened this issue 6 years ago • 7 comments

Hello,

I read that mitogen will not improve playbook time for network_cli connection but should not break the run either, it was working on 0.2.8 but now a fatal error appears on 0.2.9

with mitogen 0.2.8

[WARNING]:
[task 23195] 13:15:30.871256 W ansible_mitogen.connection: could not recover task_vars. This means some connection settings may erroneously be reset to their defaults. Please report a bug if you encounter this message.

with mitogen 0.2.9

fatal: [CHX-TEST-SWITCH]: UNREACHABLE! => {
    "changed": false,
    "msg": "could not recover task_vars. This means some connection settings may erroneously be reset to their defaults. Please report a bug if you encounter this message.",
    "unreachable": true
}

#342

  • Which version of Ansible are you running? 2.8.6
  • Have you tried the latest master version from Git? No
  • Do you have some idea of what the underlying problem may be? https://mitogen.networkgenomics.com/ansible_detailed.html#common-problems has instructions to help figure out the likely cause and how to gather relevant logs. network_cli connection
  • Mention your host and target OS and versions Ubuntu 18.04 WSL, same with a Python-alpine docker image Cisco catalyst 3850
  • Mention your host and target Python versions "none"
  • If reporting a crash or hang in Ansible, please rerun with -vvv and include 200 lines of output around the point of the error, along with a full copy of any traceback or error text in the log.
<10.1.62.10> attempting to start connection
<10.1.62.10> using connection plugin network_cli
<10.1.62.10> local domain socket does not exist, starting it
<10.1.62.10> control socket path is /home/ansermoz/.ansible/pc/3663c2bf22
<10.1.62.10>  23135 1573560495.31245: trying /usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection/__pycache__
<10.1.62.10>  23135 1573560495.31295: trying /usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection
<10.1.62.10>  23135 1573560495.31365: trying /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/__pycache__
<10.1.62.10>  23135 1573560495.31436: trying /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection
<10.1.62.10>  23135 1573560495.33618: Loaded config def from plugin (connection/ssh)
<10.1.62.10>  23135 1573560495.33630: Loading Connection 'ssh' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/ssh.py (searched paths: /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection/__pycache__:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/__pycache__) (found_in_cache=False, class_only=True)
<10.1.62.10>  23140 1573560495.46354: Loaded config def from plugin (connection/network_cli)
<10.1.62.10>  23140 1573560495.46405: Loading Connection 'network_cli' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/network_cli.py (searched paths: /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection/__pycache__:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/__pycache__)
<10.1.62.10>  23140 1573560495.46535: trying /usr/local/lib/python3.6/dist-packages/ansible/plugins/shell
<10.1.62.10>  23140 1573560495.47060: trying /usr/local/lib/python3.6/dist-packages/ansible/plugins/doc_fragments
<10.1.62.10>  23140 1573560495.48363: Loading ModuleDocFragment 'shell_common' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/doc_fragments/shell_common.py
<10.1.62.10>  23140 1573560495.48563: Loaded config def from plugin (shell/sh)
<10.1.62.10>  23140 1573560495.48574: Loading ShellModule 'sh' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/shell/sh.py
<10.1.62.10>  23140 1573560495.48589: Loading ShellModule 'sh' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/shell/sh.py (found_in_cache=True, class_only=False)
<10.1.62.10>  23140 1573560495.48783: Loading Connection 'local' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/local.py (searched paths: /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection/__pycache__:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/__pycache__)
<10.1.62.10>  23140 1573560495.48801: Loading ShellModule 'sh' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/shell/sh.py (found_in_cache=True, class_only=False)
<10.1.62.10>  23140 1573560495.48808: Loading ShellModule 'sh' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/shell/sh.py (found_in_cache=True, class_only=False)
<10.1.62.10>  23140 1573560495.48827: Loading Connection 'ssh' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/ssh.py (searched paths: /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection/__pycache__:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/__pycache__) (found_in_cache=True, class_only=True)
<10.1.62.10>  23140 1573560495.50077: trying /usr/local/lib/python3.6/dist-packages/ansible/plugins/cliconf/__pycache__
<10.1.62.10>  23140 1573560495.50182: trying /usr/local/lib/python3.6/dist-packages/ansible/plugins/cliconf
<10.1.62.10>  23140 1573560495.50727: Loading Cliconf 'ios' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/cliconf/ios.py (searched paths: /usr/local/lib/python3.6/dist-packages/ansible/plugins/cliconf:/usr/local/lib/python3.6/dist-packages/ansible/plugins/cliconf/__pycache__)
<10.1.62.10>  23140 1573560495.52622: Loaded config def from plugin (connection/paramiko_ssh)
<10.1.62.10>  23140 1573560495.52636: Loading Connection 'paramiko_ssh' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/paramiko_ssh.py (searched paths: /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection/__pycache__:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/__pycache__)
<10.1.62.10>  23140 1573560495.52647: Loading ShellModule 'sh' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/shell/sh.py (found_in_cache=True, class_only=False)
<10.1.62.10>  23140 1573560495.52654: Loading ShellModule 'sh' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/shell/sh.py (found_in_cache=True, class_only=False)
<10.1.62.10> <10.1.62.10> ESTABLISH PARAMIKO SSH CONNECTION FOR USER: ansible on PORT 22 TO 10.1.62.10
<10.1.62.10>  23140 1573560496.21538: trying /usr/local/lib/python3.6/dist-packages/ansible/plugins/terminal/__pycache__
<10.1.62.10>  23140 1573560496.21642: trying /usr/local/lib/python3.6/dist-packages/ansible/plugins/terminal
<10.1.62.10>  23140 1573560496.22188: Loading TerminalModule 'ios' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/terminal/ios.py (searched paths: /usr/local/lib/python3.6/dist-packages/ansible/plugins/terminal:/usr/local/lib/python3.6/dist-packages/ansible/plugins/terminal/__pycache__)
<10.1.62.10> connection to remote device started successfully
<10.1.62.10> local domain socket listeners started successfully
<10.1.62.10> loaded cliconf plugin for network_os ios
<10.1.62.10> ssh connection done, setting terminal
<10.1.62.10> loaded terminal plugin for network_os ios
<10.1.62.10> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.1.62.10> firing event: on_open_shell()
<10.1.62.10> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.1.62.10> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.1.62.10> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.1.62.10> ssh connection has completed successfully
<10.1.62.10>  23135 1573560495.31245: trying /usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection/__pycache__
 23135 1573560495.31295: trying /usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection
 23135 1573560495.31365: trying /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/__pycache__
 23135 1573560495.31436: trying /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection
 23135 1573560495.33618: Loaded config def from plugin (connection/ssh)
 23135 1573560495.33630: Loading Connection 'ssh' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/ssh.py (searched paths: /usr/local/lib/python3.6/dist-packages/ansible/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection/__pycache__:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/connection:/usr/local/lib/python3.6/dist-packages/ansible/plugins/connection/__pycache__) (found_in_cache=False, class_only=True)

<10.1.62.10> local domain socket path is /home/ansermoz/.ansible/pc/3663c2bf22
 23134 1573560498.42857: Loading ActionModule 'cli_command' from /usr/local/lib/python3.6/dist-packages/ansible/plugins/action/cli_command.py (searched paths: /usr/local/lib/python3.6/dist-packages/ansible/plugins/action:/usr/local/lib/python3.6/dist-packages/ansible_mitogen/plugins/action) (found_in_cache=True, class_only=True)
 23134 1573560498.46093: starting attempt loop
 23134 1573560498.47791: running the handler
 23134 1573560498.48003: _low_level_execute_command(): starting
 23134 1573560498.48811: _low_level_execute_command(): executing: /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ansermoz/.ansible/tmp/ansible-local-23092oz1ogp0k/ansible-tmp-1573560498.4796762-48870087805966 `" && echo ansible-tmp-1573560498.4796762-48870087805966="` echo /home/ansermoz/.ansible/tmp/ansible-local-23092oz1ogp0k/ansible-tmp-1573560498.4796762-48870087805966 `" ) && sleep 0'
 23134 1573560498.49060: _execute() done
 23134 1573560498.49140: dumping result to json
 23134 1573560498.49188: done dumping result, returning
 23134 1573560498.49312: done running TaskExecutor() for CHX-TEST-SWITCH/TASK: get SNMP config [34e6d77b-5205-44f9-1caa-00000000000c]
 23134 1573560498.49389: sending task result for task 34e6d77b-5205-44f9-1caa-00000000000c
 23134 1573560498.53330: done sending task result for task 34e6d77b-5205-44f9-1caa-00000000000c
fatal: [CHX-TEST-SWITCH]: UNREACHABLE! => {
    "changed": false,
    "msg": "could not recover task_vars. This means some connection settings may erroneously be reset to their defaults. Please report a bug if you encounter this message.",
    "unreachable": true
}
  • If reporting any kind of problem with Ansible, please include the Ansible version along with output of "ansible-config dump --only-changed".

Yannis100 avatar Nov 12 '19 12:11 Yannis100

Any update ?

Vebryn avatar May 25 '20 19:05 Vebryn

I have a few other things on my plate but after will look into this. Will follow what's here to repro: https://github.com/dw/mitogen/issues/342

s1113950 avatar May 25 '20 20:05 s1113950

Discovered I have no way to test this unfortunately :( I don't have access to a GNS3 setup to test against. If there was a way I could connect to something to test I'd be able to debug the issue, but until that happens the ticket will have to stay open until someone else picks it up. Sorry about that.

s1113950 avatar May 26 '20 00:05 s1113950

May I test code ?

Vebryn avatar May 26 '20 06:05 Vebryn

Certainly! Feel free to check out master. We welcome new PRs :)

s1113950 avatar May 26 '20 06:05 s1113950

I'm seeing this as well with 0.2.9 and master. Not a big deal for me at the moment because our network stuff is in a separate repo/ansible.cfg.

opoplawski avatar Jul 08 '20 19:07 opoplawski

This may be fixed by v0.3.21, which includes #1215.

moreati avatar Jan 20 '25 20:01 moreati