macOS 11 & 12: connections fail trying to execv non-existent path

[dwt]

• Which version of Ansible are you running:

  ansible 2.10.4
  config file = /Users/dwt/Code/Projekte/sntl/provisioning/ansible/ansible.cfg
  configured module search path = ['/Users/dwt/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/dwt/.virtualenvs/sntl-provisioning/lib/python3.9/site-packages/ansible
  executable location = /Users/dwt/.virtualenvs/sntl-provisioning/bin/ansible
  python version = 3.9.1 (default, Jan  4 2021, 20:49:26) [Clang 12.0.0 (clang-1200.0.32.28)]
  

• Is your version of Ansible patched in any way? no

• Are you running with any custom modules, or module_utils loaded? yes, self written modules to extract local facts

• Have you tried the latest master version from Git? yes

• Mention your host and target OS and versions: host: macos 11.1, target: cents 7.9

• Mention your host and target Python versions host: 2.7 and 3.9 (both tested)

• If reporting a crash or hang in Ansible, please rerun with -vvv and include 200 lines of output around the point of the error, along with a full copy of any traceback or error text in the log: ansible.log

• If reporting any kind of problem with Ansible, please include the Ansible version along with output of "ansible-config dump --only-changed".

  ANSIBLE_PIPELINING(/path/to/ansible/ansible.cfg) = True
  ANSIBLE_SSH_CONTROL_PATH(/path/to/ansible/ansible.cfg) = %(directory)s/%%C
  CACHE_PLUGIN(/path/to/ansible/ansible.cfg) = jsonfile
  CACHE_PLUGIN_CONNECTION(/path/to/ansible/ansible.cfg) = ./.fact_cache/
  CACHE_PLUGIN_TIMEOUT(/path/to/ansible/ansible.cfg) = 86400
  DEFAULT_FORKS(/path/to/ansible/ansible.cfg) = 500
  DEFAULT_GATHERING(/path/to/ansible/ansible.cfg) = smart
  DEFAULT_HASH_BEHAVIOUR(/path/to/ansible/ansible.cfg) = merge
  DEFAULT_HOST_LIST(/path/to/ansible/ansible.cfg) = ['/path/to/ansible/inventory']
  DEFAULT_MANAGED_STR(/path/to/ansible/ansible.cfg) = This file is managed by Ansible. Don't make changes here, they will be overwritten.
  DEFAULT_ROLES_PATH(/path/to/ansible/ansible.cfg) = ['/path/to/ansible/roles', '/path/to/ansible/vendor/icinga2-ansible', '/path/to/ansible/vendor/ansible-icinga-monitoring', '/path/to/ansible/vendor/ansible-elasticsearch']
  DEFAULT_STDOUT_CALLBACK(/path/to/ansible/ansible.cfg) = yaml
  DEFAULT_STRATEGY(/path/to/ansible/ansible.cfg) = mitogen_linear
  DEFAULT_STRATEGY_PLUGIN_PATH(/path/to/ansible/ansible.cfg) = ['/path/to/ansible/vendor/mitogen/ansible_mitogen/plugins/strategy']
  DEFAULT_VAULT_PASSWORD_FILE(/path/to/ansible/ansible.cfg) = /path/to/ansible/scripts/open_the_vault.sh
  TRANSFORM_INVALID_GROUP_CHARS(/path/to/ansible/ansible.cfg) = ignore
  

The error I'm seeing is that a synchronize action fails because it seems to try to execl() a non-existent path on my local machine. The error is contained in the above linked file, but in shortened it is this:

Python(mitogen:[email protected]:29674): execv: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python(mitogen:[email protected]:29674): No such file or directory

This looks a lot like some bootstrap code, but I'm not sure why it tries to connect to the local machine - or am I missing something?

[dw]

This looks like Apple have changed the Python version switcher binary semantics again. Old bug here -- https://github.com/mitogen-hq/mitogen/issues/277

[dw]

Possibly related to commit 518324c37, appears to be related to https://github.com/mitogen-hq/mitogen/pull/715 going by GitHub search

[opoplawski]

I'm seeing the same with the latest 0.2-release branch with ansible 2.9.18.

[opoplawski]

Any workaround? Setting ansible_python_interpreter doesn't help.

$ ls -l /usr/bin/python
lrwxr-xr-x  1 root  wheel  75 Jan  1  2020 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

[dwt]

Is there any research I can do / information I can provide to help this along? I'ts quite the blocker to use mitogen for me. :-(

[PaulSonOfLars]

I am also facing this issue when trying to bake a big-sur VM, from a big-sur machine.

    default: TASK [Gathering Facts] *********************************************************
==> default: [WARNING]: Unhandled error in Python interpreter discovery for host 127.0.0.1:
==> default: EOF on stream; last 100 lines received: MITO000
==> default: Python(mitogen:paullarsen@redacted:87569): execv: /Sys
==> default: tem/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Conte
==> default: nts/MacOS/Python(mitogen:paullarsen@redacted:87569):
==> default: No such file or directory MITO001
    default: fatal: [127.0.0.1]: UNREACHABLE! => changed=false
    default:   msg: |-
    default:     EOF on stream; last 100 lines received:
    default:     MITO000
    default:     MITO001
    default:     Python(mitogen:paullarsen@redacted:87569): execv: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python(mitogen:paullarsen@redacted:87569): No such file or directory
    default:   unreachable: true

Is there any extra data i can provide to help move this forward?

[PaulSonOfLars]

Update: editing parent.py#L1444 and removing the (mitogen:XXX) bits actually fixes this for me.

Another note is that i do specify in my vars that i want python3, so its strange that it defaults to 2.7.

[NahidOulmi]

Update: editing parent.py#L1444 and removing the (mitogen:XXX) bits actually fixes this for me.

Another note is that i do specify in my vars that i want python3, so its strange that it defaults to 2.7.

This workaround fixes it for me as well. Can we add this fix to the main branch ?

[moreati]

Seen also in prototype Integration tests for macOS 11 (#877, build 233 ), and adhoc testing by me on macOS 12. Cause unknown at the moment.

...
test_success (tests.connection_test.FetchFileTest) ... Python(mitogen:[email protected]:5693): execv: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python(mitogen:[email protected]:5693): No such file or directory
ERROR
...
======================================================================
ERROR: test_success (tests.connection_test.FetchFileTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/1/s/tests/ansible/tests/connection_test.py", line 120, in test_success
    self.conn.fetch_file(ifp.name, ofp.name)
  File "/Users/runner/work/1/s/ansible_mitogen/connection.py", line 1016, in fetch_file
    self._connect()
  File "/Users/runner/work/1/s/ansible_mitogen/connection.py", line 839, in _connect
    self._connect_stack(stack)
  File "/Users/runner/work/1/s/ansible_mitogen/connection.py", line 800, in _connect_stack
    raise ansible.errors.AnsibleConnectionFailure(dct['msg'])
ansible.errors.AnsibleConnectionFailure: EOF on stream; last 100 lines received:
MITO000
MITO001

[moreati]

This workaround fixes it for me as well. Can we add this fix to the main branch ?

I'm afraid not, without knowing more about the cause of the bug, and consequences of removing that string.

[moreati]

I'm starting to understand the cause of this. On macOS 11 and 12 Python 2.7 is deprecated, and there is some sort of wrapper binary to print a warning to saying so. The wrapper interacts badly with Mitogen's bootstrapping. My attempts to understand this, and fix it are in #877, be warned I am regularly rebasing and force pushing to that branch, and the code is far from being merge ready.

[moreati]

I think I've fixed this in https://github.com/mitogen-hq/mitogen/pull/882 (https://github.com/moreati/mitogen/tree/macos-11). @dwt please could you try this branch, to confirm it works for you?

[dwt]

ansible 2.10.15
  config file = /pat/to/ansible/ansible.cfg
  configured module search path = ['/path/to/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /path/to/venv/lib/python3.9/site-packages/ansible
  executable location = /path/to/venv/bin/ansible
  python version = 3.9.7 (default, Oct 21 2021, 19:28:55) [Clang 13.0.0 (clang-1300.0.29.3)]

Tested with Python 3.9.7, ansible 2.10.7, and the mitogen branch you pointed to and it seems to work. Nice work!

[moreati]

Thank you, I've merged the branch

[opoplawski]

Any chance that this could be fixed on the 0.2-release branch as well for those of us still on ansible 2.9? Thanks.