mitogen icon indicating copy to clipboard operation
mitogen copied to clipboard
verified publisher icon mitogen-hq

HTTP Lookups using requests crash on Ansible Core 2.18 on macOS

Open Agalin opened this issue 11 months ago • 4 comments

Looks like Ansible Core 2.18 has messed something with how lookups are done. A single reqests.get call is enough to kill the thread on macOS (Linux seems to be alright). Everything works fine on 2.17.

Note that 2.18 seems to mess with interpreter selection in general, I had to update/remove shebangs in some modules to make it usable even without mitogen. I wouldn't be surprised if that's related. Lookup is called in a playbook targetting localhost - in the past it was running with ansible_python_interpreter: "{{ ansible_playbook_python }}", tried without it and with interpreter_python set to the same value, no difference. OBJC_DISABLE_INITIALIZE_FORK_SAFETY doesn't seem to change anything either.

Note that outside of lookup requests work just fine.

While issue was found usiing Hashivault's lookup plugin, I've been able to reproduce it with a simple requests call after investigating original stacktrace for exact failure location (stacktrace in later section might have slightly mismatched line number for the call, I've removed some docstrings).

from ansible.plugins.lookup import LookupBase
from requests import get

class LookupModule(LookupBase):
    def run(self, terms, variables=None, **kwargs):
        get("https://<urll>"+terms[0])
        return []

Environment

  • Ansible version: 11.0, 11.1
  • Ansible Core version: 2.18.0, 2.18.1
  • OS: macOS 15.2
  • Python: tested 3.11 and 3.13
  • Mitogen version: tried 0.3.18-0.3.20
Ansible Config dump 
CONFIG_FILE() = /Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg
DEFAULT_FILTER_PLUGIN_PATH(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = ['/Users/<user>/Documents/projects/<project name>/ansible/plugins/filter']
DEFAULT_HOST_LIST(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = ['/Users/<user>/Documents/projects/<project name>/ansible/hosts.sh']
DEFAULT_JINJA2_NATIVE(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = True
DEFAULT_LOOKUP_PLUGIN_PATH(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = ['/Users/<user>/Documents/projects/<project name>/ansible/plugins/lookup']
DEFAULT_MODULE_PATH(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = ['/Users/<user>/Documents/projects/<project name>/ansible/plugins/modules']
DEFAULT_ROLES_PATH(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = ['/Users/<user>/Documents/projects/<project name>/ansible/.galaxy_roles']
DEFAULT_STRATEGY(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = mitogen_linear
DEFAULT_TEST_PLUGIN_PATH(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = ['/Users/<user>/Documents/projects/<project name>/ansible/plugins/test']
DEFAULT_VARS_PLUGIN_PATH(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = ['/Users/<user>/Documents/projects/<project name>/ansible/plugins/vars']
DIFF_ALWAYS(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = True
INTERPRETER_PYTHON(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = auto_silent
RETRY_FILES_ENABLED(/Users/<user>/Documents/projects/<project name>/ansible/ansible.cfg) = False

GALAXY_SERVERS:
Additional modules

From Galaxy:

collections:
  - name: ansible.utils
    version: 5.1.2
  - name: community.crypto
    version: 2.22.3

From PyPI:

ansible-modules-hashivault = "^5.0.0"
jmespath = "^1.0.1"
python-gitlab = "^5.3.1"
kubernetes = "^31.0.0"
ara = "*"
  • some in-project overrides for hashivault modules.
Ansiible run logs 
ansible-playbook [core 2.18.1]
  config file = /Users/<user>/Documents/projects/<project>>/ansible/ansible.cfg
  configured module search path = ['/Users/<user>/Documents/projects/<project>>/ansible/plugins/modules']
  ansible python module location = /Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/<user>/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/bin/ansible-playbook
  python version = 3.11.11 (main, Dec  3 2024, 17:20:40) [Clang 16.0.0 (clang-1600.0.26.4)] (/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/bin/python)
  jinja version = 3.1.5
  libyaml = True
Using /Users/<user>/Documents/projects/<project>>/ansible/ansible.cfg as config file
host_list declined parsing /Users/<user>/Documents/projects/<project>>/ansible/hosts.sh as it did not pass its verify_file() method
Parsed /Users/<user>/Documents/projects/<project>>/ansible/hosts.sh inventory source with script plugin
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/sys/audit.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/sys/auth.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/sys/mfa/list_methods.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/sys/mfa/list_enforcements.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/sys/mfa/list_methods.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/sys/mfa/list_enforcements.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/sys/secrets.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/sys/ssh_ca.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/ci/gitlab_archived_projects.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/common/policies.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/roles/k8s_roles.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/roles/token_roles.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/common/entities.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/roles/pki_roles.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/roles/pki_roles.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/roles/ssh_roles.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/roles/cert_roles.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/roles/jwt_roles.yml
statically imported: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/roles/jwt_roles.yml
[WARNING]: Skipping callback 'ara_default', unable to load due to: The server dependencies must be installed to record data offline or run the API server.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: site.yml ******************************************************************************************************************************************************************************************************************************************************************
1 plays in site.yml

PLAY [localhost] ********************************************************************************************************************************************************************************************************************************************************************
[top  18811] 11:54:16.706997 D ansible_mitogen.process: inherited open file limits: soft=1048575 hard=(infinity)
[top  18811] 11:54:16.707109 D ansible_mitogen.process: max open files already set to hard limit: 524288
[mux  18823] 11:54:16.719079 D mitogen.service: Pool(5350, size=32, th='MainThread'): initialized
[mux  18823] 11:54:16.721167 D ansible_mitogen.process: Service pool configured: size=32

TASK [configure : current user] *****************************************************************************************************************************************************************************************************************************************************
task path: /Users/<user>/Documents/projects/<project>>/ansible/roles/configure/tasks/common/entities.yml:12
Fatal Python error: Segmentation fault

Current thread 0x00007ff843eca4c0 (most recent call first):
  File "/usr/local/Cellar/[email protected]/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 2659 in proxy_bypass_macosx_sysconf
  File "/usr/local/Cellar/[email protected]/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 2683 in proxy_bypass
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/requests/utils.py", line 816 in should_bypass_proxies
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/requests/utils.py", line 832 in get_environ_proxies
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/requests/sessions.py", line 760 in merge_environment_settings
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/requests/sessions.py", line 579 in request
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/requests/api.py", line 59 in request
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/requests/api.py", line 73 in get
  File "/Users/<user>/Documents/projects/<project>>/ansible/plugins/lookup/proxy_lookup.py", line 39 in run
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/template/__init__.py", line 824 in _lookup
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/jinja2/runtime.py", line 303 in call
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/template/__init__.py", line 335 in call
  File "<template>", line 12 in root
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/template/native_helpers.py", line 159 in ansible_native_concat
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/template/__init__.py", line 959 in do_template
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/template/__init__.py", line 715 in template
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/template/__init__.py", line 742 in template
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/playbook/task.py", line 302 in _post_validate_args
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/playbook/base.py", line 549 in post_validate
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/playbook/task.py", line 294 in post_validate
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/executor/task_executor.py", line 562 in _execute
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/executor/task_executor.py", line 176 in run
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/executor/process/worker.py", line 187 in _run
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/executor/process/worker.py", line 139 in run
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible_mitogen/strategy.py", line 150 in <lambda>
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/mitogen/core.py", line 675 in _profile_hook
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible_mitogen/strategy.py", line 149 in wrap_worker__run
  File "/usr/local/Cellar/[email protected]/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/usr/local/Cellar/[email protected]/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_fork.py", line 71 in _launch
  File "/usr/local/Cellar/[email protected]/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/local/Cellar/[email protected]/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 281 in _Popen
  File "/usr/local/Cellar/[email protected]/3.11.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 121 in start
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/executor/process/worker.py", line 106 in start
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/plugins/strategy/__init__.py", line 415 in _queue_task
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible_mitogen/strategy.py", line 292 in _queue_task
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/plugins/strategy/linear.py", line 195 in run
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible_mitogen/strategy.py", line 322 in <lambda>
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/mitogen/core.py", line 675 in _profile_hook
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible_mitogen/strategy.py", line 321 in run
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/executor/task_queue_manager.py", line 346 in run
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/executor/playbook_executor.py", line 188 in run
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/cli/playbook.py", line 155 in run
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/cli/__init__.py", line 646 in cli_executor
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/lib/python3.11/site-packages/ansible/cli/playbook.py", line 239 in main
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/<project>>-BxCTzgN7-py3.11/bin/ansible-playbook", line 8 in <module>

Extension modules: markupsafe._speedups, yaml._yaml, _cffi_backend, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md (total: 6)
ERROR! A worker was found in a dead state
[mux  18823] 11:54:21.486303 D mitogen: Waker(fd=36/38): disconnecting
[mux  18823] 11:54:21.487847 D mitogen.service: thread mitogen.Pool.5350.4 exiting gracefully
[mux  18823] 11:54:21.488639 D mitogen.service: thread mitogen.Pool.5350.3 exiting gracefully
[mux  18823] 11:54:21.489848 D mitogen.service: thread mitogen.Pool.5350.5 exiting gracefully
[mux  18823] 11:54:21.490707 D mitogen.service: thread mitogen.Pool.5350.6 exiting gracefully
[mux  18823] 11:54:21.491535 D mitogen.service: thread mitogen.Pool.5350.2 exiting gracefully
[mux  18823] 11:54:21.492642 D mitogen: Router(Broker(12d0)): stats: 0 module requests in 0 ms, 0 sent (0 ms minify time), 0 negative responses. Sent 0.0 kb total, 0.0 kb avg.
[mux  18823] 11:54:21.493446 D mitogen.service: thread mitogen.Pool.5350.7 exiting gracefully
[mux  18823] 11:54:21.494270 D mitogen.service: thread mitogen.Pool.5350.0 exiting gracefully
[mux  18823] 11:54:21.495255 D mitogen.service: thread mitogen.Pool.5350.12 exiting gracefully
[mux  18823] 11:54:21.495947 D mitogen.service: thread mitogen.Pool.5350.10 exiting gracefully
[mux  18823] 11:54:21.496611 D mitogen.service: thread mitogen.Pool.5350.9 exiting gracefully
[mux  18823] 11:54:21.497314 D mitogen.service: thread mitogen.Pool.5350.11 exiting gracefully
[mux  18823] 11:54:21.498000 D mitogen.service: thread mitogen.Pool.5350.8 exiting gracefully
[mux  18823] 11:54:21.498667 D mitogen.service: thread mitogen.Pool.5350.1 exiting gracefully
[mux  18823] 11:54:21.499325 D mitogen.service: thread mitogen.Pool.5350.13 exiting gracefully
[mux  18823] 11:54:21.500041 D mitogen.service: thread mitogen.Pool.5350.14 exiting gracefully
[mux  18823] 11:54:21.500778 D mitogen.service: thread mitogen.Pool.5350.20 exiting gracefully
[mux  18823] 11:54:21.501674 D mitogen.service: thread mitogen.Pool.5350.17 exiting gracefully
[mux  18823] 11:54:21.501761 D mitogen.service: thread mitogen.Pool.5350.15 exiting gracefully
[mux  18823] 11:54:21.502394 D mitogen.service: thread mitogen.Pool.5350.16 exiting gracefully
[mux  18823] 11:54:21.503049 D mitogen.service: thread mitogen.Pool.5350.19 exiting gracefully
[mux  18823] 11:54:21.503758 D mitogen.service: thread mitogen.Pool.5350.21 exiting gracefully
[mux  18823] 11:54:21.503840 D mitogen.service: thread mitogen.Pool.5350.23 exiting gracefully
[mux  18823] 11:54:21.504497 D mitogen.service: thread mitogen.Pool.5350.22 exiting gracefully
[mux  18823] 11:54:21.505150 D mitogen.service: thread mitogen.Pool.5350.18 exiting gracefully
[mux  18823] 11:54:21.505764 D mitogen.service: thread mitogen.Pool.5350.24 exiting gracefully
[mux  18823] 11:54:21.505840 D mitogen.service: thread mitogen.Pool.5350.26 exiting gracefully
[mux  18823] 11:54:21.506519 D mitogen.service: thread mitogen.Pool.5350.25 exiting gracefully
[mux  18823] 11:54:21.507495 D mitogen.service: thread mitogen.Pool.5350.27 exiting gracefully
[mux  18823] 11:54:21.508293 D mitogen.service: thread mitogen.Pool.5350.29 exiting gracefully
[mux  18823] 11:54:21.509231 D mitogen.service: thread mitogen.Pool.5350.31 exiting gracefully
[mux  18823] 11:54:21.509324 D mitogen.service: thread mitogen.Pool.5350.30 exiting gracefully
[mux  18823] 11:54:21.510064 D mitogen.service: thread mitogen.Pool.5350.28 exiting gracefully
[mux  18823] 11:54:21.513585 D mitogen.service: FileService().on_shutdown()
[top  18811] 11:54:21.516360 D ansible_mitogen.process: multiplexer 0 PID 18823 exited with return code 0

Agalin avatar Jan 15 '25 12:01 Agalin

I'm not sure this is a Mitogen specific bug. I get crashes across multiple versions of plain Ansible (without Mitogen), and standalone Pythons from uv/tox-uv https://github.com/moreati/mitogen-issue-1219.

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60266]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60266]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state

➜  mitogen-issue-1219 git:(main) uname -a
Darwin kintha 24.2.0 Darwin Kernel Version 24.2.0: Fri Dec  6 19:01:59 PST 2024; root:xnu-11215.61.5~2/RELEASE_ARM64_T6000 arm64
➜  mitogen-issue-1219 git:(main) tox --version
4.23.0 from /Users/alex/.local/share/uv/tools/tox/lib/python3.13/site-packages/tox/__init__.py
registered plugins:
    tox-uv-1.15.0 at /Users/alex/.local/share/uv/tools/tox/lib/python3.13/site-packages/tox_uv/plugin.py with uv==0.4.25

➜  mitogen-issue-1219 tox                                                                                         
py3.11-ansible8-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60235]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60235]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.11-ansible8-linear: exit 1 (0.48 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60232
py3.11-ansible8-linear: FAIL ✖ in 0.48 seconds
py3.11-ansible9-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60238]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60238]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.11-ansible9-linear: exit 1 (0.43 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60236
py3.11-ansible9-linear: FAIL ✖ in 0.43 seconds
py3.11-ansible10-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60243]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60243]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.11-ansible10-linear: exit 1 (0.41 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60241
py3.11-ansible10-linear: FAIL ✖ in 0.41 seconds
py3.11-ansible11-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60246]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60246]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.11-ansible11-linear: exit 1 (0.41 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60244
py3.11-ansible11-linear: FAIL ✖ in 0.41 seconds
py3.12-ansible8-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60250]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60250]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.12-ansible8-linear: exit 1 (0.47 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60247
py3.12-ansible8-linear: FAIL ✖ in 0.48 seconds
py3.12-ansible9-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60253]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60253]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.12-ansible9-linear: exit 1 (0.42 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60251
py3.12-ansible9-linear: FAIL ✖ in 0.43 seconds
py3.12-ansible10-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60256]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60256]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.12-ansible10-linear: exit 1 (0.42 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60254
py3.12-ansible10-linear: FAIL ✖ in 0.42 seconds
py3.12-ansible11-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60259]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60259]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.12-ansible11-linear: exit 1 (0.42 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60257
py3.12-ansible11-linear: FAIL ✖ in 0.43 seconds
py3.13-ansible8-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60263]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60263]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.13-ansible8-linear: exit 1 (0.46 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60260
py3.13-ansible8-linear: FAIL ✖ in 0.46 seconds
py3.13-ansible9-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60266]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60266]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.13-ansible9-linear: exit 1 (0.44 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60264
py3.13-ansible9-linear: FAIL ✖ in 0.44 seconds
py3.13-ansible10-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60269]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60269]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.13-ansible10-linear: exit 1 (0.43 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60267
py3.13-ansible10-linear: FAIL ✖ in 0.44 seconds
py3.13-ansible11-linear: commands[0]> ansible-playbook playbook.yml

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
objc[60272]: +[NSNumber initialize] may have been in progress in another thread when fork() was called.
objc[60272]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state
py3.13-ansible11-linear: exit 1 (0.43 seconds) /Users/alex/src/mitogen-issue-1219> ansible-playbook playbook.yml pid=60270
  py3.11-ansible8-linear: FAIL code 1 (0.48=setup[0.00]+cmd[0.48] seconds)
  py3.11-ansible9-linear: FAIL code 1 (0.43=setup[0.00]+cmd[0.43] seconds)
  py3.11-ansible10-linear: FAIL code 1 (0.41=setup[0.00]+cmd[0.41] seconds)
  py3.11-ansible11-linear: FAIL code 1 (0.41=setup[0.00]+cmd[0.41] seconds)
  py3.12-ansible8-linear: FAIL code 1 (0.48=setup[0.00]+cmd[0.47] seconds)
  py3.12-ansible9-linear: FAIL code 1 (0.43=setup[0.00]+cmd[0.42] seconds)
  py3.12-ansible10-linear: FAIL code 1 (0.42=setup[0.00]+cmd[0.42] seconds)
  py3.12-ansible11-linear: FAIL code 1 (0.43=setup[0.00]+cmd[0.42] seconds)
  py3.13-ansible8-linear: FAIL code 1 (0.46=setup[0.00]+cmd[0.46] seconds)
  py3.13-ansible9-linear: FAIL code 1 (0.44=setup[0.00]+cmd[0.44] seconds)
  py3.13-ansible10-linear: FAIL code 1 (0.44=setup[0.00]+cmd[0.43] seconds)
  py3.13-ansible11-linear: FAIL code 1 (0.43=setup[0.00]+cmd[0.43] seconds)
  evaluation failed :( (5.31 seconds)

I also see it with Python 3.9 (as installed by Apple dev tools) and Ansible 8.7

➜  mitogen-issue-1219 v39/bin/pip install ansible requests
...
Successfully installed MarkupSafe-3.0.2 PyYAML-6.0.2 ansible-8.7.0 ansible-core-2.15.13 certifi-2024.12.14 cffi-1.17.1 charset-normalizer-3.4.1 cryptography-44.0.0 idna-3.10 importlib-resources-5.0.7 jinja2-3.1.5 packaging-24.2 pycparser-2.22 requests-2.32.3 resolvelib-1.0.1 urllib3-2.3.0
➜  mitogen-issue-1219 ANSIBLE_LOOKUP_PLUGINS=plugins/lookup v39/bin/ansible-playbook -i inventory.yml playbook.yml 

PLAY [Use requests_example lookup] ********************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************
/Users/alex/src/mitogen-issue-1219/v39/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
objc[60086]: +[NSMutableString initialize] may have been in progress in another thread when fork() was called.
objc[60086]: +[NSMutableString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
ERROR! A worker was found in a dead state

moreati avatar Jan 16 '25 15:01 moreati

I agree there are wider problems with Mitogen on macOS, particularly wrt fork(). See #941, #1112, #1185. I've also previously chased strange bugs in CI on macOS that I can't remember right now - particularly after a new Ansible version or Python.

Since Python 3.8 multiprocessing defaults to spawn on macOS. fork() is considered unsafe in the presence of threads and IIRC requests does something that results in a thread spawned by libc.

  • https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
  • https://www.wefearchange.org/2018/11/forkmacos.rst.html <- I don't necessarily endorse that title

moreati avatar Jan 16 '25 16:01 moreati

Those fails are usually solvable by exporting OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES env variable, sad truth is that Ansible doesn't work well with macOS at times. It's quite likely it would fix non-mitogen cases in your reproduction.

In this case it doesn't seem to help unfortunately.

Agalin avatar Jan 16 '25 16:01 Agalin

Have the same issue with ansible-core 2.17 and 2.18, tested on Python 3.12 and 3.13 with MacOS 14.4.1 The OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES used to work for me in the past, but right now it doesn't make a difference any more.

mateusz-kleinert avatar Oct 06 '25 13:10 mateusz-kleinert