mitogen icon indicating copy to clipboard operation
mitogen copied to clipboard
verified publisher icon mitogen-hq

ansible_mitogen: Interpreter discovery improvements

Open moreati opened this issue 11 months ago • 7 comments

A grab bag of shortcomings, ideas, and a place to disucss them

Finding python to find python

Ansible's native interpreter discovery makes one call to sh, with a list of candidate interepreters from INTERPRETER_PYTHON_FALLBACK. Mitogen turns this into several attempts to call a list of (Mitogen) candidate interpreters. So an O(1) operation becomes O(N).

  • Can the list be shortened and/or the order optimised?
  • Can the O(N) be eliminated?
    • Make an exception to Mitogen's design, call sh with Ansible's snippet instead
    • Devise a shell construction that calls one of the Python interpreters
  • Should the Mitogen list be derived from/enhanced with Ansible's INTERPRETER_PYTHON_FALLBACK list?

https://github.com/mitogen-hq/mitogen/blob/0953a931c0c4cfbfe42a66e2ab2f5ea6f065ddec/ansible_mitogen/mixins.py#L461-L476

/usr/bin/python fallback

Depending on the setting INTERPRETER_PYTHON Ansible may fallback to /usr/bin/python or (I think) throw an error. I'm not convinced Mitogen always respects that setting.

https://github.com/mitogen-hq/mitogen/blob/0953a931c0c4cfbfe42a66e2ab2f5ea6f065ddec/ansible_mitogen/transport_config.py#L146-L149

related

  • #1132

moreati avatar Jan 20 '25 23:01 moreati

auto fallback

Mitogen defaults to 'auto' discovery mode under some circumstances. This may override Ansible's default, which I don't think is desirable.

https://github.com/mitogen-hq/mitogen/blob/fca7578cdf18d6c33e3388058d1581cbba287b92/ansible_mitogen/transport_config.py#L142-L144

Changing this (e.g. to raise an Exception) currently results in atleast one test failure.

--- a/ansible_mitogen/transport_config.py
+++ b/ansible_mitogen/transport_config.py
@@ -140,8 +140,7 @@ def parse_python_path(s, task_vars, action, rediscover_python):
     discovery value in `facts_from_task_vars` like how Ansible handles this.
     """
     if not s:
-        # if python_path doesn't exist, default to `auto` and attempt to discover it
-        s = 'auto'
+        raise ValueError("Expected Python path or discovery mode, got: %r", s)
 
     s = run_interpreter_discovery_if_necessary(s, task_vars, action, rediscover_python)
     if not s:

PLAY [tc-python-path-hostvar via tc-python-path-unset] *************************
META: ran handlers

TASK [include_tasks _raw_params=../_mitogen_only.yml] **************************
task path: /home/runner/work/mitogen/mitogen/tests/ansible/integration/transport_config/python_path.yml:27
Friday 24 January 2025  13:08:09 +0000 (0:00:00.062)       0:00:05.074 ******** 
[task 4605] 13:08:09.542122 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000004
included: /home/runner/work/mitogen/mitogen/tests/ansible/integration/_mitogen_only.yml for tc-python-path-hostvar
META: 
META: 

TASK [mitogen_get_stack ] ******************************************************
task path: /home/runner/work/mitogen/mitogen/tests/ansible/integration/transport_config/python_path.yml:31
Friday 24 January 2025  13:08:09 +0000 (0:00:00.028)       0:00:05.103 ******** 
[task 4607] 13:08:09.571139 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000008
[task 4607] 13:08:09.622428 D ansible_mitogen.mixins: _remove_tmp_path(None)
The full traceback is:
Traceback (most recent call last):
  File "/home/runner/work/mitogen/mitogen/.tox/py27-mode_ansible-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/home/runner/work/mitogen/mitogen/.tox/py27-mode_ansible-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 663, in _execute
    result = self._handler.run(task_vars=variables)
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/mixins.py", line 121, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/plugins/action/mitogen_get_stack.py", line 51, in run
    _, stack = self._connection._build_stack()
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/connection.py", line 785, in _build_stack
    stack = self._stack_from_spec(spec)
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/connection.py", line 762, in _stack_from_spec
    seen_names=seen_names + (spec.inventory_name(),),
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/connection.py", line 765, in _stack_from_spec
    stack += (CONNECTION_METHOD[spec.transport()](spec),)
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/connection.py", line 144, in _connect_ssh
    'python_path': spec.python_path(),
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/transport_config.py", line 723, in python_path
    rediscover_python=rediscover_python)
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/transport_config.py", line 143, in parse_python_path
    raise ValueError("Expected Python path or discovery mode, got: %r", s)
ValueError: (u'Expected Python path or discovery mode, got: %r', None)
fatal: [tc-python-path-hostvar]: FAILED! => 
  msg: Unexpected failure during module execution.
  stdout: ''

-- https://github.com/mitogen-hq/mitogen/actions/runs/12950225330/job/36122554610?pr=1226

refs

  • #1135
  • #1226

moreati avatar Jan 24 '25 13:01 moreati

Devise a shell construction that calls one of the Python interpreters

#!/bin/sh
p=$(for p in python4 python3 python2; do command -v "${p}" 2>/dev/null && break; done;)
if [ "${p}" ]; then
    exec "${p}" "$@"
else
    echo "Not found" 1>&2
    exit 1
fi

moreati avatar Jan 25 '25 08:01 moreati

Should Mitogen know/track/guess

  • newest Python released?
  • range of Python versions supported by running Ansible release? On controller? On targets?
    • officially support vs it will run in practice

moreati avatar Nov 28 '25 09:11 moreati

https://github.com/ansible-community/ansible-build-data covers Ansible >= 2.10, but doesn't appear to have structured Python version information or existence/defaults of config such as INTERPRETER_PYTHON_FALLBACK.

Using https://gist.github.com/moreati/7e03a125b299fc019f28024a526931c2

➜  issue1225 git:(master) ✗ tox | grep "^2"              
2.10.17: ['/usr/bin/python', 'python3.7', 'python3.6', 'python3.5', 'python2.7', 'python2.6', '/usr/libexec/platform-python', '/usr/bin/python3', 'python']
2.10.17: ['/usr/bin/python', 'python3.7', 'python3.6', 'python3.5', 'python2.7', 'python2.6', '/usr/libexec/platform-python', '/usr/bin/python3', 'python']
2.11.12: ['/usr/bin/python', 'python3.9', 'python3.8', 'python3.7', 'python3.6', 'python3.5', 'python2.7', 'python2.6', '/usr/libexec/platform-python', '/usr/bin/python3', 'python']
2.12.10: ['python3.10', 'python3.9', 'python3.8', 'python3.7', 'python3.6', 'python3.5', '/usr/bin/python3', '/usr/libexec/platform-python', 'python2.7', 'python2.6', '/usr/bin/python', 'python']
2.13.13: ['python3.10', 'python3.9', 'python3.8', 'python3.7', 'python3.6', 'python3.5', '/usr/bin/python3', '/usr/libexec/platform-python', 'python2.7', '/usr/bin/python', 'python']
2.14.18: ['python3.11', 'python3.10', 'python3.9', 'python3.8', 'python3.7', 'python3.6', 'python3.5', '/usr/bin/python3', '/usr/libexec/platform-python', 'python2.7', '/usr/bin/python', 'python']
2.15.13: ['python3.11', 'python3.10', 'python3.9', 'python3.8', 'python3.7', 'python3.6', 'python3.5', '/usr/bin/python3', '/usr/libexec/platform-python', 'python2.7', '/usr/bin/python', 'python']
2.16.14: ['python3.12', 'python3.11', 'python3.10', 'python3.9', 'python3.8', 'python3.7', 'python3.6', '/usr/bin/python3', '/usr/libexec/platform-python', 'python2.7', '/usr/bin/python', 'python']
2.17.14: ['python3.12', 'python3.11', 'python3.10', 'python3.9', 'python3.8', 'python3.7', '/usr/bin/python3', 'python3']
2.18.11: ['python3.13', 'python3.12', 'python3.11', 'python3.10', 'python3.9', 'python3.8', '/usr/bin/python3', 'python3']
2.19.4: ['python3.13', 'python3.12', 'python3.11', 'python3.10', 'python3.9', 'python3.8', '/usr/bin/python3', 'python3']
2.20.0: ['python3.14', 'python3.13', 'python3.12', 'python3.11', 'python3.10', 'python3.9', '/usr/bin/python3', 'python3']

moreati avatar Nov 28 '25 10:11 moreati

INTERPRETER_PYTHON_DISTRO_MAP was renamed to _INTERPRETER_PYTHON_DISTRO_MAP in Ansible 7 (ansible-core 2.14, https://github.com/ansible/ansible/pull/78068) and removed in Ansible 12 (anssible-core 2.19, https://github.com/ansible/ansible/pull/84394, https://github.com/ansible/ansible/commit/1bd8882497a7b94b61c9cd63389df2550cecdf72).

moreati avatar Nov 28 '25 10:11 moreati

#!/usr/bin/env ... quirks

  • POSIX only guarantees #!/some/cmd arg1 will work.
  • Linux: #!/some/cmd arg1 arg2 transforms to `['/some/cmd'. 'arg1 arg2']
  • Solaris: #!/some/cmd arg1 arg2 transforms to `['/some/cmd'. 'arg1'] (truncated after whitespace
  • BSDs, macOS, and GNU offer env -S, which splits the first argument received by env into multiple before passing to next in chain.

https://www.youtube.com/watch?v=aoHMiCzqCNw

moreati avatar Dec 01 '25 12:12 moreati

Notes to self:

  • Tests that mention retry are thin on the ground, maybe none at all. Based on

    ➜  mitogen git:(master) ✗ ag 'retries|retry' -l tests/*.py
tests/testlib.py
➜  mitogen git:(master) ✗ ag 'retries|retry' -l tests/ansible 
tests/ansible/integration/transport/kubectl.yml
tests/ansible/integration/async/multiple_items_loop.yml
tests/ansible/integration/async/runner_one_job.yml
tests/ansible/integration/async/runner_new_process.yml
tests/ansible/integration/async/result_binary_producing_json.yml
tests/ansible/integration/async/runner_timeout_then_polling.yml
tests/ansible/integration/async/runner_two_simultaneous_jobs.yml
tests/ansible/ansible.cfg

  • In vanilla Ansible the SSH connection property is called reconnection_retries

  • psrp also has reconnection_retries. local, paramiko_ssh, & winrm don't.

  • Mitogen project contains no mentions of "reconnection_retries"

  • ag of code suggests no pre-existing connection retry feature in core Mitogen

moreati avatar Dec 02 '25 07:12 moreati