mitogen 0.3.3 + ansible 2.12.8+: Broker has exitted

[philfry]

Hi,

I'm experiencing a strange issue when using ansible 2.12.8 and later with mitogen 0.3.3. When running my (quite long running) playbook on more than 8 hosts, mitogen exits on (quite random, like hostname, systemd, service, template, make, …) tasks (but all hosts at the same time) with:

Traceback (most recent call last):
  File "/home/myuser/tmp/ansible/lib/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/home/myuser/tmp/ansible/lib/ansible/executor/task_executor.py", line 605, in _execute
    result = self._handler.run(task_vars=variables)
  File "/home/myuser/playbooks/plugins/strategy/mitogen/ansible_mitogen/mixins.py", line 146, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/home/myuser/tmp/ansible/lib/ansible/plugins/action/normal.py", line 47, in run
    result = merge_hash(result, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
  File "/home/myuser/playbooks/plugins/strategy/mitogen/ansible_mitogen/mixins.py", line 376, in _execute_module
    self._set_temp_file_args(module_args, wrap_async)
  File "/home/myuser/playbooks/plugins/strategy/mitogen/ansible_mitogen/mixins.py", line 355, in _set_temp_file_args
    self._connection.get_good_temp_dir()
  File "/home/myuser/playbooks/plugins/strategy/mitogen/ansible_mitogen/connection.py", line 832, in get_good_temp_dir
    self._connect()
  File "/home/myuser/playbooks/plugins/strategy/mitogen/ansible_mitogen/connection.py", line 854, in _connect
    self._connect_stack(stack)
  File "/home/myuser/playbooks/plugins/strategy/mitogen/ansible_mitogen/connection.py", line 801, in _connect_stack
    dct = mitogen.service.call(
  File "/home/myuser/playbooks/plugins/strategy/mitogen/mitogen/service.py", line 126, in call
    return call_context.call_service(service_name, method_name, **kwargs)
  File "/home/myuser/playbooks/plugins/strategy/mitogen/mitogen/core.py", line 2314, in call_service
    return recv.get().unpickle()
  File "/home/myuser/playbooks/plugins/strategy/mitogen/mitogen/core.py", line 1195, in get
    msg._throw_dead()
  File "/home/myuser/playbooks/plugins/strategy/mitogen/mitogen/core.py", line 935, in _throw_dead
    raise ChannelError(self.data.decode('utf-8', 'replace'))
mitogen.core.ChannelError: Broker has exitted

Running with 8 hosts or less or using ansible 2.12.7 and below works fine. Reducing ansible forks or MITOGEN_POOL_SIZE doesn't help.

I narrowed down the change in ansible that broke the playbook execution to https://github.com/ansible/ansible/commit/45185b03e20cb7a113a3ac7238e4a924ac1846a7 so reverting this commit fixes the problem.

Any ideas of what could be the incompatibility here?