AWS S3 module does not work with Mitogen on RedHat-based OSes

[pun-ky]

• Which version of Ansible are you running?

ansible 2.10.7
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.8 (default, Mar 15 2021, 13:10:14) [GCC 10.2.1 20201203]

• Is your version of Ansible patched in any way? - nope

• Are you running with any custom modules, or module_utils loaded? - nope

• Have you tried the latest master version from Git? - yes, and added patch #776

• Mention your host and target OS and versions:

custom image based on https://hub.docker.com/r/cytopia/ansible with AWS CLI.

AWS CLI version:
aws-cli/1.19.47 Python/3.8.8 Linux/5.10.25-linuxkit botocore/1.20.47

• Mention your host and target Python versions:

  • host 3.8.8
  • target Python2 and Python3 (does not work with both)

• Do you have some idea of what the underlying problem may be?

aws_s3 module and Mitogen incompatibility, without Mitogen works fine.

Seems that Mitogen does not play nicely with AWS S3 module on RedHat-based OSes :/

Task

- name: copy some file from S3
  amazon.aws.aws_s3:
    aws_region: "{{ aws_region }}"
    aws_access_key: "{{ aws_access_key_id }}"
    aws_secret_key: "{{ aws_secret_access_key }}"
    mode: get
    bucket: "{{ aws_bucket_resources }}"
    # ...
    overwrite: never

1) CentOS 7 - with Mitogen - error, without works

===========================================

error:

   amazon-ebs.ami: TASK [xxx : copy Java] ************************************************
    amazon-ebs.ami: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: SystemError: Parent module 'ansible_collections.amazon.aws.plugins' not loaded, cannot perform relative import
    amazon-ebs.ami: fatal: [xxx]: FAILED! => {
    amazon-ebs.ami:     "ansible_facts": {
    amazon-ebs.ami:         "discovered_interpreter_python": "/usr/bin/python"
    amazon-ebs.ami:     },
    amazon-ebs.ami:     "changed": false,
    amazon-ebs.ami:     "rc": 1
    amazon-ebs.ami: }
    amazon-ebs.ami:
    amazon-ebs.ami: MSG:
    amazon-ebs.ami:
    amazon-ebs.ami: MODULE FAILURE
    amazon-ebs.ami: See stdout/stderr for the exact error
    amazon-ebs.ami:
    amazon-ebs.ami:
    amazon-ebs.ami: MODULE_STDERR:
    amazon-ebs.ami:
    amazon-ebs.ami: Traceback (most recent call last):
    amazon-ebs.ami:   File "master:/usr/lib/python3.8/site-packages/ansible_mitogen/runner.py", line 975, in _run
    amazon-ebs.ami:     self._run_code(code, mod)
    amazon-ebs.ami:   File "master:/usr/lib/python3.8/site-packages/ansible_mitogen/runner.py", line 941, in _run_code
    amazon-ebs.ami:     exec('exec code in vars(mod)')
    amazon-ebs.ami:   File "<string>", line 1, in <module>
    amazon-ebs.ami:   File "master:/root/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/aws_s3.py", line 317, in <module>
    amazon-ebs.ami: SystemError: Parent module 'ansible_collections.amazon.aws.plugins' not loaded, cannot perform relative import
    amazon-ebs.ami:
    amazon-ebs.ami:

=====

2) RHEL8 - without Mitogen works, with Mitogen and even when adding 'vars: mitogen_task_isolation: fork' does not work too :/

amazon-ebs.ami: TASK [xxx : copy Java] ************************************************
amazon-ebs.ami: fatal: [xxx]: FAILED! => {
amazon-ebs.ami:     "ansible_facts": {
amazon-ebs.ami:         "discovered_interpreter_python": "/usr/libexec/platform-python"
amazon-ebs.ami:     },
amazon-ebs.ami:     "changed": false
amazon-ebs.ami: }
amazon-ebs.ami:
amazon-ebs.ami: MSG:
amazon-ebs.ami:
amazon-ebs.ami: Failed to import the required Python library (botocore or boto3) on ip-172-31-13-225.eu-central-1.compute.internal's Python /usr/libexec/platform-python. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter

2b)

Even when adding to Ansible invocation -e ansible_python_interpreter=/usr/bin/python3 ; it reports that Python3 interpreter but same error as above; boto is for sure already installed. Mitogen is causing that boto is not found.

===

Please help! I am blocked to use Mitogen right now. I am forced to use RedHat-based OS. I have tested also Mitogen on Ubuntu 18.04, it worked fine ;) so the case is RedHat+AWS S3 module+Mitogen incompatibility :/

PS. I have made it working once on Centos 8 by specifying ansible_python_interpreter in few places and customizing PIP install especially for Mitogen. But this was just uncomfortable - to add such customizations needed especially for Mitogen :/

[pun-ky]

hmm maybe it is related with #794 , the collection is not loaded?:

edit: or maybe not, adding collections_paths = /root/.ansible/collections:collections not helping :/

[pun-ky]

hmm now I am quite sure that on RHEL8 it is the same issue as #664 . During the same Ansible run I am installing boto3 using pip module then trying to use it in AWS3 module.

[mathewseby]

@pun-ky , Were you able to fix this issue?, I am also facing the same issue.

[krystian-panek-vmltech]

@mathewseby you need to preinstall all needed python dependencies before running Ansible with Mitogen. So you could run at first Ansible without Mitogen to install only Python dependencies and then run everything else as desired.

[krystian-panek-vmltech]

So preinstall libs via cloud-init as you are using AWS

[mathewseby]

@pun-ky , Thank you for the response, I have already installed boto3 in the target machine but still getting the issue.

[krystian-panek-vmltech]

Hmm I needed that one too https://github.com/mitogen-hq/mitogen/issues/776#issuecomment-817741822

[mathewseby]

@krystian-panek-wttech , Did you add S3 module in here, could you please share the config?.

[mathewseby]

@krystian-panek-wttech , could you please share the mitogen zip which you use to fix the issue it will be a great help.

[krystian-panek-vmltech]

https://github.com/pun-ky/mitogen/archive/refs/tags/v0.3.0-776.tar.gz

[mathewseby]

@krystian-panek-wttech , Thank you so much for sharing the zip, I added import ansible_collections.amazon.aws.plugins here and the S3 module is working now.

[krystian-panek-vmltech]

@mathewseby I am glad that I helped 🙂

[p9raffi]

I have also run into this issue with the latest (0.3.2) on Ubuntu 18.04.

[svenedge]

I'm running into this with ansible 4.10/2.11 and mitogen 0.3.2, on a centos 7 server. Tried various workarounds from the discussions in this and other issues, short of patching mitogen itself, but no joy so far.