MOID Lookup issue with Execution Environments

[ndennis-Insight]

SUMMARY

When running a vmware.vmware_rest.datacenter_moid, folder_moid, or resource_pool_moid (haven't tested the other moids) lookup with an Execution Environment I get the same error repeatedly but no error when running it locally with the same collection version.

fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'vmware.vmware_rest.datastore_moid'. Error was a <class 'ansible_collections.cloud.common.plugins.module_utils.turbo.exceptions.EmbeddedModuleUnexpectedFailure'>, original message: Cannot decode plugin answer: b''. Cannot decode plugin answer: b''"}

ISSUE TYPE

• Bug Report

COMPONENT NAME

vmware.vmware_rest

ANSIBLE VERSION

ansible [core 2.13.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ndennis/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ndennis/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/ndennis/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ndennis/.local/bin/ansible
  python version = 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /home/ndennis/.local/lib/python3.8/site-packages/ansible_collections
Collection         Version
------------------ -------
vmware.vmware_rest 2.2.0

# /home/ndennis/.ansible/collections/ansible_collections
Collection         Version
------------------ -------
vmware.vmware_rest 2.2.0

CONFIGURATION

OS / ENVIRONMENT

Ubuntu 20.04 Locally

STEPS TO REPRODUCE

Perform an MOID lookup utilizing an Execution Environment and ansible-navigator

- name: Clone the template VM
      vmware.vmware_rest.vcenter_vm:
        placement:
          datastore: "{{ lookup('vmware.vmware_rest.datastore_moid', '/test/datastore/test_1') }}"
          folder: "{{ lookup('vmware.vmware_rest.folder_moid', '/test/ansible') }}"
          resource_pool: "{{ lookup('vmware.vmware_rest.resource_pool_moid', '/test/host/General/Ansible') }}"
        source: '{{ existing_vms.value[0].vm }}'
        name: test123
        state: clone
      register: vm_output

EXPECTED RESULTS

I expect to pull back the MOID and store it in the variable to use when cloning a VM.

ACTUAL RESULTS

Instead I get the below error and the playbook stops:

Loading collection cloud.common from /usr/share/ansible/collections/ansible_collections/cloud/common
exception during Jinja2 execution: Traceback (most recent call last):
  File "/usr/share/ansible/collections/ansible_collections/cloud/common/plugins/module_utils/turbo/common.py", line 106, in communicate
    result = json.loads(raw_answer.decode())
  File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ansible/template/__init__.py", line 984, in _lookup
    ran = instance.run(loop_terms, variables=self._available_variables, **kwargs)
  File "/usr/share/ansible/collections/ansible_collections/cloud/common/plugins/plugin_utils/turbo/lookup.py", line 58, in run_on_daemon
    return self.execute(terms=terms, variables=variables, **kwargs)
  File "/usr/share/ansible/collections/ansible_collections/cloud/common/plugins/plugin_utils/turbo/lookup.py", line 88, in execute
    (result, errors) = turbo_socket.communicate(content)
  File "/usr/share/ansible/collections/ansible_collections/cloud/common/plugins/module_utils/turbo/common.py", line 109, in communicate
    raise EmbeddedModuleUnexpectedFailure(
ansible_collections.cloud.common.plugins.module_utils.turbo.exceptions.EmbeddedModuleUnexpectedFailure: Cannot decode plugin answer: b''
fatal: [localhost]: FAILED! => {
    "msg": "An unhandled exception occurred while running the lookup plugin 'vmware.vmware_rest.datastore_moid'. Error was a <class 'ansible_collections.cloud.common.plugins.module_utils.turbo.exceptions.EmbeddedModuleUnexpectedFailure'>, original message: Cannot decode plugin answer: b''. Cannot decode plugin answer: b''"
}

[dverbeek84]

Same issue here

[goneri]

I just tried to reproduce the problem without success with up to date EE. But it looks like your EE doesn't include cloud.common. Can you please include the dependency and retry.

[kbreit-insight]

I am working with @ndennis-Insight on the same project and chiming in here since I developed the execution environment. Assuming we're using the same execution environment, here is output from ansible-navigator showing the included Ansible components.

Image: ansible-ee-vmware:0.0.1 (primary) (Information about ansible and ansible collections)
 0│---
 1│ansible:
 2│  collections:
 3│    details:
 4│      ansible.utils: 2.6.1
 5│      cloud.common: 2.1.2
 6│      community.general: 5.3.0
 7│      community.hashi_vault: 3.3.1
 8│      vmware.vmware_rest: 2.2.0
 9│  version:
10│    details: ansible [core 2.13.3]

[goneri]

I just realized my EE is not as up to date as yours.

[kbreit-insight]

We are using a custom built EE for this. But as you'll see, we do have cloud.common installed.

[goneri]

[goneri@t14s-goneri a]$ cat execution-environment.yml 
---
version: 1
dependencies:
  galaxy: requirements.yml
[goneri@t14s-goneri a]$ cat requirements.yml 
---
collections:
  - name: vmware.vmware_rest
[goneri@t14s-goneri a]$ ansible-builder --version
1.0.1
[goneri@t14s-goneri a]$ ansible-builder build
Running command:
  podman build -f context/Containerfile -t ansible-execution-env:latest context
Complete! The build context can be found at: /home/goneri/tmp/a/context

I run ansible-navigator this way:

ansible-navigator exec --eei localhost/ansible-execution-env --mode=stdout --pull-policy=never --pass-environment-variable VMWARE_HOST --pass-environment-variable VMWARE_USER --pass-environment-variable VMWARE_PASSWORD --pass-environment-variable ESXI1_HOSTNAME --pass-environment-variable ESXI1_USERNAME --pass-environment-variable ESXI1_PASSWORD --pass-environment-variable VMWARE_VALIDATE_CERTS -- bash -c 'ls && cd tests/integration/targets/vcenter_vm_scenario1/ && ./runme.sh'

[kbreit-insight]

@goneri Do you think the problem is with how we're invoking it?

[goneri]

Can you share the EE? I can give it a try.

[kbreit-insight]

It's in a private container registry so we can't share the actual EE. However, here are the main files. The key is Red Hat registry images aren't allowed to be published publicly. You should be able to move it to the Quay version without much effort.

execution-environment.yml

---
version: 1

ansible_config: 'ansible.cfg'

build_arg_defaults:
  EE_BASE_IMAGE: 'registry.redhat.io/ansible-automation-platform-22/ee-minimal-rhel8:1.0.0-175'

dependencies:
  galaxy: requirements.yaml
  python: requirements.txt

requirements.txt

aiohttp==3.8.1
hvac==1.0.2

requirements.yaml

collections:
- name: vmware.vmware_rest
  version: 2.2.0
- name: community.general
  version: 5.3.0
- name: ansible.utils
  version: 2.6.1
- name: community.hashi_vault
  version: 3.3.1

ansible.cfg

[defaults]
host_key_checking = False

[goneri]

It works as expected. This time, I used ansible-builder 1.1.0 to build the container. My OS is Fedora 36 and I use podman.

[kbreit-insight]

@ndennis-Insight Can you detail the environment you're using to execute the playbooks in the EE?

[goneri]

I run the test-suite of the collection, this test actually: https://github.com/ansible-collections/vmware.vmware_rest/tree/main/tests/integration/targets/vcenter_vm_scenario1

The full command is in this comment: https://github.com/ansible-collections/vmware.vmware_rest/issues/356#issuecomment-1261332750

[kbreit-insight]

@goneri We can run some test suites. Alternatively, if you're open to it, we can setup some screen sharing to show what we're doing and troubleshoot on the fly.

[goneri]

You can ping me (@goneri) on the VMware channel on Matrix ( https://matrix.to/#/#vmware:ansible.com ) https://docs.ansible.com/ansible/latest/community/communication.html#ansible-community-on-matrix

[davepinkawa]

Encountering this same issue, was a solution or work-around ever located that can be shared here?

[gravesm]

This should be fixed in the latest version.

[devnull-mr]

Hi @gravesm @goneri

this does not seem to be fixed in the latest version. I am having the same issue with a venv currently.

The venv has been setup with these requirements:

ansible-core~=2.14.0
ansible~=7.4.0
aiohttp==3.8.4

as well as the latest ansible module version:

---
collections:
  - name: vmware.vmware_rest
    version: '==2.3.1'

This results in this venv and module version:

ansible [core 2.14.4]
  python version = 3.9.13
  jinja version = 3.1.2
  libyaml = True

Collection         Version
------------------ -------
vmware.vmware_rest 2.3.1  
cloud.common       2.1.3

With that I am getting the exact same errors as already documented above.

[mariolenz]

Could you please test with cloud.common 2.1.4 or later? I'm not 100% sure but it might be fixed there.