vmware_guest_custom_attributes fails to apply attributes for few hosts after the existing ones were deleted

[tofkentof]

SUMMARY

We are trying to deploy attributes on our vCenter with Ansible. When attributes have been applied, deleting them from vCenter and reapplying them with Ansible fails for some VMs but not all of them. Ansible throws an error about attribute already existing. Running the playbook again works fine. Also the issue never occurs if the attributes already exist prior to running the playbook.

Creating the attributes manually in vCenter after deleting them (also manually) works fine.

ISSUE TYPE

• Bug Report

COMPONENT NAME

vmware_guest_custom_attributes

ANSIBLE VERSION

ansible [core 2.11.4] 
  config file = /etc/ansible/xxx/ansible.cfg
  configured module search path = ['/home/admin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/admin/.local/lib/python3.6/site-packages/ansible
  ansible collection location = /etc/ansible/xxx
  executable location = /home/admin/.local/bin/ansible
  python version = 3.6.8 (default, Nov 16 2020, 16:55:22) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
  jinja version = 3.0.1
  libyaml = True

COLLECTION VERSION

Collection       Version
---------------- -------
community.vmware 1.13.0

CONFIGURATION

ANSIBLE_NOCOWS(/etc/ansible/xxx/ansible.cfg) = True
COLLECTIONS_PATHS(/etc/ansible/xxx/ansible.cfg) = ['/etc/ansible/xxx']
DEFAULT_LOG_PATH(/etc/ansible/xxx/ansible.cfg) = /etc/ansible/ansible.log
DEFAULT_ROLES_PATH(/etc/ansible/xxx/ansible.cfg) = ['/etc/ansible/xxx/roles']
DEFAULT_STDOUT_CALLBACK(/etc/ansible/xxx/ansible.cfg) = yaml
DEFAULT_TIMEOUT(/etc/ansible/xxx/ansible.cfg) = 30
DEPRECATION_WARNINGS(/etc/ansible/xxx/ansible.cfg) = False
HOST_KEY_CHECKING(/etc/ansible/xxx/ansible.cfg) = False

OS / ENVIRONMENT

CentOS Linux release 7.9.2009 Virtual Machine on vSphere/ESXi 6.7.0.51000

STEPS TO REPRODUCE

Inventory and playbook below have been sanitised.

Inventory (simplified)

all:
  children:
    groupC:
      hosts:
        hostC:
          vm_attribute_value3: "ModelC"
    groupD:
      hosts:
        hostD:
          vm_attribute_value3: "ModelD"
    groupE:
      hosts:
        hostE1:
          vm_attribute_value3: "ModelE"
        hostE2:
          vm_attribute_value3: "ModelE"
    groupB:
      hosts:
        hostB:
          vm_attribute_value3: "ModelB"
    groupF:
      hosts:
        hostF:
          vm_attribute_value3: "ModelF"
    groupA:
      hosts:
        hostA:
          vm_attribute_value3: "ModelA"

Playbook

- name: Add VMWare identification
  hosts: all:!localhost
  gather_facts: no
  vars:
    ansible_python_interpreter: /usr/bin/python3 #force Python3 to be used
  tasks:
    - name: Add vCenter attributes
      vmware_guest_custom_attributes:
        hostname: "{{ vmware_orchestration_login['hostname'] }}"
        username: "{{ vmware_attributes_username }}"
        password: "{{ vmware_attributes_password }}"
        validate_certs: yes
        name: "{{ hostvars[item.0].inventory_hostname }}"
        folder: "{{ item.1.folder }}"
        state: present
        attributes:
          - name: "attribute_name1"
            value: "{{ hostvars[item.0].inventory_hostname }}"
          - name: "attribute_name2"
            value: "attribute_value2"
          - name: "attribute_name3"
            value: "{{ hostvars[item.0].vm_attribute_value3 }}"
      delegate_to: localhost
      with_together:
        - "{{ inventory_hostname }}"
        - "{{ vmware_orchestration_guests }}"
  tags:
    - vcenter_attributes

EXPECTED RESULTS

Attributes to be applied successfully

ACTUAL RESULTS

Module fails for some hosts but not all of them. The result is not consistent. Different hosts fail or succeed at each execution of the playbook.

The full traceback is:
Traceback (most recent call last):
  File "/home/admin/.ansible/tmp/ansible-tmp-1660555692.6735425-21330-168540856172829/AnsiballZ_vmware_guest_custom_attributes.py", line 100, in <module>
    _ansiballz_main()
  File "/home/admin/.ansible/tmp/ansible-tmp-1660555692.6735425-21330-168540856172829/AnsiballZ_vmware_guest_custom_attributes.py", line 92, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/admin/.ansible/tmp/ansible-tmp-1660555692.6735425-21330-168540856172829/AnsiballZ_vmware_guest_custom_attributes.py", line 41, in invoke_module
    run_name='__main__', alter_sys=True)
  File "/usr/lib64/python3.6/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_vmware_guest_custom_attributes_payload_wfyrtiex/ansible_vmware_guest_custom_attributes_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_custom_attributes.py", line 264, in <module>
  File "/tmp/ansible_vmware_guest_custom_attributes_payload_wfyrtiex/ansible_vmware_guest_custom_attributes_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_custom_attributes.py", line 252, in main
  File "/tmp/ansible_vmware_guest_custom_attributes_payload_wfyrtiex/ansible_vmware_guest_custom_attributes_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_custom_attributes.py", line 189, in set_custom_field
  File "/usr/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 706, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 512, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/python3.6/site-packages/pyVmomi/SoapAdapter.py", line 1397, in InvokeMethod
    raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.DuplicateName: (vim.fault.DuplicateName) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = "The name 'attribute_name1' already exists.",
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   name = 'attribute_name1',
   object = 'vim.CustomFieldsManager:CustomFieldsManager'
}
failed: [hostE1 -> localhost]
  module_stderr: |-
    Traceback (most recent call last):
      File "/home/admin/.ansible/tmp/ansible-tmp-1660555692.6735425-21330-168540856172829/AnsiballZ_vmware_guest_custom_attributes.py", line 100, in <module>
        _ansiballz_main()
      File "/home/admin/.ansible/tmp/ansible-tmp-1660555692.6735425-21330-168540856172829/AnsiballZ_vmware_guest_custom_attributes.py", line 92, in _ansiballz_main
        invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
      File "/home/admin/.ansible/tmp/ansible-tmp-1660555692.6735425-21330-168540856172829/AnsiballZ_vmware_guest_custom_attributes.py", line 41, in invoke_module
        run_name='__main__', alter_sys=True)
      File "/usr/lib64/python3.6/runpy.py", line 205, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code
        mod_name, mod_spec, pkg_name, script_name)
      File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/tmp/ansible_vmware_guest_custom_attributes_payload_wfyrtiex/ansible_vmware_guest_custom_attributes_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_custom_attributes.py", line 264, in <module>
      File "/tmp/ansible_vmware_guest_custom_attributes_payload_wfyrtiex/ansible_vmware_guest_custom_attributes_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_custom_attributes.py", line 252, in main
      File "/tmp/ansible_vmware_guest_custom_attributes_payload_wfyrtiex/ansible_vmware_guest_custom_attributes_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_guest_custom_attributes.py", line 189, in set_custom_field
      File "/usr/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 706, in <lambda>
        self.f(*(self.args + (obj,) + args), **kwargs)
      File "/usr/lib/python3.6/site-packages/pyVmomi/VmomiSupport.py", line 512, in _InvokeMethod
        return self._stub.InvokeMethod(self, info, args)
      File "/usr/lib/python3.6/site-packages/pyVmomi/SoapAdapter.py", line 1397, in InvokeMethod
        raise obj # pylint: disable-msg=E0702
    pyVmomi.VmomiSupport.DuplicateName: (vim.fault.DuplicateName) {
       dynamicType = <unset>,
       dynamicProperty = (vmodl.DynamicProperty) [],
       msg = "The name 'attribute_name1' already exists.",
       faultCause = <unset>,
       faultMessage = (vmodl.LocalizableMessage) [],
       name = 'attribute_name1',
       object = 'vim.CustomFieldsManager:CustomFieldsManager'
    }
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1

[tofkentof]

I was able to reproduce the problem on another vCenter using different hosts.

This vCenter already had attributes defined, like with our initial environment, before running the playbook. The same error message appeared msg= "The name 'xxx' already exists" (xxx being a new attribute we want to create).

It shows that the problem occurs as long as there are already attributes defined whatever they are.

[tofkentof]

This issue was that the play tried to create attributes for all hosts at the same time, creating some fight between them. Forcing it to process one host at a time by adding "serial: 1" at the play level resolved the problem.

https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html#setting-the-batch-size-with-serial