vmware.vmware_rest
vmware.vmware_rest copied to clipboard
Appliance modules intermittently hang with Ansible 2.16
SUMMARY
After upgrading to Ansible 2.16.2, existing playbooks that run against multiple (4-12) vCenters at the same time in the Inventory will intermittently hang on various appliance related tasks, such as timezone, ntp, or proxy/noproxy.
Reverting to Ansible 2.14 works as expected with no playbook modification, or able to get 2.16 to work with serial: 1
defined, although it is slower and less efficient.
ISSUE TYPE
- Bug Report
COMPONENT NAME
appliance_networking_proxy appliance_networking_noproxy appliance_system_time_timezone Possibly others
ANSIBLE VERSION
ansible [core 2.16.2]
config file = /runner/ansible.cfg
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.11/site-packages/ansible
ansible collection location = /runner/collections
executable location = /usr/local/bin/ansible
python version = 3.11.5 (main, Jan 9 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3.11)
jinja version = 3.1.3
libyaml = True
COLLECTION VERSION
Collection Version
------------------ -------
vmware.vmware_rest 3.0.0
community.vmware 4.0.1
CONFIGURATION
COLLECTIONS_PATHS(/runner/ansible.cfg) = ['/runner/collections']
CONFIG_FILE() = /runner/ansible.cfg
DEFAULT_FILTER_PLUGIN_PATH(/runner/ansible.cfg) = ['/runner/custom_filters']
DEFAULT_ROLES_PATH(/runner/ansible.cfg) = ['/runner/roles']
DEFAULT_STRATEGY_PLUGIN_PATH(/runner/ansible.cfg) = ['/runner/custom_plugins/mitogen-0.3.4/ansible_mitogen/plugins/strategy']
DEFAULT_TIMEOUT(/runner/ansible.cfg) = 40
HOST_KEY_CHECKING(/runner/ansible.cfg) = False
PARAMIKO_LOOK_FOR_KEYS(/runner/ansible.cfg) = False
OS / ENVIRONMENT
Execution environment based on CentOS Stream9
STEPS TO REPRODUCE
- name: VMware vCenter Appliance Configuration
hosts: vCenters
gather_facts: false
tasks:
- name: DNS Servers
# https://github.com/ansible-collections/vmware.vmware_rest/blob/main/docs/vmware.vmware_rest.appliance_networking_dns_servers_module.rst
vmware.vmware_rest.appliance_networking_dns_servers:
vcenter_hostname: "{{ inventory_hostname }}"
vcenter_username: "{{ vcenter_user }}"
vcenter_password: "{{ vcenter_password }}"
vcenter_validate_certs: "{{ vmware_validate_certs }}"
servers: "{{ dns_servers }}"
mode: is_static
delegate_to: localhost
register: result
until: result is not failed
retries: 2
delay: 60
tags:
- dns
- name: Timezone
vmware.vmware_rest.appliance_system_time_timezone:
vcenter_hostname: "{{ inventory_hostname }}"
vcenter_username: "{{ vcenter_user }}"
vcenter_password: "{{ vcenter_password }}"
vcenter_validate_certs: "{{ vmware_validate_certs }}"
name: America/Chicago
delegate_to: localhost
register: result
until: result is not failed
retries: 2
delay: 60
tags:
- timezone
- name: NTP configuration
# https://github.com/ansible-collections/vmware.vmware_rest/blob/main/docs/vmware.vmware_rest.appliance_ntp_module.rst
vmware.vmware_rest.appliance_ntp:
vcenter_hostname: "{{ inventory_hostname }}"
vcenter_username: "{{ vcenter_user }}"
vcenter_password: "{{ vcenter_password }}"
vcenter_validate_certs: "{{ vmware_validate_certs }}"
servers: "{{ ntp_servers }}"
delegate_to: localhost
register: result
until: result is not failed
retries: 2
delay: 60
tags:
- ntp
notify: Restart the ntpd service
- name: Enable NTP time sync
# https://github.com/ansible-collections/vmware.vmware_rest/blob/main/docs/vmware.vmware_rest.appliance_timesync_module.rst
vmware.vmware_rest.appliance_timesync:
vcenter_hostname: "{{ inventory_hostname }}"
vcenter_username: "{{ vcenter_user }}"
vcenter_password: "{{ vcenter_password }}"
vcenter_validate_certs: "{{ vmware_validate_certs }}"
mode: NTP
delegate_to: localhost
register: result
until: result is not failed
retries: 2
delay: 60
tags:
- ntp
- name: Network Proxy - HTTP
# https://github.com/ansible-collections/vmware.vmware_rest/blob/main/docs/vmware.vmware_rest.appliance_networking_proxy_module.rst
vmware.vmware_rest.appliance_networking_proxy:
vcenter_hostname: "{{ inventory_hostname }}"
vcenter_username: "{{ vcenter_user }}"
vcenter_password: "{{ vcenter_password }}"
vcenter_validate_certs: "{{ vmware_validate_certs }}"
enabled: true
server: "{{ http_proxy }}"
port: "{{ http_proxy_port }}"
protocol: http
delegate_to: localhost
when: http_proxy is defined
register: result
until: result is not failed
# retries: 2
# delay: 60
tags:
- proxy
- http
EXPECTED RESULTS
Running against and inventory with 4-12 vCenters, all vCenter 7.0 U3 or 8.0 U2, all tasks should complete successfully and timely.
ACTUAL RESULTS
Tasks that run successfully are a bit slower than Ansible 2.14 versions, sometimes pausing for 5+ seconds on output on the screen, where earlier versions were less than 1 second.
At some point in the playbook, a task will hang after showing OK status for around 4 vCenters and no further output for the remaining vCenters. Sometimes it happens on the second task, sometimes on the 4th or 5th.
When ran in AWX, it hung for 20 hours before the task self-terminated. In the CLI, I've waited more than 15 minutes with no further output once it hangs. No errors shown, just no progress.
In this case, it successfully ran on 7 out of 12 entries before hanging on the 6th task
TASK [NoProxy Config - vCenter 7] **************************************************************************************************************************************************
task path: /runner/VMware/vcenter_configuration.yml:209
Read vars_file 'vars/ssh_settings.yml'
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Read vars_file 'vars/ssh_settings.yml'
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Read vars_file 'vars/ssh_settings.yml'
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Read vars_file 'vars/ssh_settings.yml'
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1705950034.7337685-3235-101214978140258 `" && echo ansible-tmp-1705950034.7337685-3235-101214978140258="` echo /root/.ansible/tmp/ansible-tmp-1705950034.7337685-3235-101214978140258 `" ) && sleep 0'
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1705950034.7419233-3236-55095610903876 `" && echo ansible-tmp-1705950034.7419233-3236-55095610903876="` echo /root/.ansible/tmp/ansible-tmp-1705950034.7419233-3236-55095610903876 `" ) && sleep 0'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1705950034.7701528-3237-215154837948916 `" && echo ansible-tmp-1705950034.7701528-3237-215154837948916="` echo /root/.ansible/tmp/ansible-tmp-1705950034.7701528-3237-215154837948916 `" ) && sleep 0'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1705950034.7901747-3238-127817883802305 `" && echo ansible-tmp-1705950034.7901747-3238-127817883802305="` echo /root/.ansible/tmp/ansible-tmp-1705950034.7901747-3238-127817883802305 `" ) && sleep 0'
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1705950034.8042777-3239-196449567248170 `" && echo ansible-tmp-1705950034.8042777-3239-196449567248170="` echo /root/.ansible/tmp/ansible-tmp-1705950034.8042777-3239-196449567248170 `" ) && sleep 0'
Loading collection cloud.common from /runner/collections/ansible_collections/cloud/common
Using module file /runner/collections/ansible_collections/vmware/vmware_rest/plugins/modules/appliance_networking_noproxy.py
<localhost> PUT /root/.ansible/tmp/ansible-local-19322bq4vxu8/tmpvqpr4t08 TO /root/.ansible/tmp/ansible-tmp-1705950034.7337685-3235-101214978140258/AnsiballZ_appliance_networking_noproxy.py
Using module file /runner/collections/ansible_collections/vmware/vmware_rest/plugins/modules/appliance_networking_noproxy.py
Using module file /runner/collections/ansible_collections/vmware/vmware_rest/plugins/modules/appliance_networking_noproxy.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1705950034.7337685-3235-101214978140258/ /root/.ansible/tmp/ansible-tmp-1705950034.7337685-3235-101214978140258/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> PUT /root/.ansible/tmp/ansible-local-19322bq4vxu8/tmp8j_yoj4_ TO /root/.ansible/tmp/ansible-tmp-1705950034.7901747-3238-127817883802305/AnsiballZ_appliance_networking_noproxy.py
Using module file /runner/collections/ansible_collections/vmware/vmware_rest/plugins/modules/appliance_networking_noproxy.py
Using module file /runner/collections/ansible_collections/vmware/vmware_rest/plugins/modules/appliance_networking_noproxy.py
<localhost> PUT /root/.ansible/tmp/ansible-local-19322bq4vxu8/tmpmeva6hr2 TO /root/.ansible/tmp/ansible-tmp-1705950034.7701528-3237-215154837948916/AnsiballZ_appliance_networking_noproxy.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1705950034.7901747-3238-127817883802305/ /root/.ansible/tmp/ansible-tmp-1705950034.7901747-3238-127817883802305/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> PUT /root/.ansible/tmp/ansible-local-19322bq4vxu8/tmpnwa4b0bc TO /root/.ansible/tmp/ansible-tmp-1705950034.8042777-3239-196449567248170/AnsiballZ_appliance_networking_noproxy.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1705950034.7701528-3237-215154837948916/ /root/.ansible/tmp/ansible-tmp-1705950034.7701528-3237-215154837948916/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> PUT /root/.ansible/tmp/ansible-local-19322bq4vxu8/tmphji1m34t TO /root/.ansible/tmp/ansible-tmp-1705950034.7419233-3236-55095610903876/AnsiballZ_appliance_networking_noproxy.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1705950034.8042777-3239-196449567248170/ /root/.ansible/tmp/ansible-tmp-1705950034.8042777-3239-196449567248170/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1705950034.7419233-3236-55095610903876/ /root/.ansible/tmp/ansible-tmp-1705950034.7419233-3236-55095610903876/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1705950034.7337685-3235-101214978140258/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1705950034.7901747-3238-127817883802305/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1705950034.7419233-3236-55095610903876/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1705950034.8042777-3239-196449567248170/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1705950034.7701528-3237-215154837948916/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1705950034.7337685-3235-101214978140258/ > /dev/null 2>&1 && sleep 0'
Read vars_file 'vars/ssh_settings.yml'
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1705950036.2586815-3324-231465638324340 `" && echo ansible-tmp-1705950036.2586815-3324-231465638324340="` echo /root/.ansible/tmp/ansible-tmp-1705950036.2586815-3324-231465638324340 `" ) && sleep 0'
Using module file /runner/collections/ansible_collections/vmware/vmware_rest/plugins/modules/appliance_networking_noproxy.py
<localhost> PUT /root/.ansible/tmp/ansible-local-19322bq4vxu8/tmpt4ggi3yq TO /root/.ansible/tmp/ansible-tmp-1705950036.2586815-3324-231465638324340/AnsiballZ_appliance_networking_noproxy.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1705950036.2586815-3324-231465638324340/ /root/.ansible/tmp/ansible-tmp-1705950036.2586815-3324-231465638324340/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1705950036.2586815-3324-231465638324340/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1705950034.7701528-3237-215154837948916/ > /dev/null 2>&1 && sleep 0'
ok: [vcenter0001.domain.com -> localhost]
Read vars_file 'vars/ssh_settings.yml'
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
ok: [vcenter112.domain.com -> localhost]
Read vars_file 'vars/ssh_settings.yml'
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
skipping: [vcenter331.domain.local] => {
"changed": false,
"false_condition": "http_proxy is defined",
"skip_reason": "Conditional result was False"
}
Read vars_file 'vars/ssh_settings.yml'
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1705950037.145525-3341-251828492713709 `" && echo ansible-tmp-1705950037.145525-3341-251828492713709="` echo /root/.ansible/tmp/ansible-tmp-1705950037.145525-3341-251828492713709 `" ) && sleep 0'
Using module file /runner/collections/ansible_collections/vmware/vmware_rest/plugins/modules/appliance_networking_noproxy.py
<localhost> PUT /root/.ansible/tmp/ansible-local-19322bq4vxu8/tmpq68xuqfv TO /root/.ansible/tmp/ansible-tmp-1705950037.145525-3341-251828492713709/AnsiballZ_appliance_networking_noproxy.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1705950037.145525-3341-251828492713709/ /root/.ansible/tmp/ansible-tmp-1705950037.145525-3341-251828492713709/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1705950037.145525-3341-251828492713709/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1705950034.7419233-3236-55095610903876/ > /dev/null 2>&1 && sleep 0'
skipping: [vcenter441.domain.local] => {
"changed": false,
"false_condition": "http_proxy is defined",
"skip_reason": "Conditional result was False"
}
ok: [vcenter111.domain.com -> localhost]
Read vars_file 'vars/ssh_settings.yml'
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
Trying secret ScriptVaultSecret(filename='/runner/files/vault/vault_script.sh') for vault_id=default
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1705950037.4776156-3356-7675987435922 `" && echo ansible-tmp-1705950037.4776156-3356-7675987435922="` echo /root/.ansible/tmp/ansible-tmp-1705950037.4776156-3356-7675987435922 `" ) && sleep 0'
Using module file /runner/collections/ansible_collections/vmware/vmware_rest/plugins/modules/appliance_networking_noproxy.py
<localhost> PUT /root/.ansible/tmp/ansible-local-19322bq4vxu8/tmpvi5zhy1r TO /root/.ansible/tmp/ansible-tmp-1705950037.4776156-3356-7675987435922/AnsiballZ_appliance_networking_noproxy.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1705950037.4776156-3356-7675987435922/ /root/.ansible/tmp/ansible-tmp-1705950037.4776156-3356-7675987435922/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3.11 /root/.ansible/tmp/ansible-tmp-1705950037.4776156-3356-7675987435922/AnsiballZ_appliance_networking_noproxy.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1705950034.8042777-3239-196449567248170/ > /dev/null 2>&1 && sleep 0'
ok: [vcenter222.domain.com -> localhost]
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1705950034.7901747-3238-127817883802305/ > /dev/null 2>&1 && sleep 0'
ok: [vcenter221.domain.com -> localhost]