Some ansible tasks take upwards of an hour to complete (seemingly at random)

[atammy-narmi]

Summary

Ansible is being used in deployment pipelines where I am seeing some trivial tasks which randomly take an hour to complete. These tasks aren't the same one every time either. An example task

- name: Update the worker start script
  template:
    src: var/app/shared/bin/worker.j2
    dest: '{{ app_bin_directory }}/worker'
    owner: '{{ app_deploy_user }}'
    mode: '0755'

Which resulted in an hour of runtime for that task

TASK [ansistrano-deploy : Update the web config script] ************************
ok: [i-02872c92d456728a8]
ok: [i-0f4ebc59cef1559d5]
Wednesday 06 September 2023  20:33:20 +0000 (0:00:03.608)       0:02:04.663 *** 

TASK [ansistrano-deploy : Update the worker start script] **********************
ok: [i-0f4ebc59cef1559d5]
ok: [i-02872c92d456728a8]
Wednesday 06 September 2023  21:33:26 +0000 (1:00:06.760)       1:02:11.424 *** 

Another example task

- name: ANSISTRANO | Copy release version into REVISION file
  copy:
    content: "{{ ansistrano_release_version }}"
    dest: "{{ ansistrano_release_path.stdout }}/REVISION"

Which also had a runtime of an hour

TASK [ansistrano-deploy : ANSISTRANO | Unarchive | Delete archived file] *******
changed: [i-0d7054200d4c375b6]
Wednesday 06 September 2023  21:38:22 +0000 (0:00:04.055)       0:02:15.556 *** 

TASK [ansistrano-deploy : ANSISTRANO | Copy release version into REVISION file] ***
changed: [i-0d7054200d4c375b6]
Wednesday 06 September 2023  22:38:26 +0000 (1:00:04.259)       1:02:19.815 *** 

We have ansible_aws_ssm_timeout: 3600 so I imagine that this is the reason why its 'finishing' at an hour, but am not sure why its taking so long for these simple tasks.

Issue Type

Bug Report

Component Name

aws_ssm

Ansible Version

$ ansible --version
ansible [core 2.15.2]
  config file = /Users/abhinav/GIT/infrastructure/ansible.cfg
  configured module search path = ['/Users/abhinav/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/abhinav/GIT/infrastructure/.venv/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/abhinav/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/abhinav/GIT/infrastructure/.venv/bin/ansible
  python version = 3.9.10 (main, Jul 28 2023, 17:46:54) [Clang 14.0.3 (clang-1403.0.22.14.1)] (/Users/abhinav/GIT/infrastructure/.venv/bin/python)
  jinja version = 3.1.2
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
# /Users/abhinav/GIT/infrastructure/.venv/lib/python3.9/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    6.2.0
ansible.netcommon             5.1.2
ansible.posix                 1.5.4
ansible.utils                 2.10.3
ansible.windows               1.14.0
arista.eos                    6.0.1
awx.awx                       22.5.0
azure.azcollection            1.16.0
check_point.mgmt              5.1.1
chocolatey.chocolatey         1.5.1
cisco.aci                     2.6.0
cisco.asa                     4.0.1
cisco.dnac                    6.7.3
cisco.intersight              1.0.27
cisco.ios                     4.6.1
cisco.iosxr                   5.0.3
cisco.ise                     2.5.12
cisco.meraki                  2.15.3
cisco.mso                     2.4.0
cisco.nso                     1.0.3
cisco.nxos                    4.4.0
cisco.ucs                     1.9.0
cloud.common                  2.1.3
cloudscale_ch.cloud           2.3.1
community.aws                 6.1.0
community.azure               2.0.0
community.ciscosmb            1.0.6
community.crypto              2.14.1
community.digitalocean        1.23.0
community.dns                 2.5.7
community.docker              3.4.8
community.fortios             1.0.0
community.general             7.2.0
community.google              1.0.0
community.grafana             1.5.4
community.hashi_vault         5.0.0
community.hrobot              1.8.1
community.libvirt             1.2.0
community.mongodb             1.6.1
community.mysql               3.7.2
community.network             5.0.0
community.okd                 2.3.0
community.postgresql          2.4.2
community.proxysql            1.5.1
community.rabbitmq            1.2.3
community.routeros            2.8.3
community.sap                 1.0.0
community.sap_libs            1.4.1
community.skydive             1.0.0
community.sops                1.6.4
community.vmware              3.8.0
community.windows             1.13.0
community.zabbix              2.1.0
containers.podman             1.10.2
cyberark.conjur               1.2.0
cyberark.pas                  1.0.19
dellemc.enterprise_sonic      2.2.0
dellemc.openmanage            7.6.1
dellemc.powerflex             1.7.0
dellemc.unity                 1.7.0
f5networks.f5_modules         1.25.0
fortinet.fortimanager         2.2.0
fortinet.fortios              2.3.0
frr.frr                       2.0.2
gluster.gluster               1.0.2
google.cloud                  1.2.0
grafana.grafana               2.1.4
hetzner.hcloud                1.16.0
hpe.nimble                    1.1.4
ibm.qradar                    2.1.0
ibm.spectrum_virtualize       1.12.0
infinidat.infinibox           1.3.12
infoblox.nios_modules         1.5.0
inspur.ispim                  1.3.0
inspur.sm                     2.3.0
junipernetworks.junos         5.2.0
kubernetes.core               2.4.0
lowlydba.sqlserver            2.0.0
microsoft.ad                  1.2.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.22.0
netapp.elementsw              21.7.0
netapp.ontap                  22.7.0
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.4.0
netbox.netbox                 3.13.0
ngine_io.cloudstack           2.3.0
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.3
openstack.cloud               2.1.0
openvswitch.openvswitch       2.1.1
ovirt.ovirt                   3.1.2
purestorage.flasharray        1.20.0
purestorage.flashblade        1.12.1
purestorage.fusion            1.5.0
sensu.sensu_go                1.13.2
servicenow.servicenow         1.0.6
splunk.es                     2.1.0
t_systems_mms.icinga_director 1.33.1
theforeman.foreman            3.12.0
vmware.vmware_rest            2.3.1
vultr.cloud                   1.8.0
vyos.vyos                     4.1.0
wti.remote                    1.0.5

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Ignoring invalid distribution -nsible (/Users/abhinav/GIT/infrastructure/.venv/lib/python3.9/site-packages)
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.28.17
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/abhinav/GIT/infrastructure/.venv/lib/python3.9/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.31.17
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/abhinav/GIT/infrastructure/.venv/lib/python3.9/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
CALLBACKS_ENABLED(/Users/abhinav/GIT/infrastructure/ansible.cfg) = ['profile_tasks']
CONFIG_FILE() = /Users/abhinav/GIT/infrastructure/ansible.cfg
DEFAULT_HOST_LIST(/Users/abhinav/GIT/infrastructure/ansible.cfg) = ['/Users/abhinav/GIT/infrastructure/configuration/inventory']
DEFAULT_MANAGED_STR(/Users/abhinav/GIT/infrastructure/ansible.cfg) = ansible generated: do not edit directly
DEFAULT_ROLES_PATH(/Users/abhinav/GIT/infrastructure/ansible.cfg) = ['/Users/abhinav/GIT/infrastructure/configuration/roles-vendor', '/Users/abhi>
DEFAULT_TIMEOUT(/Users/abhinav/GIT/infrastructure/ansible.cfg) = 30
DEPRECATION_WARNINGS(/Users/abhinav/GIT/infrastructure/ansible.cfg) = False
DISPLAY_SKIPPED_HOSTS(/Users/abhinav/GIT/infrastructure/ansible.cfg) = False
EDITOR(env: EDITOR) = emacs --no-window-system --no-x-resources --no-init-file
HOST_KEY_CHECKING(/Users/abhinav/GIT/infrastructure/ansible.cfg) = False
PAGER(env: PAGER) = less
STRING_CONVERSION_ACTION(/Users/abhinav/GIT/infrastructure/ansible.cfg) = error

OS / Environment

Mac OS - M2 - 13.4.1

Steps to Reproduce

Run the above tasks using an ansible-playbook.

Expected Results

Task completes extremely fast.

Actual Results

Task takes over an hour to run.

Code of Conduct

• [X] I agree to follow the Ansible Code of Conduct