community.network
community.network copied to clipboard
Published
20 hours ago •
ansible-collections
ansible-collections
ICX intermittent failures: command timeout triggered, timeout value is 45 secs
SUMMARY
Intermittently, across multiple switches (using multiple firmware versions, all > 8090/UFI), we run into the following error:
failed: [switch] (item=192.168.1.10) => {"ansible_loop_var": "item", "changed": false, "item": "192.168.1.10", "msg": "command timeout triggered, timeout value is 45 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."}
In recent testing with firmware versions 8095j, we see this issue a lot less. I can run this play 10 times in a row, and it might succeed 80% of the time some days, 20% of the time other days.
ISSUE TYPE
- Bug Report
COMPONENT NAME
icx_config icx_ping
ANSIBLE VERSION
15:27:52 + ansible --version
15:27:52 ansible [core 2.15.1]
15:27:52 config file = /etc/ansible/ansible-nocache.cfg
15:27:52 configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
15:27:52 ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
15:27:52 ansible collection location = /usr/share/ansible/collections
15:27:52 executable location = /usr/local/bin/ansible
15:27:52 python version = 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] (/usr/bin/python3)
15:27:52 jinja version = 3.1.2
15:27:52 libyaml = False
COLLECTION VERSION
15:27:52 + ansible-galaxy collection list
15:27:53
15:27:53 # /usr/local/lib/python3.10/dist-packages/ansible_collections
15:27:53 Collection Version
15:27:53 ----------------------------- -------
15:27:53 amazon.aws 6.1.0
15:27:53 ansible.netcommon 5.1.1
15:27:53 ansible.posix 1.5.4
15:27:53 ansible.utils 2.10.3
15:27:53 ansible.windows 1.14.0
15:27:53 arista.eos 6.0.1
15:27:53 awx.awx 22.3.0
15:27:53 azure.azcollection 1.16.0
15:27:53 check_point.mgmt 5.1.1
15:27:53 chocolatey.chocolatey 1.4.0
15:27:53 cisco.aci 2.6.0
15:27:53 cisco.asa 4.0.1
15:27:53 cisco.dnac 6.7.2
15:27:53 cisco.intersight 1.0.27
15:27:53 cisco.ios 4.6.1
15:27:53 cisco.iosxr 5.0.3
15:27:53 cisco.ise 2.5.12
15:27:53 cisco.meraki 2.15.1
15:27:53 cisco.mso 2.4.0
15:27:53 cisco.nso 1.0.3
15:27:53 cisco.nxos 4.4.0
15:27:53 cisco.ucs 1.8.0
15:27:53 cloud.common 2.1.3
15:27:53 cloudscale_ch.cloud 2.3.1
15:27:53 community.aws 6.0.0
15:27:53 community.azure 2.0.0
15:27:53 community.ciscosmb 1.0.6
15:27:53 community.crypto 2.14.0
15:27:53 community.digitalocean 1.23.0
15:27:53 community.dns 2.5.6
15:27:53 community.docker 3.4.8
15:27:53 community.fortios 1.0.0
15:27:53 community.general 7.1.0
15:27:53 community.google 1.0.0
15:27:53 community.grafana 1.5.4
15:27:53 community.hashi_vault 5.0.0
15:27:53 community.hrobot 1.8.0
15:27:53 community.libvirt 1.2.0
15:27:53 community.mongodb 1.6.0
15:27:53 community.mysql 3.7.2
15:27:53 community.network 5.0.0
15:27:53 community.okd 2.3.0
15:27:53 community.postgresql 2.4.2
15:27:53 community.proxysql 1.5.1
15:27:53 community.rabbitmq 1.2.3
15:27:53 community.routeros 2.8.2
15:27:53 community.sap 1.0.0
15:27:53 community.sap_libs 1.4.1
15:27:53 community.skydive 1.0.0
15:27:53 community.sops 1.6.2
15:27:53 community.vmware 3.7.0
15:27:53 community.windows 1.13.0
15:27:53 community.zabbix 2.0.1
15:27:53 containers.podman 1.10.2
15:27:53 cyberark.conjur 1.2.0
15:27:53 cyberark.pas 1.0.19
15:27:53 dellemc.enterprise_sonic 2.2.0
15:27:53 dellemc.openmanage 7.6.1
15:27:53 dellemc.powerflex 1.6.0
15:27:53 dellemc.unity 1.6.0
15:27:53 f5networks.f5_modules 1.25.0
15:27:53 fortinet.fortimanager 2.2.0
15:27:53 fortinet.fortios 2.3.0
15:27:53 frr.frr 2.0.2
15:27:53 gluster.gluster 1.0.2
15:27:53 google.cloud 1.1.3
15:27:53 grafana.grafana 2.0.0
15:27:53 hetzner.hcloud 1.11.0
15:27:53 hpe.nimble 1.1.4
15:27:53 ibm.qradar 2.1.0
15:27:53 ibm.spectrum_virtualize 1.12.0
15:27:53 infinidat.infinibox 1.3.12
15:27:53 infoblox.nios_modules 1.5.0
15:27:53 inspur.ispim 1.3.0
15:27:53 inspur.sm 2.3.0
15:27:53 junipernetworks.junos 5.1.0
15:27:53 kubernetes.core 2.4.0
15:27:53 lowlydba.sqlserver 2.0.0
15:27:53 microsoft.ad 1.2.0
15:27:53 netapp.aws 21.7.0
15:27:53 netapp.azure 21.10.0
15:27:53 netapp.cloudmanager 21.22.0
15:27:53 netapp.elementsw 21.7.0
15:27:53 netapp.ontap 22.7.0
15:27:53 netapp.storagegrid 21.11.1
15:27:53 netapp.um_info 21.8.0
15:27:53 netapp_eseries.santricity 1.4.0
15:27:53 netbox.netbox 3.13.0
15:27:53 ngine_io.cloudstack 2.3.0
15:27:53 ngine_io.exoscale 1.0.0
15:27:53 ngine_io.vultr 1.1.3
15:27:53 openstack.cloud 2.1.0
15:27:53 openvswitch.openvswitch 2.1.1
15:27:53 ovirt.ovirt 3.1.2
15:27:53 purestorage.flasharray 1.19.1
15:27:53 purestorage.flashblade 1.11.0
15:27:53 purestorage.fusion 1.5.0
15:27:53 sensu.sensu_go 1.13.2
15:27:53 servicenow.servicenow 1.0.6
15:27:53 splunk.es 2.1.0
15:27:53 t_systems_mms.icinga_director 1.33.1
15:27:53 theforeman.foreman 3.11.0
15:27:53 vmware.vmware_rest 2.3.1
15:27:53 vultr.cloud 1.8.0
15:27:53 vyos.vyos 4.1.0
15:27:53 wti.remote 1.0.5
15:27:53
15:27:53 # /usr/share/ansible/collections/ansible_collections
15:27:53 Collection Version
15:27:53 ----------------------------- -------
15:27:53 amazon.aws 6.1.0
15:27:53 ansible.netcommon 5.1.1
15:27:53 ansible.posix 1.5.4
15:27:53 ansible.utils 2.10.3
15:27:53 ansible.windows 2.0.0
15:27:53 chocolatey.chocolatey 1.5.0
15:27:53 cisco.ios 4.6.1
15:27:53 cisco.meraki 2.15.1
15:27:53 commscope.icx 1.0.5
15:27:53 community.aws 6.0.0
15:27:53 community.crypto 2.14.1
15:27:53 community.docker 3.4.8
15:27:53 community.general 7.1.0
15:27:53 community.network 5.0.0
15:27:53 community.windows 2.0.0
15:27:53 fortinet.fortios 2.3.0
CONFIGURATION
15:27:52 + ansible-config dump --only-changed
15:27:52 ANSIBLE_NOCOWS(/etc/ansible/ansible-nocache.cfg) = True
15:27:52 ANSIBLE_PIPELINING(/etc/ansible/ansible-nocache.cfg) = True
15:27:52 COLLECTIONS_PATHS(/etc/ansible/ansible-nocache.cfg) = ['/usr/share/ansible/collections']
15:27:52 CONFIG_FILE() = /etc/ansible/ansible-nocache.cfg
15:27:52 DEFAULT_FORKS(/etc/ansible/ansible-nocache.cfg) = 50
15:27:52 DEFAULT_POLL_INTERVAL(/etc/ansible/ansible-nocache.cfg) = 5
15:27:52 DEFAULT_REMOTE_USER(/etc/ansible/ansible-nocache.cfg) = ansible
15:27:52 DEFAULT_ROLES_PATH(/etc/ansible/ansible-nocache.cfg) = ['/etc/ansible/roles']
15:27:52 DEFAULT_TIMEOUT(/etc/ansible/ansible-nocache.cfg) = 20
15:27:52 DISPLAY_SKIPPED_HOSTS(/etc/ansible/ansible-nocache.cfg) = False
15:27:52 HOST_KEY_CHECKING(/etc/ansible/ansible-nocache.cfg) = False
15:27:52 PERSISTENT_COMMAND_TIMEOUT(/etc/ansible/ansible-nocache.cfg) = 30
15:27:52 PERSISTENT_CONNECT_TIMEOUT(/etc/ansible/ansible-nocache.cfg) = 40
15:27:52 RETRY_FILES_ENABLED(/etc/ansible/ansible-nocache.cfg) = False
OS / ENVIRONMENT
Switch
SSH@switch#sh ver
Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Apr 11 2023 at 00:06:40 labeled as SPR08095j
(33554432 bytes) from Primary SPR08095j.bin (UFI)
SW: Version 08.0.95jT213
Compressed Primary Boot Code size = 786944, Version:10.1.26T215 (spz10126)
Compiled on Tue Nov 29 05:43:15 2022
HW: Stackable ICX7450-48F
Internal USB: Serial #: 99013190710xxxxx
Vendor: ATP Electronics, Total size = 1919 MB
==========================================================================
UNIT 1: SL 1: ICX7450-48F 48-port Management Module
Serial #:CYS330xxxxx
Software Package: ICX7450_L3_SOFT_PACKAGE (LID: eauIIFHtFGH)
Current License: l3-prem
P-ASIC 0: type B548, rev 01 Chip BCM56548_A0
==========================================================================
UNIT 1: SL 2: ICX7400-4X10GF 4-port 40G Module
Serial #:CYV333xxxxx
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8 MB boot flash memory
2 GB code flash memory
2 GB DRAM
STACKID 1 system uptime is 4 day(s) 16 hour(s) 2 minute(s) 52 second(s)
The system started at 21:01:09 Arizona Fri Aug 04 2023
Kernel
15:27:53 + uname -a
15:27:53 Linux 070822a60e46 5.19.0-1028-aws #29~22.04.1-Ubuntu SMP Tue Jun 20 19:12:11 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
OS
15:27:53 + cat /etc/os-release
15:27:53 PRETTY_NAME="Ubuntu 22.04.2 LTS"
15:27:53 NAME="Ubuntu"
15:27:53 VERSION_ID="22.04"
15:27:53 VERSION="22.04.2 LTS (Jammy Jellyfish)"
15:27:53 VERSION_CODENAME=jammy
15:27:53 ID=ubuntu
15:27:53 ID_LIKE=debian
15:27:53 HOME_URL="https://www.ubuntu.com/"
15:27:53 SUPPORT_URL="https://help.ubuntu.com/"
15:27:53 BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
15:27:53 PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
15:27:53 UBUNTU_CODENAME=jammy
apt
15:27:53 + apt list --installed
15:27:53
15:27:53 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
15:27:53
15:27:53 Listing...
15:27:53 adduser/jammy,now 3.118ubuntu5 all [installed]
15:27:53 apt/jammy-updates,now 2.4.9 amd64 [installed]
15:27:53 base-files/jammy-updates,now 12ubuntu4.3 amd64 [installed]
15:27:53 base-passwd/jammy,now 3.5.52build1 amd64 [installed]
15:27:53 bash/jammy,now 5.1-6ubuntu1 amd64 [installed]
15:27:53 binutils-common/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 binutils-x86-64-linux-gnu/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 binutils/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 bsdutils/jammy,now 1:2.37.2-4ubuntu3 amd64 [installed]
15:27:53 ca-certificates/jammy-updates,jammy-security,now 20230311ubuntu0.22.04.1 all [installed,automatic]
15:27:53 coreutils/jammy,now 8.32-4.1ubuntu1 amd64 [installed]
15:27:53 cpp-11/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 cpp/jammy,now 4:11.2.0-1ubuntu1 amd64 [installed,automatic]
15:27:53 curl/jammy-updates,jammy-security,now 7.81.0-1ubuntu1.10 amd64 [installed]
15:27:53 dash/jammy,now 0.5.11+git20210903+057cd650a4ed-3build1 amd64 [installed]
15:27:53 debconf/jammy,now 1.5.79ubuntu1 all [installed]
15:27:53 debianutils/jammy,now 5.5-1ubuntu2 amd64 [installed]
15:27:53 diffutils/jammy,now 1:3.8-0ubuntu2 amd64 [installed]
15:27:53 dpkg/jammy-updates,now 1.21.1ubuntu2.2 amd64 [installed]
15:27:53 e2fsprogs/jammy-updates,jammy-security,now 1.46.5-2ubuntu1.1 amd64 [installed]
15:27:53 findutils/jammy,now 4.8.0-1ubuntu3 amd64 [installed]
15:27:53 gcc-11-base/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 gcc-11/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 gcc-12-base/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed]
15:27:53 gcc/jammy,now 4:11.2.0-1ubuntu1 amd64 [installed]
15:27:53 git-man/jammy-updates,jammy-security,now 1:2.34.1-1ubuntu1.9 all [installed,automatic]
15:27:53 git/jammy-updates,jammy-security,now 1:2.34.1-1ubuntu1.9 amd64 [installed]
15:27:53 gpgv/jammy-updates,jammy-security,now 2.2.27-3ubuntu2.1 amd64 [installed]
15:27:53 grep/jammy,now 3.7-1build1 amd64 [installed]
15:27:53 gzip/jammy-updates,now 1.10-4ubuntu4.1 amd64 [installed]
15:27:53 hostname/jammy,now 3.23ubuntu2 amd64 [installed]
15:27:53 init-system-helpers/jammy,now 1.62 all [installed]
15:27:53 iputils-ping/jammy,now 3:20211215-1 amd64 [installed]
15:27:53 libacl1/jammy,now 2.3.1-1 amd64 [installed]
15:27:53 libapt-pkg6.0/jammy-updates,now 2.4.9 amd64 [installed]
15:27:53 libasan6/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 libatomic1/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libattr1/jammy,now 1:2.5.1-1build1 amd64 [installed]
15:27:53 libaudit-common/jammy,now 1:3.0.7-1build1 all [installed]
15:27:53 libaudit1/jammy,now 1:3.0.7-1build1 amd64 [installed]
15:27:53 libbinutils/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 libblkid1/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 libbrotli1/jammy,now 1.0.9-2build6 amd64 [installed,automatic]
15:27:53 libbsd0/jammy,now 0.11.5-1 amd64 [installed,automatic]
15:27:53 libbz2-1.0/jammy,now 1.0.8-5build1 amd64 [installed]
15:27:53 libc-bin/jammy-updates,now 2.35-0ubuntu3.1 amd64 [installed]
15:27:53 libc-dev-bin/jammy-updates,now 2.35-0ubuntu3.1 amd64 [installed,automatic]
15:27:53 libc6-dev/jammy-updates,now 2.35-0ubuntu3.1 amd64 [installed,automatic]
15:27:53 libc6/jammy-updates,now 2.35-0ubuntu3.1 amd64 [installed]
15:27:53 libcap-ng0/jammy,now 0.7.9-2.2build3 amd64 [installed]
15:27:53 libcap2-bin/jammy-updates,jammy-security,now 1:2.44-1ubuntu0.22.04.1 amd64 [installed,automatic]
15:27:53 libcap2/jammy-updates,jammy-security,now 1:2.44-1ubuntu0.22.04.1 amd64 [installed]
15:27:53 libcbor0.8/jammy,now 0.8.0-2ubuntu1 amd64 [installed,automatic]
15:27:53 libcc1-0/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libcom-err2/jammy-updates,jammy-security,now 1.46.5-2ubuntu1.1 amd64 [installed]
15:27:53 libcrypt-dev/jammy,now 1:4.4.27-1 amd64 [installed,automatic]
15:27:53 libcrypt1/jammy,now 1:4.4.27-1 amd64 [installed]
15:27:53 libctf-nobfd0/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 libctf0/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 libcurl3-gnutls/jammy-updates,jammy-security,now 7.81.0-1ubuntu1.10 amd64 [installed,automatic]
15:27:53 libcurl4/jammy-updates,jammy-security,now 7.81.0-1ubuntu1.10 amd64 [installed,automatic]
15:27:53 libdb5.3/jammy,now 5.3.28+dfsg1-0.8ubuntu3 amd64 [installed]
15:27:53 libdebconfclient0/jammy,now 0.261ubuntu1 amd64 [installed]
15:27:53 libedit2/jammy,now 3.1-20210910-1build1 amd64 [installed,automatic]
15:27:53 liberror-perl/jammy,now 0.17029-1 all [installed,automatic]
15:27:53 libexpat1-dev/jammy-updates,jammy-security,now 2.4.7-1ubuntu0.2 amd64 [installed,automatic]
15:27:53 libexpat1/jammy-updates,jammy-security,now 2.4.7-1ubuntu0.2 amd64 [installed,automatic]
15:27:53 libext2fs2/jammy-updates,jammy-security,now 1.46.5-2ubuntu1.1 amd64 [installed]
15:27:53 libffi8/jammy,now 3.4.2-4 amd64 [installed]
15:27:53 libfido2-1/jammy,now 1.10.0-1 amd64 [installed,automatic]
15:27:53 libgcc-11-dev/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 libgcc-s1/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed]
15:27:53 libgcrypt20/jammy,now 1.9.4-3ubuntu3 amd64 [installed]
15:27:53 libgdbm-compat4/jammy,now 1.23-1 amd64 [installed,automatic]
15:27:53 libgdbm6/jammy,now 1.23-1 amd64 [installed,automatic]
15:27:53 libgmp10/jammy,now 2:6.2.1+dfsg-3ubuntu1 amd64 [installed]
15:27:53 libgnutls30/jammy-updates,jammy-security,now 3.7.3-4ubuntu1.2 amd64 [installed]
15:27:53 libgomp1/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libgpg-error0/jammy,now 1.43-3 amd64 [installed]
15:27:53 libgpm2/jammy,now 1.20.7-10build1 amd64 [installed,automatic]
15:27:53 libgssapi-krb5-2/jammy-updates,now 1.19.2-2ubuntu0.2 amd64 [installed]
15:27:53 libhogweed6/jammy,now 3.7.3-1build2 amd64 [installed]
15:27:53 libidn2-0/jammy,now 2.3.2-2build1 amd64 [installed]
15:27:53 libisl23/jammy,now 0.24-2build1 amd64 [installed,automatic]
15:27:53 libitm1/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libjs-jquery/jammy,now 3.6.0+dfsg+~3.5.13-1 all [installed,automatic]
15:27:53 libjs-sphinxdoc/jammy,now 4.3.2-1 all [installed,automatic]
15:27:53 libjs-underscore/jammy,now 1.13.2~dfsg-2 all [installed,automatic]
15:27:53 libk5crypto3/jammy-updates,now 1.19.2-2ubuntu0.2 amd64 [installed]
15:27:53 libkeyutils1/jammy,now 1.6.1-2ubuntu3 amd64 [installed]
15:27:53 libkrb5-3/jammy-updates,now 1.19.2-2ubuntu0.2 amd64 [installed]
15:27:53 libkrb5support0/jammy-updates,now 1.19.2-2ubuntu0.2 amd64 [installed]
15:27:53 libldap-2.5-0/jammy-updates,now 2.5.14+dfsg-0ubuntu0.22.04.2 amd64 [installed,automatic]
15:27:53 liblsan0/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 liblz4-1/jammy,now 1.9.3-2build2 amd64 [installed]
15:27:53 liblzma5/jammy,now 5.2.5-2ubuntu1 amd64 [installed]
15:27:53 libmd0/jammy,now 1.0.4-1build1 amd64 [installed,automatic]
15:27:53 libmount1/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 libmpc3/jammy,now 1.2.1-2build1 amd64 [installed,automatic]
15:27:53 libmpdec3/jammy,now 2.5.1-2build2 amd64 [installed,automatic]
15:27:53 libmpfr6/jammy,now 4.1.0-3build3 amd64 [installed,automatic]
15:27:53 libncurses6/jammy-updates,jammy-security,now 6.3-2ubuntu0.1 amd64 [installed]
15:27:53 libncursesw6/jammy-updates,jammy-security,now 6.3-2ubuntu0.1 amd64 [installed]
15:27:53 libnettle8/jammy,now 3.7.3-1build2 amd64 [installed]
15:27:53 libnghttp2-14/jammy,now 1.43.0-1build3 amd64 [installed,automatic]
15:27:53 libnsl-dev/jammy,now 1.3.0-2build2 amd64 [installed,automatic]
15:27:53 libnsl2/jammy,now 1.3.0-2build2 amd64 [installed]
15:27:53 libp11-kit0/jammy,now 0.24.0-6build1 amd64 [installed]
15:27:53 libpam-modules-bin/jammy-updates,jammy-security,now 1.4.0-11ubuntu2.3 amd64 [installed]
15:27:53 libpam-modules/jammy-updates,jammy-security,now 1.4.0-11ubuntu2.3 amd64 [installed]
15:27:53 libpam-runtime/jammy-updates,jammy-security,now 1.4.0-11ubuntu2.3 all [installed]
15:27:53 libpam0g/jammy-updates,jammy-security,now 1.4.0-11ubuntu2.3 amd64 [installed]
15:27:53 libpcre2-8-0/jammy-updates,jammy-security,now 10.39-3ubuntu0.1 amd64 [installed]
15:27:53 libpcre3/jammy-updates,jammy-security,now 2:8.39-13ubuntu0.22.04.1 amd64 [installed]
15:27:53 libperl5.34/jammy-updates,jammy-security,now 5.34.0-3ubuntu1.2 amd64 [installed,automatic]
15:27:53 libprocps8/jammy,now 2:3.3.17-6ubuntu2 amd64 [installed]
15:27:53 libpsl5/jammy,now 0.21.0-1.2build2 amd64 [installed,automatic]
15:27:53 libpython3-dev/jammy-updates,now 3.10.6-1~22.04 amd64 [installed,automatic]
15:27:53 libpython3-stdlib/jammy-updates,now 3.10.6-1~22.04 amd64 [installed,automatic]
15:27:53 libpython3.10-dev/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 libpython3.10-minimal/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 libpython3.10-stdlib/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 libpython3.10/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 libquadmath0/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libreadline8/jammy,now 8.1.2-1 amd64 [installed,automatic]
15:27:53 librtmp1/jammy,now 2.4+20151223.gitfa8646d.1-2build4 amd64 [installed,automatic]
15:27:53 libsasl2-2/jammy-updates,now 2.1.27+dfsg2-3ubuntu1.2 amd64 [installed,automatic]
15:27:53 libsasl2-modules-db/jammy-updates,now 2.1.27+dfsg2-3ubuntu1.2 amd64 [installed,automatic]
15:27:53 libseccomp2/jammy,now 2.5.3-2ubuntu2 amd64 [installed]
15:27:53 libselinux1/jammy,now 3.3-1build2 amd64 [installed]
15:27:53 libsemanage-common/jammy,now 3.3-1build2 all [installed]
15:27:53 libsemanage2/jammy,now 3.3-1build2 amd64 [installed]
15:27:53 libsepol2/jammy,now 3.3-1build1 amd64 [installed]
15:27:53 libsmartcols1/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 libsodium23/jammy,now 1.0.18-1build2 amd64 [installed,automatic]
15:27:53 libsqlite3-0/jammy-updates,jammy-security,now 3.37.2-2ubuntu0.1 amd64 [installed,automatic]
15:27:53 libss2/jammy-updates,jammy-security,now 1.46.5-2ubuntu1.1 amd64 [installed]
15:27:53 libssh-4/jammy-updates,jammy-security,now 0.9.6-2ubuntu0.22.04.1 amd64 [installed,automatic]
15:27:53 libssl-dev/jammy-updates,jammy-security,now 3.0.2-0ubuntu1.10 amd64 [installed]
15:27:53 libssl3/jammy-updates,jammy-security,now 3.0.2-0ubuntu1.10 amd64 [installed]
15:27:53 libstdc++6/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed]
15:27:53 libsystemd0/jammy-updates,now 249.11-0ubuntu3.9 amd64 [installed]
15:27:53 libtasn1-6/jammy,now 4.18.0-4build1 amd64 [installed]
15:27:53 libtinfo6/jammy-updates,jammy-security,now 6.3-2ubuntu0.1 amd64 [installed]
15:27:53 libtirpc-common/jammy-updates,jammy-security,now 1.3.2-2ubuntu0.1 all [installed]
15:27:53 libtirpc-dev/jammy-updates,jammy-security,now 1.3.2-2ubuntu0.1 amd64 [installed,automatic]
15:27:53 libtirpc3/jammy-updates,jammy-security,now 1.3.2-2ubuntu0.1 amd64 [installed]
15:27:53 libtsan0/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 libubsan1/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libudev1/jammy-updates,now 249.11-0ubuntu3.9 amd64 [installed]
15:27:53 libunistring2/jammy,now 1.0-1 amd64 [installed]
15:27:53 libuuid1/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 libxxhash0/jammy,now 0.8.1-1 amd64 [installed]
15:27:53 libzstd1/jammy,now 1.4.8+dfsg-3build1 amd64 [installed]
15:27:53 linux-libc-dev/jammy-updates,jammy-security,now 5.15.0-76.83 amd64 [installed,automatic]
15:27:53 login/jammy-updates,jammy-security,now 1:4.8.1-2ubuntu2.1 amd64 [installed]
15:27:53 logsave/jammy-updates,jammy-security,now 1.46.5-2ubuntu1.1 amd64 [installed]
15:27:53 lsb-base/jammy,now 11.1.0ubuntu4 all [installed]
15:27:53 mawk/jammy,now 1.3.4.20200120-3 amd64 [installed]
15:27:53 media-types/jammy,now 7.0.0 all [installed,automatic]
15:27:53 mount/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 ncurses-base/jammy-updates,jammy-security,now 6.3-2ubuntu0.1 all [installed]
15:27:53 ncurses-bin/jammy-updates,jammy-security,now 6.3-2ubuntu0.1 amd64 [installed]
15:27:53 openssh-client/jammy-updates,now 1:8.9p1-3ubuntu0.1 amd64 [installed]
15:27:53 openssl/jammy-updates,jammy-security,now 3.0.2-0ubuntu1.10 amd64 [installed,automatic]
15:27:53 passwd/jammy-updates,jammy-security,now 1:4.8.1-2ubuntu2.1 amd64 [installed]
15:27:53 perl-base/jammy-updates,jammy-security,now 5.34.0-3ubuntu1.2 amd64 [installed]
15:27:53 perl-modules-5.34/jammy-updates,jammy-security,now 5.34.0-3ubuntu1.2 all [installed,automatic]
15:27:53 perl/jammy-updates,jammy-security,now 5.34.0-3ubuntu1.2 amd64 [installed,automatic]
15:27:53 procps/jammy,now 2:3.3.17-6ubuntu2 amd64 [installed]
15:27:53 python3-dev/jammy-updates,now 3.10.6-1~22.04 amd64 [installed]
15:27:53 python3-distutils/jammy-updates,now 3.10.6-1~22.04 all [installed,automatic]
15:27:53 python3-lib2to3/jammy-updates,now 3.10.6-1~22.04 all [installed,automatic]
15:27:53 python3-minimal/jammy-updates,now 3.10.6-1~22.04 amd64 [installed,automatic]
15:27:53 python3-pip/jammy-updates,jammy-security,now 22.0.2+dfsg-1ubuntu0.3 all [installed]
15:27:53 python3-pkg-resources/jammy-updates,jammy-security,now 59.6.0-1.2ubuntu0.22.04.1 all [installed,automatic]
15:27:53 python3-setuptools/jammy-updates,jammy-security,now 59.6.0-1.2ubuntu0.22.04.1 all [installed,automatic]
15:27:53 python3-wheel/jammy-updates,jammy-security,now 0.37.1-2ubuntu0.22.04.1 all [installed,automatic]
15:27:53 python3.10-dev/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 python3.10-minimal/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 python3.10/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 python3/jammy-updates,now 3.10.6-1~22.04 amd64 [installed]
15:27:53 readline-common/jammy,now 8.1.2-1 all [installed,automatic]
15:27:53 rpcsvc-proto/jammy,now 1.4.2-0ubuntu6 amd64 [installed,automatic]
15:27:53 sed/jammy,now 4.8-1ubuntu2 amd64 [installed]
15:27:53 sensible-utils/jammy,now 0.0.17 all [installed]
15:27:53 sshpass/jammy,now 1.09-1 amd64 [installed]
15:27:53 sysvinit-utils/jammy,now 3.01-1ubuntu1 amd64 [installed]
15:27:53 tar/jammy-updates,jammy-security,now 1.34+dfsg-1ubuntu0.1.22.04.1 amd64 [installed]
15:27:53 ubuntu-keyring/jammy,now 2021.03.26 all [installed]
15:27:53 unzip/jammy-updates,jammy-security,now 6.0-26ubuntu3.1 amd64 [installed]
15:27:53 usrmerge/jammy,now 25ubuntu2 all [installed]
15:27:53 util-linux/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 vim-common/jammy-updates,jammy-security,now 2:8.2.3995-1ubuntu2.8 all [installed,automatic]
15:27:53 vim-runtime/jammy-updates,jammy-security,now 2:8.2.3995-1ubuntu2.8 all [installed,automatic]
15:27:53 vim/jammy-updates,jammy-security,now 2:8.2.3995-1ubuntu2.8 amd64 [installed]
15:27:53 wget/jammy,now 1.21.2-2ubuntu1 amd64 [installed]
15:27:53 xxd/jammy-updates,jammy-security,now 2:8.2.3995-1ubuntu2.8 amd64 [installed,automatic]
15:27:53 zlib1g-dev/jammy-updates,jammy-security,now 1:1.2.11.dfsg-2ubuntu9.2 amd64 [installed,automatic]
15:27:53 zlib1g/jammy-updates,jammy-security,now 1:1.2.11.dfsg-2ubuntu9.2 amd64 [installed]
pip
15:27:53 + pip3 list
15:27:54 Package Version
15:27:54 ------------------ ---------
15:27:54 ansible 8.1.0
15:27:54 ansible-compat 4.1.2
15:27:54 ansible-core 2.15.1
15:27:54 ansible-lint 6.17.2
15:27:54 ansible-pylibssh 1.1.0
15:27:54 async-timeout 4.0.2
15:27:54 attrs 23.1.0
15:27:54 awscli 1.27.163
15:27:54 bcrypt 4.0.1
15:27:54 black 23.3.0
15:27:54 boto 2.49.0
15:27:54 boto3 1.26.163
15:27:54 botocore 1.29.163
15:27:54 bracex 2.3.post1
15:27:54 certifi 2023.5.7
15:27:54 cffi 1.15.1
15:27:54 charset-normalizer 3.1.0
15:27:54 click 8.1.3
15:27:54 colorama 0.4.4
15:27:54 cryptography 41.0.1
15:27:54 datadog 0.45.0
15:27:54 datadog-api-client 2.14.0
15:27:54 dnspython 2.3.0
15:27:54 docker 6.1.3
15:27:54 docutils 0.16
15:27:54 filelock 3.12.2
15:27:54 fortiosapi 1.0.5
15:27:54 httplib2 0.22.0
15:27:54 idna 3.4
15:27:54 Jinja2 3.1.2
15:27:54 jmespath 1.0.1
15:27:54 jsonschema 4.17.3
15:27:54 markdown-it-py 3.0.0
15:27:54 MarkupSafe 2.1.3
15:27:54 mdurl 0.1.2
15:27:54 mitogen 0.3.3
15:27:54 mypy-extensions 1.0.0
15:27:54 netaddr 0.8.0
15:27:54 oyaml 1.0
15:27:54 packaging 23.1
15:27:54 paramiko 3.2.0
15:27:54 passlib 1.7.4
15:27:54 pathspec 0.11.1
15:27:54 pip 23.1.2
15:27:54 platformdirs 3.8.0
15:27:54 pyasn1 0.5.0
15:27:54 pycparser 2.21
15:27:54 Pygments 2.15.1
15:27:54 PyNaCl 1.5.0
15:27:54 pyOpenSSL 23.2.0
15:27:54 pyparsing 3.1.0
15:27:54 pyrsistent 0.19.3
15:27:54 pyspnego 0.9.1
15:27:54 python-dateutil 2.8.2
15:27:54 pywinrm 0.4.3
15:27:54 PyYAML 5.4.1
15:27:54 redis 4.6.0
15:27:54 requests 2.31.0
15:27:54 requests-credssp 2.0.0
15:27:54 requests-ntlm 1.2.0
15:27:54 resolvelib 1.0.1
15:27:54 rich 13.4.2
15:27:54 rsa 4.7.2
15:27:54 ruamel.yaml 0.17.32
15:27:54 ruamel.yaml.clib 0.2.7
15:27:54 s3transfer 0.6.1
15:27:54 setuptools 59.6.0
15:27:54 six 1.16.0
15:27:54 subprocess-tee 0.4.1
15:27:54 tomli 2.0.1
15:27:54 typing_extensions 4.7.0
15:27:54 urllib3 1.26.16
15:27:54 wcmatch 8.4.1
15:27:54 websocket-client 1.6.1
15:27:54 wheel 0.37.1
15:27:54 xmltodict 0.13.0
15:27:54 yamllint 1.32.0
15:27:54
15:27:54 [notice] A new release of pip is available: 23.1.2 -> 23.2.1
15:27:54 [notice] To update, run: python3 -m pip install --upgrade pip
STEPS TO REPRODUCE
The issue is intermittent and not reproducible on-demand. We see the error multiple times per job and use ignore_errors: true to work around it.
Example playbook
- name: ICX AAA role
hosts: icx_*
gather_facts: no
ignore_unreachable: true
become: yes
serial: 10
roles:
- icx-aaa
vars:
icx_username: "{{ hostvars['localhost']['network_user'] }}"
icx_password: "{{ hostvars['localhost']['network_password'] }}"
icx_enable_password: "{{ icx_password }}"
icx_radius_psk: "{{ hostvars['localhost']['radius_psk'] }}"
ansible_command_timeout: 45
Example tasks that generate the error
- name: Configure RADIUS attributes
community.network.icx_config:
lines:
- radius-server retransmit {{ aaa_radius_retransmit }}
- radius-server timeout {{ aaa_radius_timeout }}
register: _aaa_radius_attr
ignore_errors: "{{ icx_bypass_errors }}"
- name: Test reachability to RADIUS servers
community.network.icx_ping:
dest: "{{ item }}"
loop: "{{ radius_servers }}"
register: _validation
EXPECTED RESULTS
I expect that it completes successfully every run, but intermittently the task fails as indicated above.
15:06:52 TASK [icx-aaa : Configure RADIUS attributes] ***********************************
15:06:52 ok: [switch]
ACTUAL RESULTS
15:30:29 Loading collection ansible.netcommon from /usr/share/ansible/collections/ansible_collections/ansible/netcommon
15:30:29 Loading collection ansible.utils from /usr/share/ansible/collections/ansible_collections/ansible/utils
15:30:29 redirecting (type: become) ansible.builtin.enable to ansible.netcommon.enable
15:30:29 <switch> attempting to start connection
15:30:29 <switch> using connection plugin ansible.netcommon.network_cli
15:30:29 Found ansible-connection at path /usr/local/bin/ansible-connection
15:30:29 <switch> found existing local domain socket, using it!
15:30:29 <switch> updating play_context for connection
15:30:29 <switch> Loading collection ansible.builtin from
15:30:29 <switch> local domain socket path is /root/.ansible/pc/b146f0a1d8
15:30:29 Using module file /usr/share/ansible/collections/ansible_collections/community/network/plugins/modules/icx_config.py
15:30:29 Pipelining is enabled.
15:30:29 <switch> ESTABLISH LOCAL CONNECTION FOR USER: root
15:30:29 <switch> EXEC /bin/sh -c '/usr/bin/python3 && sleep 0'
15:30:29
15:30:29 TASK [icx-aaa : Configure RADIUS attributes] ***********************************
15:30:29 task path: /etc/ansible/roles/icx-aaa/tasks/icx_aaa.yml:26
15:30:29 The full traceback is:
15:30:29 File "/tmp/ansible_community.network.icx_config_payload_546o4akj/ansible_community.network.icx_config_payload.zip/ansible_collections/community/network/plugins/module_utils/network/icx/icx.py", line 34, in run_commands
15:30:29 return connection.run_commands(commands=commands, check_rc=check_rc)
15:30:29 File "/tmp/ansible_community.network.icx_config_payload_546o4akj/ansible_community.network.icx_config_payload.zip/ansible/module_utils/connection.py", line 200, in __rpc__
15:30:29 raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
15:30:29 fatal: [switch]: FAILED! => {
15:30:29 "changed": false,
15:30:29 "invocation": {
15:30:29 "module_args": {
15:30:29 "after": null,
15:30:29 "backup": false,
15:30:29 "before": null,
15:30:29 "defaults": false,
15:30:29 "diff_against": null,
15:30:29 "diff_ignore_lines": null,
15:30:29 "intended_config": null,
15:30:29 "lines": [
15:30:29 "radius-server retransmit 5",
15:30:29 "radius-server timeout 10"
15:30:29 ],
15:30:29 "match": "line",
15:30:29 "multiline_delimiter": "@",
15:30:29 "parents": null,
15:30:29 "replace": "line",
15:30:29 "running_config": null,
15:30:29 "save_when": "never",
15:30:29 "src": null
15:30:29 }
15:30:29 },
15:30:29 "msg": "command timeout triggered, timeout value is 45 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."
15:30:29 }
15:30:29 ...ignoring
15:30:30 Loading collection ansible.netcommon from /usr/share/ansible/collections/ansible_collections/ansible/netcommon
15:30:30 Loading collection ansible.utils from /usr/share/ansible/collections/ansible_collections/ansible/utils
15:30:30 redirecting (type: become) ansible.builtin.enable to ansible.netcommon.enable
15:30:30 <switch> attempting to start connection
15:30:30 <switch> using connection plugin ansible.netcommon.network_cli
15:30:30 Found ansible-connection at path /usr/local/bin/ansible-connection
15:30:30 <switch> found existing local domain socket, using it!
15:30:30 <switch> invoked shell using ssh_type: libssh
15:30:30 <switch> updating play_context for connection
15:30:30 <switch> Loading collection ansible.builtin from
15:30:30 <switch> local domain socket path is /root/.ansible/pc/b146f0a1d8
