salt
salt copied to clipboard
saltstack
[BUG] 3006.8 master take extra 4 seconds to finish test.ping
Description I downgraded to 3006.8(salt01) from 3007 and notice a simple test.ping take extra 4 seconds compared to 3007 master(rock01).
Please label it as Enhancement.
Setup time sudo salt -N testsalt test.ping
Please be as specific as possible and give set-up details.
- [ x] on-prem machine
- [x ] VM (Virtualbox, KVM, etc. please specify)
- [ ] VM running on a cloud service, please be explicit and add details
- [ ] container (Kubernetes, Docker, containerd, etc. please specify)
- [ ] or a combination, please be explicit
- [ ] jails if it is FreeBSD
- [ ] classic packaging
- [ x] onedir packaging
- [ x] used bootstrap to install
Steps to Reproduce the behavior
[me@salt01 ~]$ time sudo salt -N testsalt test.ping # 15 minions in testsalt nodegroup.
Expected behavior
rock01 and salt01 masters are on same subnet.
[me@rock01 ~]$ time sudo salt -N testsalt test.ping # same 15 minions in rocky01
Screenshots If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3006.8
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.16.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: rocky 8.9 Green Obsidian
locale: utf-8
machine: x86_64
release: 4.18.0-513.24.1.el8_9.x86_64
system: Linux
version: Rocky Linux 8.9 Green Obsidian
Additional context Add any other context about the problem here.
Following test.ping time is from 3007.1 master and minions. lower than 1 second. But this bug: https://github.com/saltstack/salt/issues/66519 is stopping it as production ready.
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 15
# of minions returned: 15
# of minions that did not return: 0
# of minions with errors: 0
-------------------------------------------
real 0m0.967s
user 0m0.015s
sys 0m0.010s
@dwoz and @Akm0d , Thanks for accepting this issue as bug and scheduled it to be fixed in 3006.9. Please let me know if you need further information from me.
@tjyang If you reboot the machine then does the issue persist? Knowing this will help me pinpoint the root cause.
@Akm0d
salt02 and salt01 are in multi-master both on 3006.8, I rebooted salt02 and do the test.ping on testsalt nodegroup with has 16 minions, see the masked test result below. let me know if you need more info.
[me@salt02 ~]$ sudo systemctl status salt-master |grep Active
Active: active (running) since Tue 2024-06-18 17:11:58 CDT; 1min 7s ago
[me@salt02 ~]$ time sudo salt -N testsalt test.ping |tail -20
<snipped>
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 16
# of minions returned: 16
# of minions that did not return: 0
# of minions with errors: 0
-------------------------------------------
real 0m5.830s
user 0m0.736s
sys 0m0.090s
[me@salt02 ~]$
- Same time spent for pinging its own minion. This is on RockyLinux using onedir rpm approach.
[me@salt02 ~]$ time sudo salt salt02 test.ping
salt02:
True
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 1
# of minions returned: 1
# of minions that did not return: 0
# of minions with errors: 0
-------------------------------------------
real 0m5.824s
user 0m0.734s
sys 0m0.080s
[me@salt02 ~]$
I am still having a hard time replicating the issue. Can you show me the output of "salt salt02 test.ping --log-level=trace"? Also, give me a versions report for the master and minions.
- log at trace level, test ping saltmaster's own minion take 5+ secs.
[me@salt02 ~]$ time sudo salt --log-level=trace salt02 test.ping
[TRACE ] Setting up log file logging: {'log_path': '/var/log/salt/master', 'log_level': 'debug', 'log_format': '%(asctime)s,%(msecs)03.0f [%(name)-17s][%(levelname)-8s] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'root'}
[TRACE ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.
[TRACE ] Error loading log_handlers.sentry_mod: Cannot find 'raven' python library, 'sentry_handler' config is empty or not defined
[TRACE ] Processing <bound method SaltfileMixIn.process_saltfile of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE ] Processing <bound method ConfigDirMixIn.process_config_dir of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/autosign_grains.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/autosign_grains.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/file_server.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/file_server.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/node-groups.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/node-groups.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/syndic.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/syndic.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/top.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/top.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: salt02
[DEBUG ] Missing configuration file: /root/.saltrc
[TRACE ] Processing <bound method ExtendedTargetOptionsMixIn.process_pillar_target of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_output of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_output_file of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_state_verbose of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE ] Processing <bound method NoParseMixin.process_no_parse of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f84278a3d90>, <CustomOption at 0x7f8414b09ae0: -H/--hosts>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8414b1e440>, <CustomOption at 0x7f8414b09f60: -E/--pcre>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03370>, <CustomOption at 0x7f8414b09fc0: -L/--list>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03400>, <CustomOption at 0x7f8414b0a110: -G/--grain>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03490>, <CustomOption at 0x7f8414b0a1a0: -P/--grain-pcre>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03520>, <CustomOption at 0x7f8414b0a1d0: -N/--nodegroup>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b035b0>, <CustomOption at 0x7f8414b0a230: -R/--range>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b036d0>, <CustomOption at 0x7f8414b0a3b0: -C/--compound>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03760>, <CustomOption at 0x7f8412ae8bb0: -J/--pillar-pcre>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03640>, <CustomOption at 0x7f8412ae8be0: -S/--ipcidr>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f8412b03880>, <CustomOption at 0x7f8412ae8fa0: --out-indent/--output-indent>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f8412b03910>, <CustomOption at 0x7f8412ae9000: --out-file-append/--output-file-append>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f8412b037f0>, <CustomOption at 0x7f8412ae9030: --no-color/--no-colour>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f8412b039a0>, <CustomOption at 0x7f8412ae9060: --force-color/--force-colour>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f8412b03a30>, <CustomOption at 0x7f8412ae9090: --state-output/--state_output>)
[TRACE ] Processing <bound method LogLevelMixIn.process_log_level of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE ] Processing <bound method LogLevelMixIn.process_log_file of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE ] Processing <bound method LogLevelMixIn.process_log_level_logfile of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE ] Processing <function TargetOptionsMixIn._mixin_after_parsed at 0x7f8412b00700>
[TRACE ] Processing <function OutputOptionsMixIn._mixin_after_parsed at 0x7f8412b00dc0>
[TRACE ] Processing <function SaltCMDOptionParser._mixin_after_parsed at 0x7f8412b01900>
[TRACE ] Processing <function LogLevelMixIn.__setup_logging_routines at 0x7f8412aef760>
[TRACE ] Processing <function MergeConfigMixIn.__merge_config_with_cli at 0x7f8412aef1c0>
[TRACE ] Processing <function LogLevelMixIn.__setup_console_logger_config at 0x7f8412aefa30>
[TRACE ] Processing <function LogLevelMixIn.__setup_logfile_logger_config at 0x7f8412aefac0>
[TRACE ] Processing <function LogLevelMixIn.__setup_logging_config at 0x7f8412aefb50>
[TRACE ] Processing <function LogLevelMixIn.__verify_logging at 0x7f8412aefc70>
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[TRACE ] Processing <function LogLevelMixIn.__setup_logging at 0x7f8412aefbe0>
[TRACE ] Setting up console logging: {'log_level': 'trace', 'log_format': '[%(levelname)-8s] %(message)s', 'date_format': '%H:%M:%S'}
[TRACE ] Setting up log file logging: {'log_path': '/var/log/salt/master', 'log_level': 'debug', 'log_format': '%(asctime)s,%(msecs)03.0f [%(name)-17s][%(levelname)-8s] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'root'}
[TRACE ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.
[TRACE ] Error loading log_handlers.sentry_mod: Cannot find 'raven' python library, 'sentry_handler' config is empty or not defined
[DEBUG ] Configuration file path: /etc/salt/master
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/autosign_grains.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/autosign_grains.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/file_server.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/file_server.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/node-groups.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/node-groups.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/syndic.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/syndic.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/top.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/top.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: salt02
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[TRACE ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc
[TRACE ] ReqChannel send clear load={'cmd': 'publish', 'tgt': 'salt02', 'fun': 'test.ping', 'arg': [], 'key': '/9bvh28eiZFrNvlJAoWQNCiqekEYiPv+Sdken4cgY6boNdObDPqt77776ZyCI7YQXO5/PE6+H6A=', 'tgt_type': 'glob', 'ret': '', 'jid': '', 'kwargs': {'show_timeout': True, 'show_jid': False, 'delimiter': ':'}, 'user': 'sudo_tjyang', 'to': 5}
[DEBUG ] Closing AsyncReqChannel instance
[TRACE ] func get_cli_event_returns()
[DEBUG ] Marking 'base64_encode' as a jinja filter
[DEBUG ] Marking 'base64_decode' as a jinja filter
[DEBUG ] Marking 'md5' as a jinja filter
[DEBUG ] Marking 'sha1' as a jinja filter
[DEBUG ] Marking 'sha256' as a jinja filter
[DEBUG ] Marking 'sha512' as a jinja filter
[DEBUG ] Marking 'hmac' as a jinja filter
[DEBUG ] Marking 'hmac_compute' as a jinja filter
[DEBUG ] Marking 'random_hash' as a jinja filter
[DEBUG ] Marking 'rand_str' as a jinja filter
[DEBUG ] Marking 'file_hashsum' as a jinja filter
[DEBUG ] Marking 'http_query' as a jinja filter
[DEBUG ] Marking 'ifelse' as a jinja global
[DEBUG ] Marking 'strftime' as a jinja filter
[DEBUG ] Marking 'date_format' as a jinja filter
[DEBUG ] Marking 'raise' as a jinja global
[DEBUG ] Marking 'match' as a jinja test
[DEBUG ] Marking 'equalto' as a jinja test
[DEBUG ] Marking 'skip' as a jinja filter
[DEBUG ] Marking 'sequence' as a jinja filter
[DEBUG ] Marking 'to_bool' as a jinja filter
[DEBUG ] Marking 'indent' as a jinja filter
[DEBUG ] Marking 'tojson' as a jinja filter
[DEBUG ] Marking 'quote' as a jinja filter
[DEBUG ] Marking 'regex_escape' as a jinja filter
[DEBUG ] Marking 'regex_search' as a jinja filter
[DEBUG ] Marking 'regex_match' as a jinja filter
[DEBUG ] Marking 'regex_replace' as a jinja filter
[DEBUG ] Marking 'uuid' as a jinja filter
[DEBUG ] Marking 'unique' as a jinja filter
[DEBUG ] Marking 'min' as a jinja filter
[DEBUG ] Marking 'max' as a jinja filter
[DEBUG ] Marking 'avg' as a jinja filter
[DEBUG ] Marking 'union' as a jinja filter
[DEBUG ] Marking 'intersect' as a jinja filter
[DEBUG ] Marking 'difference' as a jinja filter
[DEBUG ] Marking 'symmetric_difference' as a jinja filter
[DEBUG ] Marking 'method_call' as a jinja filter
[DEBUG ] Marking 'yaml_dquote' as a jinja filter
[DEBUG ] Marking 'yaml_squote' as a jinja filter
[DEBUG ] Marking 'yaml_encode' as a jinja filter
[TRACE ] loaded json json lib
[DEBUG ] The functions from module 'local_cache' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded local_cache.get_load
[DEBUG ] Reading minion list from /var/cache/salt/master/jobs/48/438fa51f2a24bd8f44e542f84e3c938ed2e2685c6b7047914f6f44e7b97e74/.minions.p
[DEBUG ] get_iter_returns for jid 20240620230015298088 sent to {'salt02'} will timeout at 18:00:20.471577
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] get_event() received = {'data': {'jid': '20240620230015298088', 'tgt_type': 'glob', 'tgt': 'salt02', 'user': 'sudo_tjyang', 'fun': 'test.ping', 'arg': [], 'minions': ['salt02'], 'missing': [], '_stamp': '2024-06-20T23:00:15.300576'}, 'tag': 'salt/job/20240620230015298088/new'}
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] get_event() received = {'data': {'cmd': '_return', 'id': 'salt02', 'success': True, 'return': True, 'retcode': 0, 'jid': '20240620230015298088', 'fun': 'test.ping', 'fun_args': [], 'user': 'sudo_tjyang', '_stamp': '2024-06-20T23:00:15.421773'}, 'tag': 'salt/job/20240620230015298088/ret/salt02'}
[DEBUG ] jid 20240620230015298088 return from salt02
[DEBUG ] return event: {'salt02': {'ret': True, 'retcode': 0, 'jid': '20240620230015298088'}}
[DEBUG ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded nested.output
[TRACE ] data = {'salt02': True}
salt02:
True
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE ] _get_event() waited 0 seconds and received nothing
[DEBUG ] Closing IPCMessageSubscriber instance
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 1
# of minions returned: 1
# of minions that did not return: 0
# of minions with errors: 0
-------------------------------------------
real 0m5.839s
user 0m0.782s
sys 0m0.091s
[me@salt02 ~]$
- Minion info: sudo salt-call -V
Salt Version:
Salt: 3006.8
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.16.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: rocky 8.9 Green Obsidian
locale: utf-8
machine: x86_64
release: 4.18.0-513.24.1.el8_9.x86_64
system: Linux
version: Rocky Linux 8.9 Green Obsidian
- Master version info: sudo salt -V
Salt Version:
Salt: 3006.8
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.16.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: rocky 8.9 Green Obsidian
locale: utf-8
machine: x86_64
release: 4.18.0-513.24.1.el8_9.x86_64
system: Linux
version: Rocky Linux 8.9 Green Obsidian
This is my result
╰─ salt -V
Salt Version:
Salt: 3007.1
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:33:04) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.4
libgit2: 1.5.0
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: 1.11.1
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.16.0
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: debian 11.9 bullseye
locale: utf-8
machine: aarch64
release: 5.10.103-v8+
system: Linux
version: Debian GNU/Linux 11.9 bullseye
sudo salt -N group4 test.ping 3.34s user 0.35s system 80% cpu 4.606 total
sudo salt testminion test.ping 2.44s user 0.27s system 86% cpu 3.134 total
@network-shark, this issue is created for minion/master version 3006.8, not 3007.1.
@Akm0d
Looks like I have misconfiguration somewhere in my production server to cause the delay. Please close this issue.
Another 3006.8 master/minion in my test lab running as VMWare VM.
- test.ping itself is less than 1 second.
[me@rocky9t01 ~]$ time sudo salt rocky9t01 -l trace test.version
[TRACE ] Setting up log file logging: {'log_path': '/var/log/salt/master', 'log_level': 'warning', 'log_format': '%(asctime)s,%(msecs)03d [%(name)-17s:%(lineno)-4d][%(levelname)-8s][%(process)d] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'salt'}
[TRACE ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.
[TRACE ] Error loading log_handlers.sentry_mod: Cannot find 'raven' python library, 'sentry_handler' config is empty or not defined
[TRACE ] Processing <bound method SaltfileMixIn.process_saltfile of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE ] Processing <bound method ConfigDirMixIn.process_config_dir of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/acl.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/acl.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/autosign_grains.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/autosign_grains.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/returners.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/returners.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: rocky9t01
[DEBUG ] Missing configuration file: /root/.saltrc
[TRACE ] Processing <bound method ExtendedTargetOptionsMixIn.process_pillar_target of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_output of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_output_file of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_state_verbose of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE ] Processing <bound method NoParseMixin.process_no_parse of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e30b63d90>, <CustomOption at 0x7f0e2ef45b10: -H/--hosts>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2ef5a440>, <CustomOption at 0x7f0e2ef45f90: -E/--pcre>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f370>, <CustomOption at 0x7f0e2ef45ff0: -L/--list>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f400>, <CustomOption at 0x7f0e2ef46140: -G/--grain>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f490>, <CustomOption at 0x7f0e2ef461d0: -P/--grain-pcre>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f520>, <CustomOption at 0x7f0e2ef46200: -N/--nodegroup>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f5b0>, <CustomOption at 0x7f0e2ef46260: -R/--range>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f6d0>, <CustomOption at 0x7f0e2ef463e0: -C/--compound>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f760>, <CustomOption at 0x7f0e2d754be0: -J/--pillar-pcre>)
[TRACE ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f640>, <CustomOption at 0x7f0e2d754c10: -S/--ipcidr>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f0e2d76f880>, <CustomOption at 0x7f0e2d754fd0: --out-indent/--output-indent>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f0e2d76f910>, <CustomOption at 0x7f0e2d755030: --out-file-append/--output-file-append>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f0e2d76f7f0>, <CustomOption at 0x7f0e2d755060: --no-color/--no-colour>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f0e2d76f9a0>, <CustomOption at 0x7f0e2d755090: --force-color/--force-colour>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f0e2d76fa30>, <CustomOption at 0x7f0e2d7550c0: --state-output/--state_output>)
[TRACE ] Processing <bound method LogLevelMixIn.process_log_level of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE ] Processing <bound method LogLevelMixIn.process_log_file of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE ] Processing <bound method LogLevelMixIn.process_log_level_logfile of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE ] Processing <function TargetOptionsMixIn._mixin_after_parsed at 0x7f0e2d76c700>
[TRACE ] Processing <function OutputOptionsMixIn._mixin_after_parsed at 0x7f0e2d76cdc0>
[TRACE ] Processing <function SaltCMDOptionParser._mixin_after_parsed at 0x7f0e2d76d900>
[TRACE ] Processing <function LogLevelMixIn.__setup_logging_routines at 0x7f0e2d75b760>
[TRACE ] Processing <function MergeConfigMixIn.__merge_config_with_cli at 0x7f0e2d75b1c0>
[TRACE ] Processing <function LogLevelMixIn.__setup_console_logger_config at 0x7f0e2d75ba30>
[TRACE ] Processing <function LogLevelMixIn.__setup_logfile_logger_config at 0x7f0e2d75bac0>
[TRACE ] Processing <function LogLevelMixIn.__setup_logging_config at 0x7f0e2d75bb50>
[TRACE ] Processing <function LogLevelMixIn.__verify_logging at 0x7f0e2d75bc70>
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[TRACE ] Processing <function LogLevelMixIn.__setup_logging at 0x7f0e2d75bbe0>
[TRACE ] Setting up console logging: {'log_level': 'trace', 'log_format': '[%(levelname)-8s] %(message)s', 'date_format': '%H:%M:%S'}
[TRACE ] Setting up log file logging: {'log_path': '/var/log/salt/master', 'log_level': 'warning', 'log_format': '%(asctime)s,%(msecs)03d [%(name)-17s:%(lineno)-4d][%(levelname)-8s][%(process)d] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'salt'}
[TRACE ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.
[TRACE ] Error loading log_handlers.sentry_mod: Cannot find 'raven' python library, 'sentry_handler' config is empty or not defined
[DEBUG ] Configuration file path: /etc/salt/master
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/acl.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/acl.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/autosign_grains.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/autosign_grains.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/returners.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/returners.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: rocky9t01
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[TRACE ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc
[TRACE ] ReqChannel send clear load={'cmd': 'publish', 'tgt': 'rocky9t01', 'fun': 'test.version', 'arg': [], 'key': 'B5sLpYwuKvNAj56LPptQMGChQ3c+9yBEkmhXvkx01oHOfE2DHp7jw/e55wzLC91hnZDPO3Yn/F4=', 'tgt_type': 'glob', 'ret': '', 'jid': '', 'kwargs': {'show_timeout': True, 'show_jid': False, 'delimiter': ':'}, 'user': 'sudo_me'}
[DEBUG ] Closing AsyncReqChannel instance
[TRACE ] func get_cli_event_returns()
[DEBUG ] Marking 'base64_encode' as a jinja filter
[DEBUG ] Marking 'base64_decode' as a jinja filter
[DEBUG ] Marking 'md5' as a jinja filter
[DEBUG ] Marking 'sha1' as a jinja filter
[DEBUG ] Marking 'sha256' as a jinja filter
[DEBUG ] Marking 'sha512' as a jinja filter
[DEBUG ] Marking 'hmac' as a jinja filter
[DEBUG ] Marking 'hmac_compute' as a jinja filter
[DEBUG ] Marking 'random_hash' as a jinja filter
[DEBUG ] Marking 'rand_str' as a jinja filter
[DEBUG ] Marking 'file_hashsum' as a jinja filter
[DEBUG ] Marking 'http_query' as a jinja filter
[DEBUG ] Marking 'ifelse' as a jinja global
[DEBUG ] Marking 'strftime' as a jinja filter
[DEBUG ] Marking 'date_format' as a jinja filter
[DEBUG ] Marking 'raise' as a jinja global
[DEBUG ] Marking 'match' as a jinja test
[DEBUG ] Marking 'equalto' as a jinja test
[DEBUG ] Marking 'skip' as a jinja filter
[DEBUG ] Marking 'sequence' as a jinja filter
[DEBUG ] Marking 'to_bool' as a jinja filter
[DEBUG ] Marking 'indent' as a jinja filter
[DEBUG ] Marking 'tojson' as a jinja filter
[DEBUG ] Marking 'quote' as a jinja filter
[DEBUG ] Marking 'regex_escape' as a jinja filter
[DEBUG ] Marking 'regex_search' as a jinja filter
[DEBUG ] Marking 'regex_match' as a jinja filter
[DEBUG ] Marking 'regex_replace' as a jinja filter
[DEBUG ] Marking 'uuid' as a jinja filter
[DEBUG ] Marking 'unique' as a jinja filter
[DEBUG ] Marking 'min' as a jinja filter
[DEBUG ] Marking 'max' as a jinja filter
[DEBUG ] Marking 'avg' as a jinja filter
[DEBUG ] Marking 'union' as a jinja filter
[DEBUG ] Marking 'intersect' as a jinja filter
[DEBUG ] Marking 'difference' as a jinja filter
[DEBUG ] Marking 'symmetric_difference' as a jinja filter
[DEBUG ] Marking 'method_call' as a jinja filter
[DEBUG ] Marking 'yaml_dquote' as a jinja filter
[DEBUG ] Marking 'yaml_squote' as a jinja filter
[DEBUG ] Marking 'yaml_encode' as a jinja filter
[TRACE ] loaded json json lib
[DEBUG ] The functions from module 'local_cache' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded local_cache.get_load
[DEBUG ] Reading minion list from /var/cache/salt/master/jobs/93/48d208fe005cd60e7e1faddbcd1a76f44203a8ccd383213e630d6b5cc93c36/.minions.p
[DEBUG ] get_iter_returns for jid 20240624193307945925 sent to {'rocky9t01'} will timeout at 14:33:13.092168
[TRACE ] Get event. tag: salt/job/20240624193307945925
[TRACE ] _get_event() waited 0 seconds and received nothing
[TRACE ] Get event. tag: salt/job/20240624193307945925
[TRACE ] get_event() received = {'data': {'jid': '20240624193307945925', 'tgt_type': 'glob', 'tgt': 'rocky9t01', 'user': 'sudo_me', 'fun': 'test.version', 'arg': [], 'minions': ['rocky9t01'], 'missing': [], '_stamp': '2024-06-24T19:33:07.947187'}, 'tag': 'salt/job/20240624193307945925/new'}
[TRACE ] Get event. tag: salt/job/20240624193307945925
[TRACE ] get_event() received = {'data': {'cmd': '_return', 'id': 'rocky9t01', 'success': True, 'return': '3006.8', 'retcode': 0, 'jid': '20240624193307945925', 'fun': 'test.version', 'fun_args': [], 'user': 'sudo_me', '_stamp': '2024-06-24T19:33:08.054649'}, 'tag': 'salt/job/20240624193307945925/ret/rocky9t01'}
[DEBUG ] jid 20240624193307945925 return from rocky9t01
[DEBUG ] return event: {'rocky9t01': {'ret': '3006.8', 'retcode': 0, 'jid': '20240624193307945925'}}
[DEBUG ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded nested.output
[TRACE ] data = {'rocky9t01': '3006.8'}
rocky9t01:
3006.8
[TRACE ] Get event. tag: salt/job/20240624193307945925
[TRACE ] _get_event() waited 0 seconds and received nothing
[DEBUG ] jid 20240624193307945925 found all minions {'rocky9t01'}
[DEBUG ] Closing IPCMessageSubscriber instance
real 0m0.834s
user 0m0.662s
sys 0m0.052s
[me@rocky9t01 ~]$
- minion: salt-call -V
[me@rocky9t01 ~]$ salt-call -V
Salt Version:
Salt: 3006.8
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.16.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: rocky 9.4 Blue Onyx
locale: utf-8
machine: x86_64
release: 5.14.0-427.22.1.el9_4.x86_64
system: Linux
version: Rocky Linux 9.4 Blue Onyx
[me@rocky9t01 ~]$
- master: salt -V
[me@rocky9t01 ~]$ salt -V
Salt Version:
Salt: 3006.8
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.16.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: rocky 9.4 Blue Onyx
locale: utf-8
machine: x86_64
release: 5.14.0-427.22.1.el9_4.x86_64
system: Linux
version: Rocky Linux 9.4 Blue Onyx
[me@rocky9t01 ~]$
- This is a sub one sec test.ping on rockylinux 8 test master instance.
- Still need to figure out what caused the production master instances spent extra 4 secs.
[me@rocky8t01 ~]$ time sudo salt rocky8t01 test.ping
rocky8t01:
True
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 1
# of minions returned: 1
# of minions that did not return: 0
# of minions with errors: 0
-------------------------------------------
real 0m0.998s
user 0m0.013s
sys 0m0.013s
[me@rocky8t01 ~]$ cat /etc/redhat-release
Rocky Linux release 8.10 (Green Obsidian)
[me@rocky8t01 ~]$ salt-call -V
Salt Version:
Salt: 3006.8
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.16.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: rocky 8.10 Green Obsidian
locale: utf-8
machine: x86_64
release: 4.18.0-553.el8_10.x86_64
system: Linux
version: Rocky Linux 8.10 Green Obsidian
[me@rocky8t01 ~]$
Wed Jul 31 07:28:48 EDT 2024
@Akm0d and all
The extra 4 seconds delay is found and fixed by comment out order_masters: True line.
- This is my syndic.conf in /etc/salt/master.d , master was set True but syndic_master was set to fake name.
##### Syndic settings #####
##########################################
# The Salt syndic is used to pass commands through a master from a higher
# master. Using the syndic is simple. If this is a master that will have
# syndic servers(s) below it, then set the "order_masters" setting to True.
#
# If this is a master that will be running a syndic daemon for passthrough, then
# the "syndic_master" setting needs to be set to the location of the master server
# to receive commands from.
# Set the order_masters setting to True if this master will command lower
# masters' syndic interfaces.
#order_masters: False
#order_masters: True
# If this master will be running a salt syndic daemon, syndic_master tells
# this master where to receive commands from.
#syndic_master: masterofmaster