minion log get A message with an invalid serial was receive

[kokorobin]

Description of Issue When I use the salt python api to send asynchronous commands to some minion, errors in the title often occur. But at this time, salt test.ping is normal. When I try again. The command was sent normally again

I cannot reproduce the problem 100% because it seems to occur randomly. And the probability of occurrence in my cluster is a little high

The mionion error log is as follows

Versions Report

master version
Salt Version:
          Salt: 3004.2
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.3
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4
 
System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.71.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

**salt minon version**
Salt Version:
          Salt: 3004.1
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.3
       libgit2: Not Installed
      M2Crypto: 0.38.0
          Mako: Not Installed
       msgpack: 1.0.3
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.11.0
        pygit2: Not Installed
        Python: 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0]
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 22.3.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04 Jammy Jellyfish
        locale: utf-8
       machine: x86_64
       release: 5.15.0-48-generic
        system: Linux
       version: Ubuntu 22.04 Jammy Jellyfish

[welcome[bot]]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[kokorobin]

I checked the source code and found that the module file of the error log is in salt/crypt.py, Commits on Feb 8, 2022. Modify the record as Job replay mission, but I don't know why my legal master will be judged invalid，which makes me very frustrated

[korjek]

same error here, but on 3005.1

[darkpixel]

I'm running in to this on 3007.1. The error pops up on the minion that runs on the salt master and is in charge of signing x509 certs. If I go highstate on ~30 minions that all need certificate requests, 5-10 of them will hang at x509.pem_managed while waiting on a reply from the master's minion while the master's minion displays this error.

EDIT: When this starts happening, I have to restart the minion on the master and then more of them will succeed--maybe 2 or 3 will fail hang at signing. Then I restart a third time and the rest will finish.