salt icon indicating copy to clipboard operation
salt copied to clipboard

Published 20 hours ago • verified publisher icon saltstack

[BUG] 3007.4 minion doesn't connect to 3007.3 master

Open lee-harmonic opened this issue 5 months ago • 8 comments

Description

Upgrade salt deb packages on minion computer to 3007.4 (latest available currently) and on restarting the salt-minion service, it doesn't connect to the master.

Logs:

Jun 13 12:15:21 chorale salt-minion[436232]: [ERROR   ] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncI>
Jun 13 12:15:21 chorale salt-minion[436232]: Traceback (most recent call last):
Jun 13 12:15:21 chorale salt-minion[436232]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/ioloop.py", line 750, in _run_callback
Jun 13 12:15:21 chorale salt-minion[436232]:     ret = callback()
Jun 13 12:15:21 chorale salt-minion[436232]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/ioloop.py", line 774, in _discard_future_result
Jun 13 12:15:21 chorale salt-minion[436232]:     future.result()
Jun 13 12:15:21 chorale salt-minion[436232]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/gen.py", line 779, in run
Jun 13 12:15:21 chorale salt-minion[436232]:     yielded = self.gen.throw(exc)
Jun 13 12:15:21 chorale salt-minion[436232]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 821, in _authenticate
Jun 13 12:15:21 chorale salt-minion[436232]:     creds = yield self.sign_in(channel=channel)
Jun 13 12:15:21 chorale salt-minion[436232]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/gen.py", line 766, in run
Jun 13 12:15:21 chorale salt-minion[436232]:     value = future.result()
Jun 13 12:15:21 chorale salt-minion[436232]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/gen.py", line 785, in run
Jun 13 12:15:21 chorale salt-minion[436232]:     yielded = self.gen.send(value)
Jun 13 12:15:21 chorale salt-minion[436232]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 961, in sign_in
Jun 13 12:15:21 chorale salt-minion[436232]:     ret = self.handle_signin_response(sign_in_payload, payload)
Jun 13 12:15:21 chorale salt-minion[436232]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1002, in handle_signin_response
Jun 13 12:15:21 chorale salt-minion[436232]:     payload["session"], self.opts["encryption_algorithm"]
Jun 13 12:15:21 chorale salt-minion[436232]: KeyError: 'session'
Jun 13 12:16:20 chorale salt-minion[436232]: [ERROR   ] Minion unable to successfully connect to a Salt Master.

Setup

  • [ x ] VM (KVM, but likely any deb installed instance, and maybe wider)
  • [ x ] onedir (I think)
    • deb [signing-etc] https://packages.broadcom.com/artifactory/saltproject-deb/ stable main

Steps to Reproduce the behavior Take existing salt network, upgrade salt-minion.

Expected behavior Connect to master.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Updated minion

Salt Version:
          Salt: 3007.4
 
Python Version:
        Python: 3.10.17 (main, Jun  9 2025, 20:41:48) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.8.0
  cryptography: 42.0.5
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.19.3
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.4.2
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 13.0 trixie
        locale: utf-8
       machine: x86_64
       release: 6.12.25-amd64
        system: Linux
       version: Debian GNU/Linux 13.0 trixie

Master

Salt Version:
          Salt: 3007.3
 
Python Version:
        Python: 3.10.17 (main, May 11 2025, 04:07:13) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.8.0
  cryptography: 42.0.5
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.19.2
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.4.2
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 12.11 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.1.0-35-amd64
        system: Linux
       version: Debian GNU/Linux 12.11 bookworm

lee-harmonic avatar Jun 13 '25 01:06 lee-harmonic

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

welcome[bot] avatar Jun 13 '25 01:06 welcome[bot]

We ran into this and after upgrading our salt-master to 3007.4 the error went away.

jlrcontegix avatar Jun 13 '25 01:06 jlrcontegix

I can confirm that updating the master fixed this issue. It may still be a bug, I would expect at least some level of compatibility. (Though it is a security update, and there is no news item about it yet, so could relate to that).

lee-harmonic avatar Jun 13 '25 02:06 lee-harmonic

This regression also affects the stable series. I can reproduce with Saltstack 3006.12 on the minion and an older version than 3006.12 on the master.

The release notes for 3006.12 and 3007.4 do not mention the requirement to update the master before updating the minion: https://docs.saltproject.io/en/3006/topics/releases/3006.12.html https://docs.saltproject.io/en/3007/topics/releases/3007.4.html

Log with a 3006.12 minion:

Jun 12 22:06:47 server12 salt-minion[1019]: [ERROR   ] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f5e97896560>, <salt.ext.tornado.concurrent.Future object at 0x7f5e97977f40>)
Jun 12 22:06:47 server12 salt-minion[1019]: Traceback (most recent call last):
Jun 12 22:06:47 server12 salt-minion[1019]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
Jun 12 22:06:47 server12 salt-minion[1019]:     ret = callback()
Jun 12 22:06:47 server12 salt-minion[1019]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Jun 12 22:06:47 server12 salt-minion[1019]:     return fn(*args, **kwargs)
Jun 12 22:06:47 server12 salt-minion[1019]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
Jun 12 22:06:47 server12 salt-minion[1019]:     future.result()
Jun 12 22:06:47 server12 salt-minion[1019]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Jun 12 22:06:47 server12 salt-minion[1019]:     raise_exc_info(self._exc_info)
Jun 12 22:06:47 server12 salt-minion[1019]:   File "<string>", line 4, in raise_exc_info
Jun 12 22:06:47 server12 salt-minion[1019]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Jun 12 22:06:47 server12 salt-minion[1019]:     yielded = self.gen.throw(*exc_info)
Jun 12 22:06:47 server12 salt-minion[1019]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 745, in _authenticate
Jun 12 22:06:47 server12 salt-minion[1019]:     creds = yield self.sign_in(channel=channel)
Jun 12 22:06:47 server12 salt-minion[1019]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Jun 12 22:06:47 server12 salt-minion[1019]:     value = future.result()
Jun 12 22:06:47 server12 salt-minion[1019]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Jun 12 22:06:47 server12 salt-minion[1019]:     raise_exc_info(self._exc_info)
Jun 12 22:06:47 server12 salt-minion[1019]:   File "<string>", line 4, in raise_exc_info
Jun 12 22:06:47 server12 salt-minion[1019]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
Jun 12 22:06:47 server12 salt-minion[1019]:     yielded = self.gen.send(value)
Jun 12 22:06:47 server12 salt-minion[1019]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 885, in sign_in
Jun 12 22:06:47 server12 salt-minion[1019]:     ret = self.handle_signin_response(sign_in_payload, payload)
Jun 12 22:06:47 server12 salt-minion[1019]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 926, in handle_signin_response
Jun 12 22:06:47 server12 salt-minion[1019]:     payload["session"], self.opts["encryption_algorithm"]
Jun 12 22:06:47 server12 salt-minion[1019]: KeyError: 'session'
Jun 12 22:07:47 server12 salt-minion[1019]: [ERROR   ] Minion unable to successfully connect to a Salt Master.

hailfinger avatar Jun 13 '25 06:06 hailfinger

I can confirm this problem with a 3006.9 salt-master and a 3006.12 salt-minion. We try to update soon. Right now we pinned to 3006.11 for salt minions.

christophlsa avatar Jun 13 '25 07:06 christophlsa

Same here for 3006.10 or 3006.11 master and 3006.12 minion. Pinning to 3006.11, too.

MartinEmrich avatar Jun 13 '25 13:06 MartinEmrich

Just a me too: ubuntu 24.04, minion 3007.4 saltmaster 3007.2 Same error.

Johanhen avatar Jun 13 '25 17:06 Johanhen

beware of upgrading the master if you use gitfs... https://github.com/saltstack/salt/issues/68072

ITJamie avatar Jun 16 '25 09:06 ITJamie

@lee-harmonic can you please update the bug title to include 3006.12 as well?

hailfinger avatar Jun 16 '25 21:06 hailfinger

We also have problems since 3007.4 is out . Minions running 3007.4 can no longer communicate with our SLES salt-master running 3006.0-150500.4.50.3. Switching the minions back to 3007.3 everything is fine again.

rainer042 avatar Jun 25 '25 08:06 rainer042

The same issue with version 3007.5.

hadret avatar Jun 27 '25 06:06 hadret

We too encountered this issue when updating to salt-minion 3006.12 with salt-master 3006.9. Issue persists with salt-minion 3006.13.

tmainzer72 avatar Jun 27 '25 09:06 tmainzer72

has anyone found that it's been fixed in ? 3007.6 running Raas

Geoffmoss77 avatar Jul 10 '25 11:07 Geoffmoss77

omg, why is this bug still not fixed?This is a super critical bug

rimskij avatar Jul 17 '25 12:07 rimskij

From the documentation on upgrading:

When upgrading Salt, the master(s) should always be upgraded first. Running minions with versions of Salt newer than their masters is not guaranteed to function as expected since the minion may include changes not yet available in the master. Also, whenever possible, backward compatibility between new masters and old minions will be preserved. Generally, the only exception to this policy is in case of a security vulnerability.

clayoster avatar Jul 17 '25 14:07 clayoster

The problem is that for a long while, as long as I use salt things were comaptible. We used SLES Salt-Server and Ubuntu and SLES minions. I used salt first on SLES thats why the master became an SLES system with 3006er Server and ubuntu was until recently capabable of using this LTS master as well. Now suddenly this no longer works with current minions that are available for ubuntu but are not available for SLES. So I cannot upgrade the master first. An installtion that worked for years suddenly breaks completely, which is certainly not good. What we did as an initial fix was to pin ubuntu minions to an older version 3007.4 that are still able to talk to the 3006 LTS master server so things work for now. But this is of course not a real solution.

Am 17.07.25 um 16:37 schrieb Clay Oster:

clayoster left a comment (saltstack/salt#68071) <https://github.com/ saltstack/salt/issues/68071#issuecomment-3084331037>

From the documentation on upgrading <https://docs.saltproject.io/salt/ install-guide/en/latest/topics/upgrade.html>:

When upgrading Salt, the master(s) should always be upgraded first.
Running minions with versions of Salt newer than their masters is
not guaranteed to function as expected since the minion may include
changes not yet available in the master. Also, whenever possible,
backward compatibility between new masters and old minions will be
preserved. Generally, the only exception to this policy is in case
of a security vulnerability.

— Reply to this email directly, view it on GitHub <https://github.com/ saltstack/salt/issues/68071#issuecomment-3084331037>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ AETIPJQPKQ2ZQHOCQ5U73VT3I6YLPAVCNFSM6AAAAAB7GZYO7OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAOBUGMZTCMBTG4>. You are receiving this because you commented.Message ID: <saltstack/ @.***>

rainer042 avatar Jul 17 '25 14:07 rainer042

I agree. As we are on opinions now, here's mine:

While hiding behind the "upgrade master first" documentation might be formally correct, it is not what most of us expect from a patch release (3006.11 -> 3006.12) in a "LTS" branded branch. It would "feel more valid" if from 3006.x to 3007.x

If there was a major security hole in the 3006.11 protocol, it should have been announced, but put behind a configuration switch on minion and master and default should be the old behaviour or autodetect for several versions.

But no one has until now explained what actually changed, there was no error message "you are running a too-old master, upgrade master first", just some random python backtrace.

running "apt-get dist-upgrade", "yum update" or equvalent on a minion should never cut you off the branch you sit on.

MartinEmrich avatar Jul 17 '25 15:07 MartinEmrich

In between SUSE released a new salt-version: 3006.0-150500.4.55.1 . This salt version is especially also able to talk to Ubuntu salt-Version 3007.6 as well as 3007.3 . So our master can now talk to SLES minions (same version like new master) as well as to the Ubuntu minions.
Thanks for all who helped to get here

rainer042 avatar Jul 25 '25 06:07 rainer042

Though connecting to an older master may work sometimes, it is not guaranteed. Salt guidance is that the master must be >= minion.

twangboy avatar Aug 14 '25 21:08 twangboy