[salt-ssh] No module named 'salt.exceptions'

[gray380]

Description of Issue

Hi,

I have three identical nodes (lets say agentless minions). One of nodes stops to execute calls like this: salt-ssh 'one-*' grains.get 'kernelrelease'

the following error has appeared:

    _error:
        Failed to return clean data
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.sadm_2296c1_salt/salt-call", line 26, in <module>
            from salt.scripts import salt_call
          File "/var/tmp/.sadm_2296c1_salt/pyall/salt/scripts.py", line 21, in <module>
            from salt.exceptions import SaltSystemExit, SaltClientError, SaltReqTimeoutError
        ModuleNotFoundError: No module named 'salt.exceptions'

cmd-run call also stops to execute but "salt-ssh -r ..." executes successfully.

The rest two nodes are okay. Nodes (minions): CentOS Linux release 7.6.1810 (Core)

Could you help to solve this issue?

Versions Report

Master

Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Jun 24 2019, 04:54:02)
   python-gnupg: Not Installed
         PyYAML: 5.1.1
          PyZMQ: Not Installed
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: Not Installed
 
System Versions:
           dist: arch Manjaro Linux
         locale: UTF-8
        machine: x86_64
        release: 4.19.62-1-MANJARO
         system: Linux
        version: arch Manjaro Linux

[gray380]

Just to be clear. Pay attention for the one-srv-01.

Successfull result:

$ salt-ssh 'one-*' -r 'python3 --version'
one-kvm-node-03:
    ----------
    retcode:
        0
    stderr:
    stdout:
        Python 3.6.8
one-kvm-node-02:
    ----------
    retcode:
        0
    stderr:
    stdout:
        Python 3.6.8
one-srv-01:
    ----------
    retcode:
        0
    stderr:
    stdout:
        Python 3.6.8

An error happened:

$ salt-ssh 'one-*' cmd.run 'python3 --version'
one-srv-01:
    ----------
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.sadm_2296c1_salt/salt-call", line 26, in <module>
            from salt.scripts import salt_call
          File "/var/tmp/.sadm_2296c1_salt/pyall/salt/scripts.py", line 21, in <module>
            from salt.exceptions import SaltSystemExit, SaltClientError, SaltReqTimeoutError
        ModuleNotFoundError: No module named 'salt.exceptions'
    stdout:
one-kvm-node-02:
    Python 3.6.8
one-kvm-node-03:
    Python 3.6.8

[twangboy]

Could you try salt-ssh -t to regenerate the tarball?

[gray380]

Output with '-t' applied:

salt-ssh -t 'one-*' cmd.run 'python3 --version'
[ERROR   ] Failed collecting tops for Python binary python2.
one-kvm-node-03:
    Python 3.6.8
one-srv-01:
    ----------
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.sadm_2296c1_salt/salt-call", line 26, in <module>
            from salt.scripts import salt_call
          File "/var/tmp/.sadm_2296c1_salt/pyall/salt/scripts.py", line 21, in <module>
            from salt.exceptions import SaltSystemExit, SaltClientError, SaltReqTimeoutError
        ModuleNotFoundError: No module named 'salt.exceptions'
    stdout:
one-kvm-node-02:
    Python 3.6.8

[gray380]

Weird... but it works with 'python':

$ salt-ssh -t 'one-*' cmd.run 'python --version'
[ERROR   ] Failed collecting tops for Python binary python2.
one-kvm-node-02:
    Python 2.7.5
one-kvm-node-03:
    Python 2.7.5
one-srv-01:
    Python 2.7.5

[twangboy]

@saltstack/team-ssh Could someone take a look at this?

[gray380]

Time passed and everything works again. AFAIK no changes have been applied to the systems. Some kind of space rays issue ;)

$ salt-ssh 'one*' grains.get 'kernelrelease'
one-kvm-node-02:
    3.10.0-957.27.2.el7.x86_64
one-kvm-node-03:
    3.10.0-957.21.3.el7.x86_64
one-srv-01:
    3.10.0-957.27.2.el7.x86_64

[waynew]

Theory: you had something funky cached somewhere and it's gone now.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[meaksh]

FYI we have some reports from customers that are facing this same issue but running Salt 3000.

Apparently, removing the cached "thin" in the ssh minion (i.a. /var/tmp/.sadm_2296c1_salt/) fixes the issue, but it happens again once the "thin" is cached.

Let see if we can bring some more light into this.

[hartwork]

Seen with 3001.1 with a Fedora 32 target as well:

# salt-ssh '*' test.ping
host1:
    ----------
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.root_XXXXXX_salt/salt-call", line 26, in <module>
            from salt.scripts import salt_call
        ModuleNotFoundError: No module named 'salt.scripts'
    stdout:
host2:
    True

It seems to have been caused by a system-wide upgrade from Python 3.7 to 3.8 on the managed box. rm -R /var/tmp/.root_??????_salt/ worked well for a workaround.

[ZackMattor]

Had to remove the tmp file as other have stated... needs to be done on the remote target.

[neighbour-oldhuang]

you should exec salt-ssh -w xxx to remove dist host cache, then you can exec your cmd next time, but i am not found how to resolve it completely