[BUG] Unprivileged users in salt 3006 can no longer run salt CLI on a salt master

[david-pulkowski]

Description Unprivileged users can no longer run salt CLI on a salt master. See error message in the Setup. We are upgrading our masters from salt 3004.2 (classic) to 3006.3 (onedir). The upgrade went fine, except the case for unprivileged users running salt commands on the master. This worked fine in 3004 where we ran salt systemd as root and had automated tasks that ran with unprivileged user that was restricted by publisher ACL configuration.

Setup This can be a simple master/minion setup. Then as the root user run a test.ping

salt minion01 test.ping

minion01:
    True 

Run as an unprivileged user.

salt minion01 test.ping
[ERROR   ] Encountered StreamClosedException [salt.utils.event ] [94300]
[ERROR   ]  [salt.client      ] [94300]
Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.

For testing purposes, we added the unprivileged user to the master.conf as the user to run the salt-master systemd:

user: <unprivileged user> 

The unprivileged user was then able to run the processes with no errors. (this is not the prefered setup but we did this for investigating permission issues) alc.conf details:

publisher_acl:
  <unprivileged user>:
    - test.*
    - state.apply
    - state.show_sls
    - service.restart
    - service.stop
    - service.status
    - service.start
    - grains.item
    - pkg.list_upgrades
    - cmd.run 

• [ ] on-prem machine
• [X ] VM (Virtualbox, KVM, etc. please specify)
• [ ] VM running on a cloud service, please be explicit and add details
• [ ] container (Kubernetes, Docker, containerd, etc. please specify)
• [ ] or a combination, please be explicit
• [ ] jails if it is FreeBSD
• [X ] classic packaging
• [X ] onedir packaging
• [ ] used bootstrap to install

Steps to Reproduce the behavior Do as above. Setup a master/minion in 3006 & see if running as an unprivlaged user still works.

Expected behavior In 3006 we expect the user to be able to successfully run the commands as it is authorized to. If additional permissions need to be set with switching to salt-ondir, we would like the documentation updated.

Screenshots NA

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.3

Python Version:
        Python: 3.10.13 (main, Sep  6 2023, 02:11:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.10
     gitpython: 3.1.36
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.10
         smmap: 5.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rhel 7.9 Maipo
        locale: utf-8
       machine: x86_64
       release: 3.10.0-1160.99.1.el7.x86_64
        system: Linux
       version: Red Hat Enterprise Linux Server 7.9 Maipo</summary>

Additional context Using --async did not provide any additional information.

[welcome[bot]]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[broxio]

Just add in the log from /var/log/syslog it's running good on 3006.1 and once accidentally upgrade package with latest tag repo to 3006.3 then no longer able to run CLI with non-root user.

Sep 13 23:36:09 master salt-master[501487]: No permissions to access "/var/log/salt/master", are you running as the correct user?

[OrangeDog]

This is probably a duplicate of #64275 though more details should be in your master log.

As the user changed from root to salt, you need to migrate permissions on existing files manually if you're upgrading, or change it back to root.

[david-pulkowski]

Thanks for the response. There information in the master logfile even changing to debug is as seen below

[ERROR   ] Encountered StreamClosedException [salt.utils.event ] [75237]
[ERROR   ]  [salt.client      ] [75237]
Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.

Same output as seen in the cli.

I changed permissions from salt to root on the /etc/salt/master.d directory & the /var/cache/salt/master directories and still get the same error.

[OrangeDog]

Either you didn't change the master to log at debug, or that's not the master log.

I changed permissions from salt to root

That's the opposite of what I said.

Did you look at the duplicate and try the workaround there?

[david-pulkowski]

Sorry, I did misread and that changed the permissions from root to salt

I did read that bug report, and we are not using it the same way, and we are on rhel, as they indicate their workaround was for Debian and not RHEL.

Logging is set to debug:

cat /etc/salt/master.d/logging.conf
log_level: debug

Master log after running a simple job.

2023-09-18 14:52:54,163 [salt.utils.event ] [DEBUG   ] Flushing 1 events.
2023-09-18 14:52:54,163 [salt.utils.event ] [DEBUG   ] Calling event returner pgjsonb, only one configured.
2023-09-18 09:53:03,016 [salt.utils.parsers] [DEBUG   ] Configuration file path: /etc/salt/master
2023-09-18 09:53:03,017 [salt.config      ] [DEBUG   ] Missing configuration file: /etc/salt/master
2023-09-18 09:53:03,017 [salt.config      ] [DEBUG   ] Including configuration from '/etc/salt/master.d/acl.conf'
2023-09-18 09:53:03,017 [salt.config      ] [DEBUG   ] Reading configuration from /etc/salt/master.d/acl.conf
2023-09-18 09:53:03,018 [salt.config      ] [DEBUG   ] Including configuration from '/etc/salt/master.d/environment-options.conf'
2023-09-18 09:53:03,018 [salt.config      ] [DEBUG   ] Reading configuration from /etc/salt/master.d/environment-options.conf
2023-09-18 09:53:03,030 [salt.config      ] [DEBUG   ] Including configuration from '/etc/salt/master.d/fileserver.conf'
2023-09-18 09:53:03,030 [salt.config      ] [DEBUG   ] Reading configuration from /etc/salt/master.d/fileserver.conf
2023-09-18 09:53:03,030 [salt.config      ] [DEBUG   ] Including configuration from '/etc/salt/master.d/gitfs-remotes.conf'
2023-09-18 09:53:03,030 [salt.config      ] [DEBUG   ] Reading configuration from /etc/salt/master.d/gitfs-remotes.conf
2023-09-18 09:53:03,033 [salt.config      ] [DEBUG   ] Including configuration from '/etc/salt/master.d/logging.conf'
2023-09-18 09:53:03,033 [salt.config      ] [DEBUG   ] Reading configuration from /etc/salt/master.d/logging.conf
2023-09-18 09:53:03,033 [salt.config      ] [DEBUG   ] Including configuration from '/etc/salt/master.d/options.conf'
2023-09-18 09:53:03,033 [salt.config      ] [DEBUG   ] Reading configuration from /etc/salt/master.d/options.conf
2023-09-18 09:53:03,035 [salt.config      ] [DEBUG   ] Including configuration from '/etc/salt/master.d/pgjsonb_returner.conf'
2023-09-18 09:53:03,035 [salt.config      ] [DEBUG   ] Reading configuration from /etc/salt/master.d/pgjsonb_returner.conf
2023-09-18 09:53:03,035 [salt.config      ] [DEBUG   ] Including configuration from '/etc/salt/master.d/winrepo.conf'
2023-09-18 09:53:03,036 [salt.config      ] [DEBUG   ] Reading configuration from /etc/salt/master.d/winrepo.conf
2023-09-18 09:53:03,036 [salt.config      ] [DEBUG   ] Including configuration from '/etc/salt/master.d/ztesting-debugging.conf'
2023-09-18 09:53:03,036 [salt.config      ] [DEBUG   ] Reading configuration from /etc/salt/master.d/ztesting-debugging.conf
2023-09-18 09:53:03,037 [salt.config      ] [DEBUG   ] Using cached minion ID from /etc/salt/minion_id: minion01
2023-09-18 09:53:03,038 [salt.config      ] [DEBUG   ] Missing configuration file: /home/svc-USER/.saltrc
2023-09-18 09:53:03,040 [salt.utils.event ] [DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
2023-09-18 09:53:03,040 [salt.utils.event ] [DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
2023-09-18 09:53:13,080 [salt.utils.event ] [ERROR   ] Encountered StreamClosedException
2023-09-18 09:53:13,080 [salt.client      ] [ERROR   ]
2023-09-18 09:53:13,080 [salt.channel.client] [DEBUG   ] Closing AsyncReqChannel instance
2023-09-18 09:53:13,088 [salt.loader.lazy ] [DEBUG   ] The functions from module 'nested' are being loaded by dir() on the loaded module

We never had a .saltrc file set previously for this user.

[broxio]

Quick add workaround for solved the permission issue , in my case when I start upgrade salt-master from 3006.1 -> 3006.4 -> 3006.7 , for our salt-master has been use non-root user since 3006.1 and found out permission on /var/log/salt is change back to root

drwxr-s---   2 root      root                 4096 Mar 17 12:00 salt

then I do change permission  from root to salt on /var/log/salt 

#sudo chown salt.salt /var/log/salt

drwxr-s---   2 salt      salt                 4096 Mar 17 12:00 salt

restart salt-master service and everything back in shape.

[drexljo]

Problem is as follows, at least with our setup:

Run 'salt -v -l trace test.ping' shows it: ... [DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc [DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc [TRACE ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc [TRACE ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc [TRACE ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc [TRACE ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc [TRACE ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc [TRACE ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc [ERROR ] Encountered StreamClosedException [ERROR ] [DEBUG ] Closing AsyncReqChannel instance ...

ls -lah shows: srw-rw---- 1 salt salt 0 Sep 23 14:01 /var/run/salt/master/master_event_pub.ipc

As the executing user is not part of the salt group, there's a permission denied on the socket and that's it. The master service never gets any request.

Quick solution: Add your user to the salt group, re-login and go.

[dmurphy18]

@david-pulkowski Recommend upgrading to latest version of Salt 3006.9, as it has additional fixes for non-root.

[david-pulkowski]

On Salt 3006.9 and errors gone now. Thanks for when the fix was implemented!