saltstack
[BUG] pam eauth fails with salt-master/salt-api running as user salt on 3006.x
Description
Running the salt-master process under the salt user breaks pam eauth as the salt user can't read /etc/shadow and /etc/gshadow
This issue is documented in SEP19 and Issue 7762
Setup
- Salt 3006.x installed from Debian packages
- Same issue will exist with RPM packages but I haven't specifically tested
Steps to Reproduce the behavior
Configure the salt master to use pam eauth -
$ cat /etc/salt/master.d/ext_auth.conf
external_auth:
pam:
testuser:
- '.*'
$ sudo salt -a pam --username=testuser \* test.ping
password: <password>
Authentication error occurred.
Expected behavior
$ sudo salt -a pam --username=testuser \* test.ping
password: <password>
testhost:
True
Versions Report
$ salt --versions-report
Salt Version:
Salt: 3006.1
Python Version:
Python: 3.10.11 (main, May 5 2023, 02:31:54) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 23.2.0
relenv: 0.12.3
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.
ZMQ: 4.3.4
System Versions:
dist: ubuntu 20.04.6 focal
locale: utf-8
machine: x86_64
release: 5.4.0-148-generic
system: Linux
version: Ubuntu 20.04.6 focal
Additional context
This was a known issue with running with the salt-master as a non-root user - documented in SEP19, issue 61770 and has been called out in the eauth documentation for a long time...
There's a simple fix for Debian derived distributions - adding the salt user to the shadow group allows the salt-master process to read /etc/shadow and resolves
the issue. This doesn't work for RH derived distros.
I'd suggest that for Debian derivatives whether to add salt to the shadow group should be asked through debconf - that way folk that need to use eauth/pam can do so but those who don't, don't expose the content of /etc/shadow to the salt user.
I'm not sure how to easily handle this to RPM packages as I'm less familiar with the options there.
Best long term solution would be to have a small separate privileged process for pam eauth that the salt master queries for authentication.
so, right now this might be more of a documentation issue. if you need pam, change the user back to root. which can be done in /etc/salt/master as blindly allowing access to shadow to a non root user by default is a huge nono, and having the user make a change to the system forces knowledge onto the user that pam needs shadow access.
but i like the idea of a separate process for authentication, but that becomes more of a feature request for that part.
so, right now this might be more of a documentation issue. if you need pam, change the user back to root. which can be done in
/etc/salt/masteras blindly allowing access to shadow to a non root user by default is a huge nono, and having the user make a change to the system forces knowledge onto the user that pam needs shadow access.
It was a documentation issue for the 3006.0 release. Now, it's a breaking change that wasn't handled or communicated.
I'm also going to disagree that users that need to use pam eauth are better off running the master as root rather than adding the salt user to the shadow group.. with the former, you've given the salt-master process the ability to do anything as root, with the latter you have only given it the ability to read /etc/shadow and /etc/gshadow.
I think requiring (rather than forcing) knowledge on to the user that pam needs shadow access is better than requiring them to have no understanding and just defaulting to running stuff as root.
Asking through debconf if a user that runs a daemon should be given additional privilege to enable a feature is not unusual and is perfectly reasonable.
I guess for the moment I'll work on some docs to cover this.
but i like the idea of a separate process for authentication, but that becomes more of a feature request for that part.
Yeah, I think that is probably the way to go longer term as will work across the various Linux distros. I'll have a think about how to do that.
thank goodness I am using debian, otherwise the last 4 hours of my life would have been for naught!
Why is salt trying to read the group file at all? Shouldn't it being going through the nss interface in the same way that getent does?
getent group salt
PAM+NSS is nice because it can transparently handle NIS/NIS+, LDAP, ActiveDirectory, etc such that all standard libc calls still work. Tools such as getent, id, passwd transparently work for any of these services w/out any special configurations. Why is salt doing something non-standard here?
@major0 As far as I see it, Salt is already using PAM (and most likely, at least implicitly through libc, NSS). But that's not a silver bullet. If you run as a non-root user, getent shadow should not work. That's the whole point of the shadow database: move the (sensible) password hashes to a separate, root-only database.
If a process should be able to password-authenticate users without access to the password hashes (or even the raw passwords), this would have to be delegated to a trusted service (like SASL). (But please, don't do SASL, IMHO that's a dinosaur and a pain in the rearwards-facing-end to configure).
I could imagine a salt-master root process taking care of authentication, while the more "salty" processes (Jobs, Highstate, etc.) runs as a non-root user.
@major0 As far as I see it, Salt is already using PAM (and most likely, at least implicitly through libc, NSS). But that's not a silver bullet. If you run as a non-root user,
getent shadowshould not work. That's the whole point of the shadow database: move the (sensible) password hashes to a separate, root-only database.
While I agree with all of your statements about the shadow database, I am not entirely certain how we went from getent group <group>, which works w/out root, to getent shadow which I agree shouldn't work and requires root.
I must be missing something, because looking to see if someone is a member of a group does not use getent shadow, and I am not certain why it was even mentioned. Are we saying that the code ignores the regular group database used by every tool from ls (gid to name translation) to id (group memberships) and instead only inspects the shadow database for group membership? That seems a pretty odd design choice.
@major0 As far as I can read the conversation here, the issue is not about checking group membership, but about authenticating users, and for that, the salt master (or rather the chain salt-master -> libpam -> pam_unix) needs, at some point, read access to /etc/shadow, as there are the password hashes.
But digging deeper (see pam(7), pam_unix(8), and unix_chkpwd(8)), pam_unix seems to come prepared for usage as non-root, and giving xlock as a well-known example. This makes me wonder (for academic reasons*) if salt really uses libpam, or just calls the auth component "pam"?
( *) for practical reasons, I am fine running salt-master as root on a VM with no other purpose.)
I was just curious:
salt indeed loads the actual libpam library, and uses (by default) the pam "login" service: https://github.com/saltstack/salt/blob/master/salt/auth/pam.py#L229
and (at least here on Debian 12), my /etc/pam.d/login contains a lot more stuff, which might indeed only work if run as root...
Found the cause, described here: https://github.com/linux-pam/linux-pam/issues/112#issuecomment-491193418
tl;dr, as I understand it: Non-root-Users can only validate themselves. So if salt ran as user "salt", only the user "salt" could authenticate via PAM.
What happens? Salt runs a subprocess calling libpam functions. libpam in turn runs its unix_chkpwd binary with that behaviour, which returns an authentication failure when it cannot read the shadow file.
IMHO this is not a bug in Salt, but rather a result of various security design choices in libpam etc.
@major0 - this isn't about getting group info, this is about being able to authenticate a user.
@MartinEmrich your understanding is correct, without doing something such as adding the salt user to the shadow group, it's only going to be able to auth itself with the pam "login" service.
Whilst not necessarily a "bug" in salt, it's a "breaking change" in the packaging for 3006.x when the default user that the salt-master process runs as was changed to salt without handling this or documenting it.
I'd also like to actually resolve this issue in a way that isn't just "run the salt-master as root if you want to use pam eauth" - the most common use case I come across for using pam eauth is when using salt-api - that's essentially exposing the salt-master on the network.. which is exactly the sort of situation where we should be running as an un-privileged user.
Hey guys this is still a problem in 3006.9 right ? I'm unable to login to Salt API on a master running as a non-root user
Are these 'pam.acl' is not available and 'pam.process_acl' is not available log messages possible errors (instead of debug messages ?
2024-09-09 15:08:20,950 [salt.utils.lazy :101 ][DEBUG ][19135] Could not LazyLoad pam.acl: 'pam.acl' is not available.
2024-09-09 15:08:20,951 [salt.utils.lazy :101 ][DEBUG ][19135] Could not LazyLoad pam.process_acl: 'pam.process_acl' is not available.
2024-09-09 15:08:20,952 [salt.master :2269][WARNING ][19135] Authentication failure of type "eauth" occurred.
eapi pam authentication started to work after we found this line in salt\pam.py
https://github.com/saltstack/salt/blob/v3006.9/salt/auth/pam.py#L231C4-L231C88
and we set auth.pam.python on the master to the local python installation used on this setup
auth.pam.python: /local/data/salt-master/3006.9/bin/python3