[BUG] salt-master reports authentication failed after pam reports auth success

[major0]

Description When using curl against the salt-api endpoint:

• PAM reports that login:auth authentication success.
• salt-master reports: salt.master :2260][WARNING ][760504] Authentication failure of type "eauth" occurred.
• cherrypy reports: salt.loaded.int.netapi.rest_cherrypy.app:867 ][ERROR ][660943] Authorization error occurred.

Setup

saltmaster is configured with:

auth.pam.service: salt

external_auth:
  pam:
    my group%: # yes there is a space in the group name
      - '.*'
      - '@runner'
      - '@wheel'
      - '@jobs'

rest_cherrypy:
  port: 443
  disable_ssl: False
  ssl_crt: /etc/pki/salt/certs/saltmaster.crt
  ssl_key: /etc/pki/salt/certs/saltmaster.key

netapi_enable_clients:
  - local
  - local_async
  - local_batch
  - local_subset
  - runner
  - runner_async
  - wheel
  - wheel_async

Please be as specific as possible and give set-up details.

• [ ] on-prem machine
• [ ] VM (Virtualbox, KVM, etc. please specify)
• [X] VM running on a cloud service, please be explicit and add details. Ubuntu 22.04 on an AWS EC2 x86_64 instance
• [ ] container (Kubernetes, Docker, containerd, etc. please specify)
• [ ] or a combination, please be explicit
• [ ] jails if it is FreeBSD
• [ ] classic packaging
• [X] onedir packaging
• [X] used bootstrap to install

Steps to Reproduce the behavior (Include debug logs if possible and relevant)

Copy/Pasted/Modified curl command from: https://docs.saltproject.io/en/latest/topics/netapi/netapi-enable-clients.html

curl -sSkKi https://localhost/run \
    -H 'Accept: application/x-yaml' \
    -d client='local' \
    -d tgt='*' \
    -d fun='test.ping' \
    -d username='myuser' \
    -d password='mypass' \
    -d eauth='pam'

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.1
 
Python Version:
        Python: 3.10.11 (main, May  5 2023, 02:31:54) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: 1.1.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.6.1
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.12.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04.2 jammy
        locale: utf-8
       machine: x86_64
       release: 5.19.0-1029-aws
        system: Linux
       version: Ubuntu 22.04.2 jammy

Additional context I have set auth.pam.service: salt so that I can try to debug this more by modifying the PAM configs.

[whytewolf]

this looks like a duplicate of https://github.com/saltstack/salt/issues/64275

[major0]

I tried adding salt to the shadow group, which did not work.

I set user: root in the salt master config, which also did not work.

Again, PAM claims auth succeeded, so the master is calling into PAM correctly and validating authentication, but then the master returns Authentication error after PAM claims authentication success.

[major0]

I am getting the impression that Salt isn't doing something like the following:

from grp import getgrnam

def memberOf(group, user):
  data = getgrnam(group)    
  for member in data.gr_mem:
    if user == member:
      return True
  return False

Since this would not require any special permissions, would work with NSS abstraction (and this NIS/NIS+, LDAP, ActiveDirectory, IPA, etc..).

[major0]

I have subsequently done salt-pip install pysss and now have an extra error.

==> /var/log/auth.log <==
Jul 27 21:30:42 ip-10-50-10-219 python3.10: pam_unix(salt:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=  user=major0
Jul 27 21:30:42 ip-10-50-10-219 python3.10: pam_sss(salt:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=major0
Jul 27 21:30:42 ip-10-50-10-219 python3.10: pam_sss(salt:account): Access denied for user major0: 6 (Permission denied)

==> /var/log/salt/master <==
2023-07-27 21:30:42,592 [salt.loaded.int.auth.pam:254 ][ERROR   ][855866] Pam auth failed for major0:  
2023-07-27 21:30:43,387 [salt.auth        :313 ][WARNING ][855866] Authentication failure of type "eauth" occurred.
2023-07-27 21:30:43,387 [salt.master      :2221][WARNING ][855866] Authentication failure of type "eauth" occurred.

[garethgreenaway]

@major0 The issue here is that the current implementation of the PAM authentication modules rely on the local system shadow and group authentication information and are unable to query information against other external systems. Perhaps one of the other authentication modules might work better in this instance. Where is your user & group information stored?

[major0]

@garethgreenaway What is particularly confusing is that this used to work just fine. What was changed that removed existing functionality? Why would Salt be using PAM modules that don't fully support PAM? Is this a side-effect of the move to onedir?

At the end of the day, this is a domain joined system using sssd, and I am not likely to give Salt a special password so it can query AD to do external LDAP authentication. The natural solution for me is to just let the applications on the system use the local authentication (PAM) and have everything work, which historically has been a pretty safe assumption; until Salt.

[Ch3LL]

What is particularly confusing is that this used to work just fine

What version did this previously work on? And what version of python did it previously work on?

[Sxderp]

This is related to

https://github.com/saltstack/salt/issues/64953

Which sems to have been fixed in

https://github.com/saltstack/salt/commit/28be150d1a4ccbccfe3e02ad2af13dea4c7a8284

[chutz]

I am seeing this bug as well, it worked in 3005, and broke in 3006. Salt is running as root, and pam eauth is refusing to work.

[chutz]

Replacing salt/auth/pam.py with the version from 3005.5 fixes this issue for me.