saltstack
[BUG] salt-ssh gpg pillar rendering failure
Description I use gpg encrypted pillar files. This has worked for many versions of salt, going back years. Now the same setup with 3003 throws errors on state.highstate. If I downgrade to 3002.6, and run the same command it works. Even 3003 with pillar.items works.
The file is encrypted with a key in /home/user/.gnupg.
Setup ./Saltfile:
salt-ssh:
config_dir: .
max_procs: 1
ssh_wipe: True
./master:
gpg_keydir: /home/user/.gnupg
file_roots:
base:
- ..
- ../../salt-grains
pillar_roots:
base:
- ../../salt-pillars
- ../../salt-grains
salt-pillars/top.sls:
base:
'*':
- kube.foo.ssl.ca.key
raw: salt-pillars/kube/foo/ssl/ca/key/init.sls:
kube:
foo:
ssl:
ca:
key:
data: |
-----BEGIN EC PRIVATE KEY-----
...
...
...
-----END EC PRIVATE KEY-----
encrypted: salt-pillars/kube/foo/ssl/ca/key/init.sls:
#!gpg|yaml
-----BEGIN PGP MESSAGE-----
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
-----END PGP MESSAGE-----
Errors
[WARNING ] Could not decrypt cipher b'-----BEGIN PGP MESSAGE-----\n\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n............................\n.....
\n-----END PGP MESSAGE-----', received: b"gpg: keyblock resource '/var/tmp/.user_ab4926_salt/gpgkeys/pubring.kbx': No such file or directory\n[GNUPG:] ERROR add_keyblock_resource 33587281\n[GNUPG:] ENC_TO A0251D47D061704C 1 0\n[GNUPG:] ERROR keydb_search 33554445\n[GNUPG:] ERROR keydb_search 33554445\ngpg: encrypted with RSA key, ID A0251D47D061704C\n[GNUPG:] NO_SECKEY A0251D47D061704C\n[GNUPG:] BEGIN_DECRYPTION\n[GNUPG:] DECRYPTION_FAILED\ngpg: decryption failed: No secret key\n[GNUPG:] END_DECRYPTION\n"
[ERROR ] SLS 'kube.foo.ssl.ca.key' does not render to a dictionary
[ERROR ] The rendered pillar sls file, 'kube.foo.ssl.ca.key' state did not return the expected data format. This is a sign of a malformed pillar sls file. Returned errors: 'SLS 'kube.foo.ssl.ca.key' does not render to a dictionary'
[CRITICAL] Pillar render error: SLS 'kube.foo.ssl.ca.key' does not render to a dictionary
Steps to Reproduce the behavior 3003 fails: salt-ssh -i fqdn state.highstate
3003 works: salt-ssh -i fqdn pillar.items
3002.6 works: salt-ssh -i fqdn state.highstate
Expected behavior The encrypt pillars to decrypt properly running an salt-ssh state.highstate
GnuPG version
gnupg2-2.2.25-2.fc33.x86_64
Versions Report
salt-ssh --versions-report
Salt Version:
Salt: 3003
Dependency Versions:
cffi: 1.14.1
cherrypy: Not Installed
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.3
libgit2: Not Installed
M2Crypto: Not Installed
Mako: 1.1.3
msgpack: 1.0.0
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 2.6.1
pycryptodome: 3.10.1
pygit2: Not Installed
Python: 3.9.2 (default, Feb 20 2021, 00:00:00)
python-gnupg: Not Installed
PyYAML: 5.4.1
PyZMQ: 22.0.3
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: fedora 33
locale: utf-8
machine: x86_64
release: 5.11.10-200.fc33.x86_64
system: Linux
version: Fedora 33
I'm getting similar errors with salt-ssh 3003.1:
gpg: keyblock resource '/var/tmp/.***_salt/gpgkeys/pubring.kbx': No such file or directory
salt-ssh 3002.6 works fine.
Same issue still happening with 3003.2 which was just released. Just intrigued, since this is a P2, which release can we expect this to be fixed on?
Salt 3004 was released but no mention of this issue being fixed yet. Any updates?
It looks like we have the same issue on our platform. We are using Ubuntu 20.04 with these packages installed on our master:
- salt-master 3004.2+ds-1
- gpg 2.2.19-3ubuntu2.2
- libgpgme11:amd64 1.13.1-7ubuntu2
- libgpg-error0:amd64 1.37-1
On minion side, we firstly add a version issue (3003 installed). After the upgrade, it did not solve the issue. Here the packages installed:
- salt-minion 3004.2+ds-1
- gpg 2.2.19-3ubuntu2.2
- libgpgme11:amd64 1.13.1-7ubuntu2
- libgpg-error0:amd64 1.37-1
The issues appeared after we update our server to the latest packages versions. Here the generated message on our master when running state.apply. Same behaviors can be generated on all minions connected to the master.
[WARNING ] Could not decrypt cipher '-----BEGIN PGP MESSAGE-----\n...\n-----END PGP MESSAGE-----', received: "gpg: keyblock resource '/var/tmp/.ubuntu_3998c6_salt/gpgkeys/pubring.kbx': No such file or directory\n[GNUPG:] ERROR add_keyblock_resource 33587281\n[GNUPG:
] ENC_TO XXXXXXXXXXXX 1 0\n[GNUPG:] ERROR keydb_search 33554445\n[GNUPG:] ERROR keydb_search 33554445\ngpg: encrypted with RSA key, ID XXXXXXXXXXXX\n[GNUPG:] NO_SECKEY XXXXXXXXXXXX\n[GNUPG:] BEGIN_DECRYPTION\n[GNUPG:] DECRYPTION_FAILED\ngpg: decryption failed:
No secret key\n[GNUPG:] END_DECRYPTION\n"
When creating an encrypted message with gpg and decipher it manually, it works without any issue. When reusing thefailing encrypted payload from the warning message, it works as well.
$ echo test | gpg -a -e -r salt-master
$ cat test.gpg | sudo gpg --homedir /etc/salt/gpgkeys/ -d -a
gpg: encrypted with 3072-bit RSA key, ID XXXXXXXXXXXX, created 2021-09-14
"salt-master"
test
A gpg-agent is correctly running on our servers using this command: gpg-agent --homedir /etc/salt/gpgkeys --use-standard-socket --daemon.
Running into the same issue with salt-ssh on 3006.1, 3004.2 and 3005 (whatever the latest is right now).
I'm seeing the same issue on 3006.2. Is there a workaround to this other than not using gpg encrypted pillar?
I found the cause for this, see https://github.com/saltstack/salt/issues/36796#issuecomment-1785775927. Will submit a test + fix for this shortly (if this does not turn into a yak shaving situation at least).
