salt
salt copied to clipboard
saltstack
[BUG] Slots cmd.run does not behave as expected
Description
While attempting to use slots in a salt state, the cmd.run function is not behaving the same way as it does when run from the command line. (see the example command used, below)
Setup
Salt 3005.1 onedir installation on Ubuntu22. Install salt-master and salt-minion
Steps to Reproduce the behavior
Create a state to test slots cmd.run, e.g. test_slots.sls
manage_file:
file.managed:
- name: /tmp/slots
- contents: __slot__:salt:cmd.run("$(echo pip) --version")
Run the state
salt <minion-id> state.sls test_slots
Error shown below
root@salt:/srv/salt# salt \* state.sls test_slots
salt:
----------
ID: manage_file
Function: file.managed
Name: /tmp/slots
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/opt/saltstack/salt/run/salt/modules/cmdmod.py", line 729, in _run
proc = salt.utils.timed_subprocess.TimedProc(cmd, **new_kwargs)
File "salt/utils/timed_subprocess.py", line 53, in __init__
self.process = subprocess.Popen(args, **kwargs)
File "salt/utils/pyinstaller/rthooks/_overrides.py", line 65, in __init__
super().__init__(*args, **kwargs)
File "subprocess.py", line 951, in __init__
File "subprocess.py", line 1821, in _execute_child
FileNotFoundError: [Errno 2] No such file or directory: '$(echo'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "salt/state.py", line 2275, in call
self.format_slots(cdata)
File "salt/state.py", line 2507, in format_slots
cdata[atype][ind] = self.__eval_slot(arg)
File "salt/state.py", line 2418, in __eval_slot
slot_return = self.functions[fun](*args, **kwargs)
File "salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "/opt/saltstack/salt/run/salt/modules/cmdmod.py", line 1286, in run
ret = _run(
File "/opt/saltstack/salt/run/salt/modules/cmdmod.py", line 736, in _run
raise CommandExecutionError(msg)
salt.exceptions.CommandExecutionError: Unable to run command '['$(echo', 'pip)', '--version']' with the context '{'cwd': '/root', 'shell': False, 'env': {'PWD': '/', 'SYSTEMD_EXEC_PID': '848', 'LANG': 'C.UTF-8', 'SSL_CERT_DIR': '/usr/lib/ssl/certs', 'INVOCATION_ID': 'a059a6ca6a5e4c309f226f9384155dce', 'SHLVL': '0', 'SSL_CERT_FILE': '/usr/lib/ssl/cert.pem', 'JOURNAL_STREAM': '8:19873', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin', 'TCL_LIBRARY': '/opt/saltstack/salt/run/tcl', 'TK_LIBRARY': '/opt/saltstack/salt/run/tk', 'LC_CTYPE': 'C', 'LC_NUMERIC': 'C', 'LC_TIME': 'C', 'LC_COLLATE': 'C', 'LC_MONETARY': 'C', 'LC_MESSAGES': 'C', 'LC_PAPER': 'C', 'LC_NAME': 'C', 'LC_ADDRESS': 'C', 'LC_TELEPHONE': 'C', 'LC_MEASUREMENT': 'C', 'LC_IDENTIFICATION': 'C', 'LANGUAGE': 'C', 'LD_LIBRARY_PATH': ''}, 'stdin': None, 'stdout': -1, 'stderr': -2, 'with_communicate': True, 'timeout': None, 'bg': False, 'close_fds': True}', reason: [Errno 2] No such file or directory: '$(echo'
Started: 18:22:28.373924
Duration: 22.541 ms
Changes:
Summary for salt
------------
Succeeded: 0
Failed: 1
------------
Total states run: 1
Total run time: 22.541 ms
ERROR: Minions returned with non-zero exit code
Also, all permutations of different quoting that I tried on the contents line of the test state still threw errors, e.g.
manage_file:
file.managed:
- name: /tmp/slots
- contents: __slot__:salt:cmd.run($(echo pip) --version)
root@salt:/srv/salt# salt \* state.sls test_slots
salt:
----------
ID: manage_file
Function: file.managed
Name: /tmp/slots
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/opt/saltstack/salt/run/salt/modules/cmdmod.py", line 729, in _run
proc = salt.utils.timed_subprocess.TimedProc(cmd, **new_kwargs)
File "salt/utils/timed_subprocess.py", line 53, in __init__
self.process = subprocess.Popen(args, **kwargs)
File "salt/utils/pyinstaller/rthooks/_overrides.py", line 65, in __init__
super().__init__(*args, **kwargs)
File "subprocess.py", line 951, in __init__
File "subprocess.py", line 1821, in _execute_child
FileNotFoundError: [Errno 2] No such file or directory: '$(echopip)--version'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "salt/state.py", line 2275, in call
self.format_slots(cdata)
File "salt/state.py", line 2507, in format_slots
cdata[atype][ind] = self.__eval_slot(arg)
File "salt/state.py", line 2418, in __eval_slot
slot_return = self.functions[fun](*args, **kwargs)
File "salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "/opt/saltstack/salt/run/salt/modules/cmdmod.py", line 1286, in run
ret = _run(
File "/opt/saltstack/salt/run/salt/modules/cmdmod.py", line 736, in _run
raise CommandExecutionError(msg)
salt.exceptions.CommandExecutionError: Unable to run command '['$(echopip)--version']' with the context '{'cwd': '/root', 'shell': False, 'env': {'PWD': '/', 'SYSTEMD_EXEC_PID': '848', 'LANG': 'C.UTF-8', 'SSL_CERT_DIR': '/usr/lib/ssl/certs', 'INVOCATION_ID': 'a059a6ca6a5e4c309f226f9384155dce', 'SHLVL': '0', 'SSL_CERT_FILE': '/usr/lib/ssl/cert.pem', 'JOURNAL_STREAM': '8:19873', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin', 'TCL_LIBRARY': '/opt/saltstack/salt/run/tcl', 'TK_LIBRARY': '/opt/saltstack/salt/run/tk', 'LC_CTYPE': 'C', 'LC_NUMERIC': 'C', 'LC_TIME': 'C', 'LC_COLLATE': 'C', 'LC_MONETARY': 'C', 'LC_MESSAGES': 'C', 'LC_PAPER': 'C', 'LC_NAME': 'C', 'LC_ADDRESS': 'C', 'LC_TELEPHONE': 'C', 'LC_MEASUREMENT': 'C', 'LC_IDENTIFICATION': 'C', 'LANGUAGE': 'C', 'LD_LIBRARY_PATH': ''}, 'stdin': None, 'stdout': -1, 'stderr': -2, 'with_communicate': True, 'timeout': None, 'bg': False, 'close_fds': True}', reason: [Errno 2] No such file or directory: '$(echopip)--version'
Started: 18:29:25.332102
Duration: 22.499 ms
Changes:
Summary for salt
------------
Succeeded: 0
Failed: 1
------------
Total states run: 1
Total run time: 22.499 ms
ERROR: Minions returned with non-zero exit code
And finally
manage_file:
file.managed:
- name: /tmp/slots
- contents: "__slot__:salt:cmd.run($(echo pip) --version)"
root@salt:/srv/salt# salt \* state.sls test_slots
salt:
----------
ID: manage_file
Function: file.managed
Name: /tmp/slots
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/opt/saltstack/salt/run/salt/modules/cmdmod.py", line 729, in _run
proc = salt.utils.timed_subprocess.TimedProc(cmd, **new_kwargs)
File "salt/utils/timed_subprocess.py", line 53, in __init__
self.process = subprocess.Popen(args, **kwargs)
File "salt/utils/pyinstaller/rthooks/_overrides.py", line 65, in __init__
super().__init__(*args, **kwargs)
File "subprocess.py", line 951, in __init__
File "subprocess.py", line 1821, in _execute_child
FileNotFoundError: [Errno 2] No such file or directory: '$(echopip)--version'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "salt/state.py", line 2275, in call
self.format_slots(cdata)
File "salt/state.py", line 2507, in format_slots
cdata[atype][ind] = self.__eval_slot(arg)
File "salt/state.py", line 2418, in __eval_slot
slot_return = self.functions[fun](*args, **kwargs)
File "salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "/opt/saltstack/salt/run/salt/modules/cmdmod.py", line 1286, in run
ret = _run(
File "/opt/saltstack/salt/run/salt/modules/cmdmod.py", line 736, in _run
raise CommandExecutionError(msg)
salt.exceptions.CommandExecutionError: Unable to run command '['$(echopip)--version']' with the context '{'cwd': '/root', 'shell': False, 'env': {'PWD': '/', 'SYSTEMD_EXEC_PID': '848', 'LANG': 'C.UTF-8', 'SSL_CERT_DIR': '/usr/lib/ssl/certs', 'INVOCATION_ID': 'a059a6ca6a5e4c309f226f9384155dce', 'SHLVL': '0', 'SSL_CERT_FILE': '/usr/lib/ssl/cert.pem', 'JOURNAL_STREAM': '8:19873', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin', 'TCL_LIBRARY': '/opt/saltstack/salt/run/tcl', 'TK_LIBRARY': '/opt/saltstack/salt/run/tk', 'LC_CTYPE': 'C', 'LC_NUMERIC': 'C', 'LC_TIME': 'C', 'LC_COLLATE': 'C', 'LC_MONETARY': 'C', 'LC_MESSAGES': 'C', 'LC_PAPER': 'C', 'LC_NAME': 'C', 'LC_ADDRESS': 'C', 'LC_TELEPHONE': 'C', 'LC_MEASUREMENT': 'C', 'LC_IDENTIFICATION': 'C', 'LANGUAGE': 'C', 'LD_LIBRARY_PATH': ''}, 'stdin': None, 'stdout': -1, 'stderr': -2, 'with_communicate': True, 'timeout': None, 'bg': False, 'close_fds': True}', reason: [Errno 2] No such file or directory: '$(echopip)--version'
Started: 18:34:53.287669
Duration: 18.219 ms
Changes:
Summary for salt
------------
Succeeded: 0
Failed: 1
------------
Total states run: 1
Total run time: 18.219 ms
ERROR: Minions returned with non-zero exit code
Expected behavior The command executes as expected when run from the cli; it should behave the same when run using slots.
root@salt:/srv/salt# salt \* cmd.run "$(echo pip) --version"
salt:
pip 22.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Versions Report
root@salt:/srv/salt# salt-call --versions-report
Salt Version:
Salt: 3005.1
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.0
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
Python: 3.9.15 (main, Nov 8 2022, 03:42:58)
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 23.2.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 22.10 kinetic
locale: utf-8
machine: x86_64
release: 5.19.0-23-generic
system: Linux
version: Ubuntu 22.10 kinetic
Slots have their own argument parsing to make it "easier" to use.
Did you try maybe __slot__:salt:cmd.run($(echo pip) --version)?
However, slots should not be necessary for your case.
file.managed supports templated contents, and those templates are not built until the state runs.
{% raw %}
manage_file:
file.managed:
- name: /tmp/slots
- contents: {{ salt["cmd.run"]("$(echo pip) --version") }}
- template: jinja
{% endraw %}
If you make the sls just #!yaml, or have the template in a separate source file, then you don't need the {% raw %} block.
@OrangeDog The additional slots argument quoting did not solve this, but your suggestion regarding the runtime rendering of contents for file.managed will probably solve the use case I had in mind. I did not realize that functionality existed. Thanks.
I still think the slots command behaving different than at the cli is less than ideal, but I will defer the salt dev team on wether it is deemed a "bug". As a side note, the additional quoted version you suggested, (1) did not work in bash cli (outside of salt), (2) did work when run via cmd.run at the cli, and (3) did not work when run via the sls shown above; which is peculiar behavior in its own rite.
Oh of course, the single quotes will disable the expansion. More complex quoting would probably be needed.
Oh hey I think I encountered this same problem. I wanted to share my use case in case it helps when thinking about how to address the problem. I'm creating a user and group with auto-assigned UID and GID and trying to pass the UID and GID to docker_container.running. Something like this:
my-container-name:
docker_container.running:
...
- user: __slot__:salt:cmd.run("$(echo '$(id --user mastodon):$(id --group mastodon)')")
...
But this fails with [Errno 2] No such file or directory: '$(echo'
I can get a single ID pretty easily with either of these lines:
- user: __slot__:salt:user.info(mastodon).uid
- user: __slot__:salt:cmd.run('id --user mastodon')
But building the combined string of UID:GID is where I'm getting stuck. A few notes:
- The username and groupname that I'm using do not exist in /etc/passwd and /etc/group within the Docker image so Docker does not allow me to pass in the names--I am only allowed to pass in the IDs.
- The user and group might not exist the first time the states are applied so I think I can't look them up with Jinja. I think that means I need to use Slots.
- For now my workaround is to specify the UID and GID when creating the user and when running the container. It's fine. Not a big deal.
@markdoliner did you try your command in bash first? You've added an extra $() around it for some reason, and there's no substitution inside single quotes.
$(id: command not found
Try this instead:
- user: __slot__:salt:cmd.run(echo "$(id --user mastodon):$(id --group mastodon)")
there's no substitution inside single quotes.
Good point. I think I probably changed the single quotes to double quotes at some point when trying to get it to work in Salt.
Try this instead:
- user: __slot__:salt:cmd.run(echo "$(id --user mastodon):$(id --group mastodon)")
That gives me this error:
salt.exceptions.CommandExecutionError:
Unable to run command '['echo$(id', '--user', 'mastodon):$(id', '--group', 'mastodon)']'
with the context {large dict of env vars and other args},
reason: [Errno 2] No such file or directory: 'echo$(id'
It looks like a space and the double quotes are being removed. I think the relevant Slot parsing might be here. I see similar behavior if I call parse_function() directly:
>>> import salt.utils.args
>>> salt.utils.args.parse_function('cmd.run(echo "$(id --user mastodon):$(id --group mastodon")')
('cmd.run', ['echo$(id --user mastodon):$(id --group mastodon'], {})
It seems like this should work:
- user: __slot__:salt:cmd.run('echo "$(id --user mastodon):$(id --group mastodon)"')
Here's what parse_function() returns:
('cmd.run', ['echo "$(id --user mastodon):$(id --group mastodon)"'], {})
And it does what I expect when I call cmd.run:
$ sudo salt-call cmd.run 'echo "$(id --user mastodon):$(id --group mastodon)"'
local:
998:998
But for whatever reason the commands within $() aren't evaluated when applying highstate. It looks like Docker is trying to use the literal id commands as the UID and GID:
----------
ID: redis
Function: docker_container.running
Result: False
Comment: Replaced container 'redis'. Failed to start container 'redis': 'Error 500: unable to find user $(id --user mastodon): no matching entries in passwd file'.
Started: 17:43:25.833569
Duration: 310.332 ms
Changes:
----------
container:
----------
Config:
----------
User:
----------
new:
$(id --user mastodon):$(id --group mastodon)
old:
998:998
container_id:
----------
added:
bad34e3e338fd9ad6f04f3f0a318b3abf7aa079f361e098ed6269af28fea2e9b
removed:
- 9a43f216b6a95301b4c9d8205ef9867339adf7a3ec3e8126e708d6c38732aa6f
----------
I don't know. Slots is just being different with command parsing, hence this issue. There might be a trick to get another level of shell so the string gets re-parsed by bash.
Instead of using cmd.run use cmd.shell which does the exact same thing as cmd.run except that it sets python_shell=True -- which prevents the command from being split and parsed weird.
I don't run into the issue when I run the state like this:
run_command:
file.managed:
- name: /tmp/slots
- contents: __slot__:salt:cmd.shell("$(echo pip) --version")