saltstack
runas : WindowsError: The system could not find the environment option that was entered.
Description of Issue
When using 'cmd.run' with 'runas', The call fails with "WindowsError: The system could not find the environment option that was entered."
This looks like https://github.com/saltstack/salt/issues/54060 but that is Win32 and did not wan't to hijack the issue if it was really different.
Setup
No special setup needed
Steps to Reproduce Issue
root@Hermiston-salt:/srv/salt/fix# salt HEDT8 cmd.run 'echo "Hello"' runas=csuser password=xx
HEDT8:
The minion function caused an exception: Traceback (most recent call last):
File "c:\salt\bin\lib\site-packages\salt\minion.py", line 1660, in _thread_return
return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
File "c:\salt\bin\lib\site-packages\salt\executors\direct_call.py", line 12, in execute
return func(*args, **kwargs)
File "c:\salt\bin\lib\site-packages\salt\modules\cmdmod.py", line 1167, in run
**kwargs)
File "c:\salt\bin\lib\site-packages\salt\modules\cmdmod.py", line 407, in _run
return win_runas(cmd, runas, password, cwd)
File "c:\salt\bin\lib\site-packages\salt\utils\win_runas.py", line 185, in runas
environment=env,
File "c:\salt\bin\lib\site-packages\salt\platform\win.py", line 1055, in CreateProcessWithTokenW
raise exc
WindowsError: The system could not find the environment option that was entered.
ERROR: Minions returned with non-zero exit code
The debug logs did not provide any more data than this traceback
Versions Report
csuser@HEDT8 C:\>salt-call --versions
Salt Version:
Salt: 2019.2.0
Dependency Versions:
cffi: 1.11.5
cherrypy: 17.4.1
dateutil: 2.7.5
docker-py: Not Installed
gitdb: 2.0.5
gitpython: 2.1.10
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: 1.6.1
M2Crypto: Not Installed
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.19
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)]
python-gnupg: 0.4.3
PyYAML: 3.13
PyZMQ: 17.1.2
RAET: Not Installed
smmap: 2.0.5
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.2.5
System Versions:
dist:
locale: cp1252
machine: AMD64
release: 7
system: Windows
version: 7 6.1.7601 SP1 Multiprocessor Free
I also updated a minion to 2019.2.1 and it showed the same error (When run from the same 2019.2.0 master).
Another interesting note is that when I add a log.warning just before advapi32.CreateProcessWithTokenW it changes the error from WindowsError: The system could not find the environment option that was entered. to WindowsError: The operation completed successfully.. (With identical backtrack)
platform\win.py
log.warning("ENV FOUND (%s)", environment)
environment = ctypes.pointer(
environment_string(environment)
)
process_info = PROCESS_INFORMATION()
ret = advapi32.CreateProcessWithTokenW(
token,
I am not sure if that is useful, common or another bug but it is interesting.
Changing the log to just print "Hello World" gave the same output so it is the logging process and not some weird byproduct of the use of the 'environment' variable.
@breshead Thanks for reporting this, I can confirm the behavior although the only difference is the type of exception that gets raised (even though the error message is the same).
DESKTOP-SQIU2I1:
The minion function caused an exception: Traceback (most recent call last):
File "c:\dev\salt\salt\minion.py", line 1664, in _thread_return
return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
File "c:\dev\salt\salt\executors\direct_call.py", line 12, in execute
return func(*args, **kwargs)
File "c:\dev\salt\salt\modules\cmdmod.py", line 1171, in run
**kwargs)
File "c:\dev\salt\salt\modules\cmdmod.py", line 407, in _run
return win_runas(cmd, runas, password, cwd)
File "c:\dev\salt\salt\utils\win_runas.py", line 185, in runas
environment=env,
File "c:\dev\salt\salt\platform\win.py", line 1058, in CreateProcessWithTokenW
raise exc
OSError: The system could not find the environment option that was entered.
ERROR: Minions returned with non-zero exit code
Using both a master and minion on windows will reproduce this error. However, using a master on linux with a windows minion doesn't when you execute the same command. The output on the minion looks interesting too.
Output...
root@DESKTOP-SQIU2I1:~# salt DESKTOP-SQIU2I1 cmd.run 'echo "hi"' runas=jeaco password=xxx
DESKTOP-SQIU2I1:
hi
Minion log...
[INFO ] Executing command 'echo "hi"' as user 'jeaco' in directory 'C:\Users\jeaco'
[DEBUG ] Unable to OpenProcess pid=4 name=System
[DEBUG ] Unable to OpenProcess pid=68 name=python2.7
[DEBUG ] Unable to OpenProcess pid=104 name=Registry
[DEBUG ] Token for pid does not match user sid: S-1-5-18
[DEBUG ] Token for pid does not match user sid: S-1-5-18
[DEBUG ] Unable to OpenProcess pid=352 name=smss.exe
[DEBUG ] OpenProcessToken failed pid=400 name=svchost.exe userNT AUTHORITY\SYSTEM
[DEBUG ] Unable to OpenProcess pid=412 name=SgrmBroker.exe
[DEBUG ] Unable to OpenProcess pid=488 name=sudo
[DEBUG ] Unable to OpenProcess pid=608 name=csrss.exe
[DEBUG ] output: hi
Same issue. Anyone have a workaround that sidesteps the runas option in the mean time?
only on some commands ... for instance:
salt DESKTOP-REL3EPD cmd.run "whoami" runas=kiosk
DESKTOP-REL3EPD: desktop-rel3epd\kiosk
but running:
salt DESKTOP-REL3EPD cmd.run "echo 'paul'" runas=kiosk
throw the "WindowsError: The system could not find the environment option that was entered." error
My issue is on windows desktops (yuk) so I use psexec to work around it, although the runas felt much cleaner.
cmd.run:
name: psexec /accepteula -h -u .\{{ admname }} -p {{ admpwd }} cmd.exe /c <command here>
Lol this would be SICK to have fixed though. I mean it really nerfs the functionality in a lot of places. For instance, even with the psexec hack, I can't seem to locate a windows DDE service running under the logged in user. If I start my DDE client while logged in I find it just fine. When I start my DDE client even through psexec as the logged in user, it somehow can't find the DDE process. So, this is by no means a real fix.
Can any maintainers point me in a possible direction? Happy to troubleshoot but let me know if y'all have a good place to start :)
YES, it is a real pain not having this built in and psexec does seem to fall down in areas. Help in this area would be awesome.
runas uses a low level windows api to launch programs. The problem here is that the 'echo' is not a program located in the path.. It's actually a built in command of cmd.exe. When trying to use built-ins you need to launch cmd.exe directly.
Here is a list of cmd.exe builtin commands.
https://renenyffenegger.ch/notes/Windows/dirs/Windows/System32/cmd_exe/commands/index
instead of
salt minion cmd.run "echo hello" runas=Administrator
use
salt minion cmd.run "cmd.exe /c echo hello" runas=Administrator
We probably need to catch this error and return a more helpful message.
Hi @dwoz , I tried the command you mentioned in the comment above to start a windows service. But, I get the same Windows Error that system could not find the environment option that was entered. Command used: sudo salt 'target' cmd.run "net start Wecsvc" runas=administrator password=abc123
net start/net stop command is not part of the cmd.exe built-in commands you provided in: https://renenyffenegger.ch/notes/Windows/dirs/Windows/System32/cmd_exe/commands/index
I tried with powershell as well: sudo salt 'target' cmd.run 'start powershell "Stop-Service -name Wecsvc"' worked fine from command line. But in state.sls file, it throws same error. Your help is much appreciated. I am stuck here.
Thanks, Sriram
Do you guys actually care about backwards compatibility? This worked fine in 2016.11 (and possibly some later versions) but doesn't work anymore in 3000.8.
What do you mean by backwards compatibility? If you are asking if I care about using this feature (without work-around) moving forward - YES! If you are asking if I care about older version of salt - NO.
Regarding the statement from @sriramperumalla : Is this (in windows) limited to only 'built-in functions' provided by cmd.exe or should it handle any command line program call?
By "guys" I meant Salt developers :-) What I mean is it ran fine in older minions, definitely in 2016.11 which - due to large organisation inertia and the reasons described below - is still used in my environment. Runas works fine in 2016.11 so "runas uses a low level windows api to launch programs. The problem here is that the 'echo' is not a program located in the path.. It's actually a built in command of cmd.exe. When trying to use built-ins you need to launch cmd.exe directly." is not exactly a credible excuse. Newer minions have multiple incompatibilities, for example speaking of cmd.run - return events have now success=False if exit code is != 0, unlike it was in 2016.11 (where success=False would be set if there was really some serious failure in the minion preventing the job from running, now most likely undistinguishable from a command returning exit code 1 if you look at _thread_return() func in minion.py). This all makes it real fun to migrate complex environments, although I cannot grumble much since thanks to that I have an interesting job with lots of Python coding xD Just one more thing from the top of my head - guid is not set properly for Linux run_as in 3000.8 too - a patch from Jul 2019 (https://github.com/saltstack/salt/pull/53681) never made it to Salt 3000 which seems otherwise maintained.
Mystery solved. So back in the old days there was cmd = 'cmd /c {0}'.format(cmd) in win_runas.py:
https://github.com/saltstack/salt/blob/v2016.11.10/salt/utils/win_runas.py#L416
until someone came along and decided that python_shell (set by default to true in most cases) will do that for cmd.run w/o runas (which goes via subprocess.Popen() where python_shell is translated to shell argument) and it will just start failing for the executions with runas.
Mystery solved. So back in the old days there was cmd = 'cmd /c {0}'.format(cmd) in win_runas.py:
https://github.com/saltstack/salt/blob/v2016.11.10/salt/utils/win_runas.py#L416
until someone came along and decided that python_shell (set by default to true in most cases) will do that for cmd.run w/o runas (which goes via subprocess.Popen() where python_shell is translated to shell argument) and it will just start failing for the executions with runas.
omg I'm gonna have to unpack this mentally but it seems like you just made my day
@szjur It would help me a lot if you could state your discovery as a proposal to updating core code. I would be happy to make the pull requests if you could help me wrap my head around what needs to be done here
@dwoz at some point changed the windows runas code a lot, which is good (cool geeky stuff, hats off) - it may even run w/o providing a password but he also stripped the code that added cmd /c before the command: https://github.com/saltstack/salt/commit/ff781617e74232080ab85b9741cae5f4505d767a
For me personally it's a bit confusing that commands which run fine with cmd.run start failing with a cryptic message once you add runas parameter. As I mentioned above, without runas the magic is done by subprocess.Popen() which effectively runs the commands via a shell. You would need to take a look into modules/cmdmod.py to understand the difference better and also how python_shell parameter behaves, even though it seemingly defaults to None but turns into True at some point (don't get fooled by that and see the difference between cmd.run dir and cmd.run dir python_shell=False if still in doubt). Ideally runas could consider that python_shell too (that was not the case up to Salt 2019.3 - it would just always prepend the command with cmd /c). But it's Salt devs call in the end - I'm fine with patching my own build to behave like versions up to 2019.3.* did. It's always lots of fun later to port all these custom patches to a new major release :)