Salt's cmd.run ignoring shell script return code

[RSpeaker-FD]

I am using a simple salt state to send (file.managed) and execute (cmd.run) a shell script on a minion/target. No matter what exit or return value the shell script sends, the salt master is interpreting the result as successful.

I tried using cmd.script, but keep getting a permission denied error on the temp version of the file under /tmp. Filesystem is not mounted with noexec so we can't figure out why it won't work. I saw earlier references to a bug for this but was supposedly fixed in 2012.

For cmd.run, stdout in the job output shows the failed return code and message but Salt still says Success. Running the script locally on the minion reports the return/exit code as expected.

I tried adding stateful: True into the cmd.run block and formatted the key value pairs at the end of the shell script as demonstrated in the docs.

Running against 2 minion target, 1 fail 1 succeed. Both report Result as True but correctly populate Comment with my key value pair.

I've tried YES/NO, TRUE/FALSE, 0/1 - nothing works.

The end of my shell script, formatted as shown in the docs.

echo Return_Code=${STATUS}

if [[ ${STATUS} -ne 0 ]]
   then
      echo ""
      echo "changed=False comment='Failed'"
   else
      echo ""
      echo "changed=True comment='Success'"
fi

The SLS block:

stop_oracle:
  cmd.run:
    - name: {{scriptDir}}/{{scriptName}}{{scriptArg}}
    - stateful: True
    - failhard: True

SLS output from Successful minion:

----------
          ID: stop_oracle
    Function: cmd.run
        Name: /u01/orastage/oraclepsu/scripts/oracle_ss_wrapper.ksh stop
      Result: True
     Comment: Success
     Started: 14:37:44.519131
    Duration: 18930.344 ms
     Changes:
              ----------
              changed:
                  True
              pid:
                  26195
              retcode:
                  0
              stderr:
                  stty: standard input: Inappropriate ioctl for device
              stdout:
                  Script running under ROOT

                  Mon Jul  1 14:38:03 EDT 2019 : Successful
                  Return_Code=0

SLS output from Failed minion:

----------
          ID: stop_oracle
    Function: cmd.run
        Name: /u01/orastage/oraclepsu/scripts/oracle_ss_wrapper.ksh stop
      Result: True
     Comment: Failed
     Started: 14:07:14.153940
    Duration: 38116.134 ms
     Changes:

Output from shell script run locally on fail target:

$ /u01/orastage/oraclepsu/scripts/oracle_ss_wrapper.ksh stop

Mon Jul  1 15:29:18 EDT 2019 : There are errors in the process
Return_Code=1

changed=False comment='Failed'

Output from shell script run locally on success target:

$ /u01/orastage/oraclepsu/scripts/oracle_ss_wrapper.ksh stop

Mon Jul  1 16:03:18 EDT 2019 : Successful
Return_Code=0

changed=True comment='Success'

Version Information:

Master: Oracle linux, 2018.3.2 (Oxygen) 
Minion: Redhat Linux, 2017.7.4 (Nitrogen) 
Minion: AIX, 2017.7.2 (Nitrogen) 

[weswhet]

You probably want to put this issue under the core ‘salt’ repo, not community.

[RSpeaker-FD]

I reposted it under 'saltstack/salt', will close this one. Thanks for the tip.