!cmd hook runs in `sh`, which is not my default shell

[cleonard-godaddy]

Subject of the issue

When I use a !cmd hook, it appears to be run by sh because I receive this error:

/bin/sh: 1: [[: not found

Is there a way to force the underlying subprocess.run() call to use Python? I know I can do this in Python by using arguments that look like ["bash", "-c", "your command goes here"], and, in fact, if I simply pass in ["your command goes here"] in Python, it will run in bash because it is my default shell. Is sceptre really running my hook using sh? If so, how can I make it use bash instead?

Your environment

• version of sceptre (sceptre --version) Sceptre, version 2.4.0
• version of python (python --version) Python 3.8.10
• which OS/distro Ubuntu 20.04.3 LTS on 4.4.0-19041-Microsoft #1237-Microsoft x86_64

Steps to reproduce

Tell us how to reproduce this issue. Please provide sceptre project files if possible, you can use https://plnkr.co/edit/ANFHm61Ilt4mQVgF as a base.

Here is a hook that causes the behavior:

hooks:
    before_update:
        - !cmd 'if [[ "{{ var.update_lambdas | default("false") }}" == "true" ]]; then lambdas/deploy-lambdas.sh -u "$(./get-stack-output.sh "{{ environment }}/{{ region }}/LambdaS3Bucket.yaml" LambdaBucketName)" HandleServiceAccountAppGitHubWebhook; fi;'

Expected behaviour

The hook should run successfully using my default shell, which is bash.

Actual behaviour

I see the error /bin/sh: 1: [[: not found instead.

[jfalkenstein]

Hi @cleonard-godaddy, the cmd hook simply executes subprocess.check_call(self.argument, shell=True): https://github.com/Sceptre/sceptre/blob/master/sceptre/hooks/cmd.py#L22

According to https://stackoverflow.com/a/15449462/6136118, the default shell python uses is /bin/sh. So it looks like, in its current configuration, it won't execute code in bash. It might be possible for you to invoke bash with sh and then execute your script that way.

With that said, I can imagine a way for that hook to evaluate self.argument and decide:

• If self.argument is a str, it would execute the command just like it does now
• If self.argument is a dict, it would let you specify subprocess kwargs, like:

  hooks
      before_update:
          - !cmd
              args: "echo I'm using bash!"
              executable: "/bin/bash"
  

I would accept a pull request for this, as I think that could be pretty useful, such as also being able to specify things like environment variables, etc...

[jfalkenstein]

@cleonard-godaddy I suggested a change that could make this work. What did you think about putting up a PR to add that functionality? I'd be happy to help you with that in getting it through review.

[cleonard-godaddy]

@jfalkenstein, we came up with a different solution for this at work, which was to write the !cmd command in dash-compatible syntax, which also works on bash, so that they would run on basically anything we use that sh points to.

Another idea we may use in the future is to use !cmd to invoke shell scripts, and to have those shell scripts choose their own interpreters with their hash-bang directives.

However, I think there may still be value in implementing your suggestion because it could make one-liners and brief scripts very accessible from within sceptre when something is very easy to do in a particular language, without having to force it into sh syntax. If you agree, I would like to help with this. Thoughts?

Also, thanks for the nudge. :)

[jfalkenstein]

Hey, that does sound good. I'd be happy to work with you on that. If you're not already in the Sceptre slack channel, I'd suggest you join the #sceptre channel after joining the og-aws Slack server via http://slackhatesthe.cloud/. A bunch of us hang around on there.

[cleonard-godaddy]

I've not forgotten/don't want a bot to mark this as stale.

[iainelder]

Another idea we may use in the future is to use !cmd to invoke shell scripts, and to have those shell scripts choose their own interpreters with their hash-bang directives.

That's how I've done it. I'll share the solution for my use case in another issue because I have some questions about whether it's the right way to use Sceptre.

I wrote my first pre-launch hook today because I needed to fetch some assets that the template refers to.

hooks:
  before_launch:
    - !cmd "hooks/get-assets.sh"

The script get-assets.sh is executable. The first line contains a shebang to invoke bash: #!/bin/env bash.

My script is complex enough that it makes sense to store it in a separate file anyway. But for a one-liner maybe it would be extra hassle.

I can imagine a way for that hook to evaluate self.argument and decide

@jfalkenstein , what you described reminds me of the GiHub workflow shell settings to control how the run command is interpreted. Would something like that be enough to support one-liners?

You mentioned support for environment variables as well and the GitHub workflow syntax also has a precedent for that. I think just the shell setting would be enough to support this use case.

[jfalkenstein]

I don't know what "shell" setting you're talking about here. In reference to environment variables, I just mean something like this:

hooks:
    before_update:
        - !cmd
            args: "echo I'm using bash!"
            executable: "/bin/bash"
            env:
                MY_ENVIRONMENT_VARIABLE: "some value"

[iainelder]

Sorry for the confusion. I was referring to the similar concept from GitHub workflows.

I think it's relevant because it sets a precedent for the feature you describe.

In a GitHub workflow job step, a command may be as simple as this:

job1:
  - name: Get caller identity
    run: aws sts get-caller-identity

In GitHub the default shell is Bash. You can override it like this:

job1:
  - name: Get caller identity
    run: aws sts get-caller-identity
    shell: sh

And your suggested syntax for environment variables even matches that of GitHub actions:

job1:
  - name: Get caller identity
    run: aws sts get-caller-identity
    shell: sh
    env:
      AWS_PROFILE: "xxx"
      AWS_DEFAULT_REGION: "yyy"

I'm not saying that copying GitHub actions syntax should be a design goal, but that there is a similar use case that is popular that sets a precedent.

[jfalkenstein]

Oh, I see. Yeah, sounds good. I'd be happy to work with you on implementing that feature. It actually wouldn't be really hard at all.

[iainelder]

Sure, @jfalkenstein , I'd like to help and learn about the Sceptre development process. We can discuss it on the Slack channel.