Check directories recursively

[untitaker]

Thanks for this useful program! It would be nice, however, if i could just point this at my dotfiles directory and let it automatically check all shell files (recognized by shebang?)

[koalaman]

Is there any advantage to shellcheck supporting this over just invoking it recursively with find?

[untitaker]

It'd be definetly shorter to type. From a technical POV i don't see any advantages either.

[koalaman]

I'll leave this up to find.

[dialex]

For beginners, can you provide a code snippet of using find to achieve it? This worked for me:

find path/to/scripts -type f -exec "shellcheck" "--format=gcc" {} \;

[coudenysj]

The problem with this approach is that you cannot rely on the exit code of the find script (in a build/test script for instance).

You can use this to accomplish that:

for file in $(find path/to/scripts -type f); do shellcheck --format=gcc $file; done;

[untitaker]

The logic for detecting what is supposed to be a shellscript and what not is sometimes quite contrived. There is not a single file extension to detect files by, executable bit is not always set... I actually now agree with @koalaman, but probably for different reasons.

[coudenysj]

Oh, yes off course.

I use a recursive grep to find #!/bin/sh line in files, and pipe those filenames to shellcheck.

I just wanted to note the fact that if you want to use the exit codes, you need to run the commands in a loop.

[koalaman]

This topic needs a wiki page.

[fkromer]

@dialex Your snippet doesn't distinguish between *.sh and other files (and could lead to false positives), right?

[fkromer]

@coudenysj If I run your snippet via gitlab-runner exec docker shellcheck

and a .gitlab-ci.yml

shellcheck:
  image: koalaman/shellcheck-alpine
  script:
    - shellcheck --version
    - for file in $(. -type f); do shellcheck --format=gcc $file; done;

I get this error

$ for file in $(. -type f); do shellcheck --format=gcc $file; done;
/bin/sh: .: line 21: illegal option -t

Any idea how to fix that?

---

EDIT: .gitlab-ci.yml

shellcheck:
  image: koalaman/shellcheck-alpine
  script:
    - shellcheck --version
    - for file in $(find . -type f); do shellcheck --format=gcc $file; done;

[coudenysj]

@fkromer You need to use the find command.

$ for file in $(find . -type f); do shellcheck --format=gcc $file; done;

[coudenysj]

@koalaman I've created a Recursiveness wiki page.

[fkromer]

@coudenysj Ooh, thanks for catching that.

[pawamoy]

@coudenysj, about how to tell if a file is a shell script: I remembered myself using the file command:

for f in $(file path/to/scripts/* | grep 'shell script' | cut -d: -f1); do
  shellcheck "$f"
done

I think file uses the shebang as well to determine if a file is a shell script, and it might handle it better than a custom regex (#\!\(/usr/bin/env \|/bin/\)sh), though I can't confirm.

As a consequence, file will not detect shell scripts that do not have shebangs (like the ones we source, therefore not needing a shebang), but the grep search will not either. Though, I think it's conventional to name sourced files with the .sh extension, or .bash depending on the shell being targeted.

So a combination of find and file would be:

for f in $({ find path/to/scripts -type f -regex ".*\.\w*sh"
             file path/to/scripts/* | grep 'shell script' | cut -d: -f1
           } | sort -u); do  # note that we remove potential duplicates
  shellcheck "$f"
done

Do you think it's worth noting in the Recursiveness page :smile: ?

[coudenysj]

Thanks for the feedback @Pawamoy (learning more every day).

This is the biggest reason shellcheck is not bothering with recursiveness (Do One Thing and Do It Well) :nerd_face:. Everyone has its own way of writing scripts. I was just trying to kickstart the Recursiveness page, I'll try to incorporate your suggestions!

A couple of questions/notes:

• @koalaman Does shellcheck autodetect the shell?
• If I test different files with file I get "mixed" results (so no "parsed" type based on shebang).

bin/console: a /usr/bin/env php script, ASCII text executable
bin/restart-app: a /usr/bin/env sh script, ASCII text executable
bin/test: Bourne-Again shell script, ASCII text executable

I will include the file command on the wiki page, nevertheless.

[pawamoy]

@coudenysj You are absolutely right, finding all shell scripts is not a trivial task, and it's not in the scope of shellcheck!

About your mixed results: a better regex on file's output would be grep -E 'sh(ell)? script'. It will catch the /usr/bin/env ones, but not the Python script ones, nor the ones that contain "sh" like Flash.

However, file is not perfect: it recognizes several of my scripts as C source code, even though they have a #!/usr/bin/env bash shebang :confounded:. It might be because of the documentation comments I use, or the main function, I don't know.

[coudenysj]

@Pawamoy It doesn't really matter. We don't need to cover every single possibility.

The main reason of the Recursiveness is to give people hints about how to get started.

I'll update the page with the information we have, if other use cases come out of this thread, we can update the page accordingly.

[artem-zinnatullin]

Great addition to the project wiki!

Here is a snippet i use to run it in Docker container using shellcheck-alpine image provided by the project:

# shellcheck disable=SC2016
docker run \
--rm \
--volume "$(pwd)":/project:ro \
--entrypoint sh \
koalaman/shellcheck-alpine:v0.4.7 \
-c 'for file in $(find /project/ -type f -name "*.sh"); do
if ! shellcheck --format=gcc $file; then export FAILED=true; fi; done;
if [ "$FAILED" != "" ]; then exit 1; fi'

Note that it requires Alpine image to run sh (regular koalaman/shellcheck image doesn't have sh) and --entrypoint sh to override default one which is shellcheck itself

I've combined it with wiki snippet that produces exit code, so you can use it for CI and build scripts, but of course you can customize it.

Hope someone will find it useful! Maybe wiki should have it as well :)

---

Edit: update snippet to fix errors found by shellcheck (meta)

[rivertam]

@artem-zinnatullin Thank you so much for this. This is exactly what I needed.

It's worth noting, however, that I think if the inner part (the command) were to actually be shellchecked, I think it would complain about using find output for a for loop (SC2044).

I modified it once more to be cleaner IMO and to fit my needs personally.

docker run \
  --rm \
  --volume "$(pwd)/scripts":/mnt \
  --entrypoint sh \
  koalaman/shellcheck-alpine:latest \
  -c "
exitcode=0
find /mnt -type f > /tmp/filestolint
while IFS= read -r file
do
  echo \"Checking \$file\"
  if ! shellcheck --format=gcc \"\$file\"; then
    exitcode=1
  fi
done < /tmp/filestolint
exit \$exitcode"

The find command will need to be tweaked for your needs (for example, most people use .sh to represent shell files, while I decided not to for this project).

[pd93]

I found @rivertam's snippet above really useful, but I wanted to use this in my CI without having to write to a temporary file, so I wrote this script based on it.

Note the brackets around the while loop. This keeps the $EXITCODE variable from going out of scope because of the pipe. It also prints the error/success in red/green respectively for readability.

[JohnVonNeumann]

First of all, I'll say that this is an annoyance, I'm not convinced on the idea of a linter without the ability to recursively run, out of the box. I get that finding bash files can be annoying, but even if you just provided default "we only search for *.sh" files, I think this would satisfy a lot of requirements.

An actually productive comment though, I used @rivertam 's script and didn't get any luck (and also felt uncomfortable with such a long command) but edited it to:

docker run   --rm   --volume "$(pwd)":/mnt   --entrypoint sh   koalaman/shellcheck-alpine:latest   -c "shellcheck **/*.sh"

And got some love.

Basically, ripped from the Travis ci docs where they bundle SC into their boxes.

Hopefully this works for other people.

Peace.

[HariSekhon]

My check_bash_syntax.sh script does this and a bit more by pushd'ing to each script's src directory so relative source directives work too, working around issues https://github.com/koalaman/shellcheck/issues/908 and https://github.com/koalaman/shellcheck/issues/1837:

https://github.com/HariSekhon/DevOps-Bash-tools

recurse all *.sh under $PWD:

check_bash_syntax.sh

recurse all *.sh under somedir:

check_bash_syntax.sh somedir

check only these files and recurse only these given directories:

check_bash_syntax.sh file1 file2 file3 directory1 directory2

The reason it's called check_bash_syntax is that I also warn for using #!/bin/bash instead of #!/usr/bin/env bash.

This script is easily modified if you wanted to auto-recurse for contents of #!/bin/sh for example, but I think it's a good practice to put the .sh extension on all scripts as it is more explicit and intuitive at a glance without needing historical project knowledge or running grep / file on the contents of every file which is slower.

[corbolais]

The problem with this approach is that you cannot rely on the exit code of the find script (in a build/test script for instance).

You can use this to accomplish that:

for file in $(find path/to/scripts -type f); do shellcheck --format=gcc $file; done;

This recommendation in a shellcheck thread is ..uh ..curious. Using the above snippet in a script yields:

For loops over find output are fragile. Use find -exec or a while read loop. [SC2044]

cheers

[cove]

I think this would be better if it was inside of shellcheck like shfmt does it by default (e.g. shfmt -d .).

there's a few problems with using find with shellcheck I ran into:

• shell scripts sometimes don't have .sh extensions, so you have to determine if it's a shell script using file etc.
• if the shell script has relative paths in it, shellcheck needs to be run from the directory the script is in, which makes processing a git repo more complex to do efficiently

[sanmai-NL]

You can use shfmt -f . to get a list of all shell scripts. Complex logic can be added in shfmt.

[jmcantrell]

This is what I ended up using in my project, and what I consider the correct application of find+iteration. See https://mywiki.wooledge.org/BashFAQ/001 for details.

#!/usr/bin/env bash

set -eu

is_bash() {
    [[ $1 == *.sh ]] && return 0
    [[ $1 == */bash-completion/* ]] && return 0
    [[ $(file -b --mime-type "$1") == text/x-shellscript ]] && return 0
    return 1
}

while IFS= read -r -d $'' file; do
    if is_bash "$file"; then
        shellcheck -W0 -s bash "$file" || continue
    fi
done < <(find . -type f \! -path "./.git/*" -print0)

[dongleivip]

@koalaman I've created a Recursiveness wiki page.

Ah... I searched around and find this issue finally. However, it's been there for years...

I'd like to suggest putting it on the Readme page. 🙂

[iarjunphp]

For Windows (PowerShell script ) -https://arjunphp.com/shellcheck-check-directories-recursively/

[avidseeker]

Ironically, the suggested script to find by shebang has SC2038 warning.

I suppose this fixes it?

shellchecks() {
	find "${1:-.}" -type f -exec grep -Eq '^#!(.*/|.*env +)(sh|bash|ksh)' {} \; -exec shellcheck {} +
}

[navrkald]

Personally I want check all shell scripts, which are tracked in Git. For this purpose I am using KISS:

git ls-files | grep '.*.sh$' | xargs shellcheck