why shellcheck -x very slow?

[kamontat]

For bugs

• Rule Id (if any, e.g. SC1000):
• My shellcheck version (shellcheck --version or "online"): 0.4.6
• [ ] I tried on shellcheck.net and verified that this is still a problem on the latest commit
• [ ] It's not reproducible on shellcheck.net, but I think that's because it's an OS, configuration or encoding issue

Here's what shellcheck currently says:

Too slow

Here's what I wanted or expected to see:

Faster?

[nihilus]

good question, Ive noticed this as well, I guess following the sourced files and then rebuilding the script is what takes time.

[Wizek]

One thing that might help you is to run/compile+run shellcheck with profiling enabled. That will give you a .perf file detailing which functions take the longest to run that you can inspect.

If you are unsure how you could do that, this may help: http://book.realworldhaskell.org/read/profiling-and-optimization.html

[nihilus]

any progress on this?

[koalaman]

Are you saying that checking one script with -x is significantly slower than checking all the same scripts without -x?

[kamontat]

Yes! @koalaman

[nihilus]

@koalaman yupp, when you have got about 80 000 LOC this becomes kinda annoying since shellcjeck is the bottleneck in the build environment.

[kamontat]

Woww, Your project is huge. But It unnecessarily much like that. My project about 4xxx loc also extremely slow.

[koalaman]

Do either of you have a github repo or similar that I can use as a test case for this particular issue?

I can imagine several reasons why it would happen, but it would be good to have a realistic codebase to test it on.

[nihilus]

@koalaman one thing that adds overhead for me is the invokation of proot for chroot-magic for every invokation of shellcheck.

[koalaman]

@nihilus Your scripts source by absolute paths that don't correspond to the current system's?

[nihilus]

@koalaman that is correct... Most build enviroments for embedded Linux targets a "romfs" root directory in the build enviroment, and from there you create the file system for the flash mtd.

[rakus]

I can confirm the bottleneck. In my case it is easy to explain. I have a seven test scripts that test functions that are sourced from another file. Overall my test scripts source the script under test 20 times.

Checking without -x takes 2200 ms, with -x about 10092 ms (all numbers average of 20 executions).

Checking the sourced file with shellcheck takes about 412ms.

So: 2200 + 20 * 412 = 10440. Pretty close.

My case would benefit, if the sourced file would be cached.
BTW: The -x is only needed to prevent SC2034 and SC2154.

PS: My "problem" is insignificant compared to the others above.

[balupton]

For bevry/dorothy we've been using shellcheck via trunk. However, on our dev/devilbird branch which includes a year or updates, and also now has our 200 or so bash commands now sourcing a 5000 line bash.bash file, using shellcheck via trunk is now untenably slow. Running shellcheck commands/{the-command} works fine, however if I do shellcheck -x commands/{the-command} then yeah, it is extremely slow, so I guess Trunk uses the -x option. Running shellcheck commands/setup-util takes 5 seconds, shellcheck -x commands/setup-util takes 3-6 minutes. I've had to disable the shellcheck linter.

$  shellcheck --version
ShellCheck - shell script analysis tool
version: 0.9.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net

Here's some activity monitor stats for shellcheck -x commands/setup-util after just a minute or so of running:

After two minutes:

Okay, version 0.11 is an improvement, taking 2 minutes to complete, with an average memory of 2GB but peaking at 6GB once output starts:

$ shellcheck --version
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net

With shellcheck commands/setup-util (without -x) still being instant.