Support for pre-declared "external" globals for sourced scripts to avoid SC2154 warnings.

[deryni]

We have a series of scripts that operate on a set of shared variables that are created by the top-level script.

The secondary scripts use these variables without declaring them as they are known to exist when run.

shellcheck, obviously, doesn't know this. It would be nice if we could tell it about acceptable external global variables to disable SC2154 warnings about them.

I realize we could use # shellcheck disable=SC2154 for them but that's a larger hammer than I would like as it will also disable other warnings from typos and the like if they happen to appear on that line.

The context=sourced suggestion from https://github.com/koalaman/shellcheck/issues/281 is somewhat related to this request.

[koalaman]

Global variables are conventionally in all uppercase (PATH, DISPLAY, SSH_CLIENT, etc). For this reason, ShellCheck generally doesn't warn when using undefined uppercase variables.

The best solution would probably be for ShellCheck to process sourced files though.

[deryni]

That's true for system variables but I'm not sure that's as true for script-internal global variables (but maybe it is though that feels like playing in the system's/shell's playground a little bit).

Processing sourced files would be interesting though, assuming you mean when you pass shellcheck the toplevel file it would "recurse" into the sourced files and consider them in that context.

That wouldn't solve all of my cases (we have some scripts that get sourced and others that get run that all expect the external globals) but it would solve some of them.

[moeriki]

When writing scripts for preset systems this uppercase convention can't always be adhered to.

Case in point. I'm writing scripts for Atlassian Bamboo. There is a global variable bamboo you can refer to for certain configurational values.

[deryni]

Bamboo uses many variables with the bamboo_ prefix but not a single variable. (Though using a single associative array variable would certainly be nice.) But yes, same idea.

[koalaman]

I ran into the same issue with a library set FLAGS_verbosity style variables.

How about a flag for setting a pattern/regex to ignore? This would let you do something like

export SHELLCHECK_OPTS="--ignore-variable-pattern=FLAGS_|bamboo"

to not get these warnings anymore. Alternatively as a directive, starting the file with

# shellcheck ignore-variable-pattern=FLAGS_|bamboo

[deryni]

Seems reasonable and would work for me. Though I'd probably suggest making it require explicit pattern markings (or having both pattern and non-pattern versions so that literal variables could be included in the list safely as well (i.e.

--ignore-unassigned-variables='FLAGS_*|bamboo_*|explicit_global'

or

--ignore-unassigned-variable-patterns='FLAGS_|bamboo_' --ignore-unassigned-variables=explicit_global

That said it also might make sense to handle this more generically (the way we discussed in #380, that is with names/patterns attached to the normal directives).

As the link in my comment on #380 is apparently now dead here's an updated link to the luacheck documentation for how it handles this idea.

The gist is that in addition to a directive like # shellcheck disable=SC2154 you could use # shellcheck disable=SC2154/FLAGS_* or # shellcheck disable=SC2154/bamboo_* or # shellcheck disable=SC2154/explicit_global to scope the disabling to that specific variable/pattern. (I realize that is likely a fair bit more complex of an addition though. Though if my haskell was anything but non-existent I'd be interested in taking a crack at it.)

[moeriki]

Bamboo uses many variables with the bamboo_ prefix but not a single variable.

@deryni That depends. When running inline scripts it's a single variable. ${bamboo.myvar} When running linked scripts it's all separate. ${bamboo_myvar}

Bamboo variables are exported as bash shell variables. All full stops (periods) are converted to underscores. For example, the variable bamboo.my.variable is $bamboo_my_variable in bash. This is related to File Script tasks (not Inline Script tasks).

• https://confluence.atlassian.com/bamboo/bamboo-variables-289277087.html

[deryni]

@Moeriki ${bamboo.myvar} isn't a shell variable. It isn't a legal variable name. (And even if it was that would still be many variables and not a single variable as that isn't how you index into a shell array variable that would be ${bamboo["myvar"]}.)

What's happening there is that bamboo is pre-processing the script for you and replacing that (and I didn't know that worked in inline script bodies, I've never used it, but did know it worked in other places in bamboo itself).

The fact that it works in inline script bodies is sort of scary actually (likely safe-ish and possibly even entirely safe but still not something I want). Just one more reason for me to work on removing them from my builds entirely. =)

And I'm not sure what that "This is related to File Script tasks (not Inline Script tasks)." bit is supposed to mean but the shell variables work in inline scripts just fine as well (I suppose it would have been possible for Atlassian not to export the variables to the environment for inline scripts but that would be odd and they didn't.)

[moeriki]

@deryni Ah I see. Thanks for the insight!

@koalaman your proposition would work fine for me.

[dflock]

I have the same issue. Maybe it would be possible to use a new directive to point to the parent script, where the variables are defined?

[tomprince]

For example, for this file, I get (slightly reformated):

   7:18  warning  srcdir is referenced but not assigned.                    2154 (shellcheck)
  13:19  warning  _objdir is referenced but not assigned.                   2154 (shellcheck)
  18:5   warning  moztopsrcdir appears unused. Verify it or export it.      2034 (shellcheck)
  19:5   warning  commtopsrcdir appears unused. Verify it or export it.     2034 (shellcheck)
  21:5   warning  mozreltopsrcdir appears unused. Verify it or export it.   2034 (shellcheck)
  22:5   warning  commreltopsrcdir appears unused. Verify it or export it.  2034 (shellcheck)
  24:5   warning  commtopobjdir appears unused. Verify it or export it.     2034 (shellcheck)

I would like to specify something like

# shellcheck external_vars="srcdir _objdir"
# shellcheck exported_vars="moztopsrcdir commtopsrcdir mozreltopsrcdir commreltopsrcdir commtopobjdir"

to be explicit about what variables are defined by the sourcing script, and which are used by the sourcing script.

[mat813]

In the FreeBSD ports tree, we use environment variables extensively to pass values from the make(1) framework to the shell scripts doing the more complex stuffs, and I am in the process of cleaning things up it would be nice to have a directive where one could add an exhaustive list of already existing variables, to be able to turn SC2154 on.

[plambert]

This'd be very useful for me, as well; while I'm not a strong enough developer to make a patch or pull request, if there's anything I can do to help accomplish this, let me know.

[matthewpersico]

Am I missing something? Seems to me that if you allow

# shellcheck external_vars="srcdir _objdir"

in one file, you're gonna need it in others. And then you'll get the bright idea to put the common declarations in one file, which you'll need to...

wait for it...

source in! :-)

So instead of a directive, how about just dropping a bunch of declare statements in a file foo.bashinc and sourcing that in with a hard coded path?

[mat813]

So, your idea is nice, for people who have variables that are the same in all their scripts.

It is absolutely not the case for the FreeBSD ports framework usage, a directive is really the only way to go for us to be able to use SC2154.

[Screwtapello]

Another use-case: The editor Kakoune uses POSIX sh for scripting, and exports various bits of internal state as shell variables. All these variables follow the pattern kak_* so it would be really handy to be able to tell ShellCheck to ignore variables matching that pattern.

[jasonkarns]

Another use case: Bats, which sets status, output and lines (https://github.com/sstephenson/bats#run-test-other-commands)

(related to #417 as a directive to silence bats-specific variables would aid in shellcheck supporting bats files.)

[dermoth]

Hi... I have a similar requirement, however since the script is closely tied to another I was rather thinking to tell shellcheck to "source" the file... I tried adding a source=<otherfile> but it's a no-op unless followed by something like . /dev/null

Wouldn't it make sense to allow source=... just as a hint of pre-declared variables, without actually sourcing it. Or maybe even better cross-link source file and parent and scope it from there, ex (maybe with something else than source= in the 2nd script...):

main.sh

#!/bin/bash

myvar=foo

for f in inc/*.sh
do
    #shellcheck source=inc/foo.sh source=inc/bar.sh source=inc/baz.sh#  BTW wildcards would be nice here....
    . "$f"
done

myprivatevar=bar

foo.sh

# because we don't actually source main.sh but main.sh sources us, scope this script from matching main.sh source
# shellcheck shell=bash source=../main.sh # (or maybe something else like parent=../main.sh)

othervar=$myvar       # OK - already defined before main.sh source
echo "$myprivatevar"  # ERROR - not in scope

[abouvier]

Could it also be possible to support source=path/to/script.sh directive in a .shellcheckrc file, with script.sh containing all the exported variables.

[Jonta]

Could it also be possible to support source=path/to/script.sh directive in a .shellcheckrc file, with script.sh containing all the exported variables.

Hopefully not in home. Ref. https://wiki.archlinux.org/index.php/XDG_Base_Directory =)

[mnuccioarpae]

Hi... I have a similar requirement, however since the script is closely tied to another I was rather thinking to tell shellcheck to "source" the file... I tried adding a source=<otherfile> but it's a no-op unless followed by something like . /dev/null

@dermoth I have tried this approach, but it does not work because it creates an infinite loop.

For example, if Shellcheck starts from the main script, then sources the child script and then sources the main script again. The same thing happens if you start from the child script. I imagine Shellcheck can detect the loop, but the problem is that you are passing to Shellcheck the wrong information: the child script never sources the main script during execution.

[dermoth]

Can you break when you have already read a file?

Or maybe a "sourced by" directive, where you start form the sourced file up to the line or directive that sources the current script.

[mnuccioarpae]

I think a "sourced by" directive is cleaner because it expresses what happens during script execution.

But what do you do if the sourcing script is unknown or when is not a script that defines the external variables?

For now, I am using the workarounds suggested by @koalaman at https://github.com/koalaman/shellcheck/issues/2356#issuecomment-944608623

[ShamrockLee]

Another use case: Nixpkgs setup hooks.

Nixpkgs is a package repository of Nix package manager, and setup hooks are shell scripts meant to be sourced when building a package. There are dependency relation between setup hooks, and all hooks depend on a base setup.sh, assuming the latter has been sourced in advance.

It would be great to have a way to hint ShellCheck that some scripts has been sourced before this one.

[ShamrockLee]

Or, as a workaround, this would also be achievable by supporting a per-file check-sourced directive. This way, one can write a meta-script, sourcing the external scripts without checking, and eventually source the target script and specify that it be checked.