Feature Request: Support linting of shebang recipes

[JonathanDoughty]

I like that just lets me put a lot of what I would normally write as a mass of shell scripts into a Justfile. But when recipe logic and documentation as code leads me to write a shebang recipe it is not long before I wish I could use a linter to make sure I have not blundered in writing that. Since most of my shebang recipes are using bash I personally would like some way to have just save a recipe to its temporary file and then, instead of executing it, have it run a locally installed linter like ShellCheck against that file. Those using other languages would likely need a way to specify their own linters of choice, however.

Implementation-wise, perhaps a new just variable linter could be defined as a companion to shell to specify the default linter for recipe lines.

As an initial implementation perhaps it would be sufficient to have a just --lint [recipe] option that would write the designated recipe's body to the "temporary file" and have just simply output the path to that for users to lint manually. That might enable the maximum flexibility for users and other tools that might incorporate just capabilities. Or maybe a --linter option would work similarly to --shell to invoke the designated linter on recipe bodies. Another possibility might have just recursively call itself to invoke lint: target: recipes. Maybe recipe attributes could be extended to cover different shebang languages, i.e.,

[bash]
lint target:
    shellcheck {{target}}

[python3]
lint target:
    pylint {{target}}  

where again, the target path would come from just's temporary path generation. Perhaps ~/.user.justfile is a mechanism to leverage to specify user preferences for shell/linter correspondences via a map/associative array or recipe attributes as above.

I could do something with junk -s [recipe] | tail -n +3 | shellcheck - but then I have to figure out how to deal with variable substitutions. I'm not a rust developer nor familiar with just internals so I'm spit-balling here as a happy just user wanting to be happier.

Finally, perhaps this request overlaps somewhat with #1094 Question: Language Server, though that seems to me like a much bigger ask. Thanks for considering it, and for just.

[casey]

What about adding some way to print the body of a recipe to standard out, which could then be linted with the linter of the user's choice? We could add a recipe argument to just --dump which would make it print out just the body that recipe, the user could then lint it with whatever linter they liked.

[JonathanDoughty]

Sure, that should be sufficient. I'm assuming the dumped recipes would have {{variables}} replaced so they would not have syntax surprising to linters.

The only possible problem that occurs to me is that variable substitution might be beyond the normal meaning/user expectations of --dump. Would users expect that just --dump [recipe] abides by --dump-format? That would currently not do substitution for either just or json formats. Perhaps you would need to add a third --dump-format value - e.g. cooked vs. the raw of just format?

Related: what happens wrt variable substitution in non-shebang recipes with just --dump [recipe]? I can imagine uses for both cooked (which scratches my itch for shebang recipe linting) and just format.

Or perhaps I am misunderstanding and you had something else in mind with "... lint it with whatever recipe they liked".

[casey]

I'd probably just dump the text first, and worry about evaluating variables in a later PR. You're right that users probably expect --dump recipe to abide by --dump-format. Adding a cooked format is a good idea.

Related: what happens wrt variable substitution in non-shebang recipes with just --dump [recipe]? I can imagine uses for both cooked (which scratches my itch for shebang recipe linting) and just format.

I think i'd probably do raw first and cooked later.

Or perhaps I am misunderstanding and you had something else in mind with "... lint it with whatever recipe they liked".

Whoops! I meant whatever linter they liked.

[JonathanDoughty]

As a result of discovering #737 I've learned that if one uses the --verbose flag twice (who knew?, not sure when that got added) shebang recipes get printed with just variables substituted. Combine that with --dry-run and, at least with shellcheck capable scripts, a command line like

just -vv -n [recipe] 2>&1 | grep -v '===' > ${TMPDIR}$$ && shellcheck ${TMPDIR}/$$ || rm ${TMPDIR}$$

pretty much accomplishes the necessary goal, stripping out the "===> Running recipe..." verbose output, and saving the shebang text to a temporary file that a linter can run against. The only issue I've noticed so far are variables exported in the environment that the shebang script assumes exist but the linter knows nothing about. Some munging of --variables output along with the -vv -n output might address that.

For other just users who want to lint their shebang recipes this provides at least a partial solution that avoids the complexity of general linting support being added to just.

[wiktor-k]

Thanks for the workaround @JonathanDoughty ! I was also looking for a way to dump a single recipe (or all recipes which use the same shebang lines) to shellcheck it.

Even though the workaround works it'd be good to have a more sane approach. There's --dump flag maybe --dump-recipe would be a good idea?

[starthal]

A more general solution might look like a "structured dry-run". Something like

some_var = 'var_value'
export SOME_ENV := 'env_value'

a:
  touch foo
  @echo {{ some_var }}

b: a
  #!/usr/bin/env bash
  touch bar
  echo $SOME_ENV

c:
  echo "I'm unused"

Output of just --dry-run --dry-run-format=json b: (Using YAML for my sanity here)

recipes:
- recipe: 'a'
  is_dependency: true
  working_directory: '/some/path'
  kind: 'linewise'
  env: { 'SOME_ENV': 'env_value' }
  lines:
  - line: ['/bin/sh', '-c', 'touch foo']
    quiet: false
  - line: ['/bin/sh', '-c', 'echo var_value']
    quiet: true
- recipe: 'b'
  is_dependency: false
  working_directory: '/some/path'
  kind: 'scriptwise'
  env: { 'SOME_ENV': 'env_value' }
  script: |
  #!/usr/bin/env bash
  touch bar
  echo $SOME_ENV

Is there some internal intermediate structure that could serialize to something like this?

It should be enough information to convey to Shellcheck -- but would need to be done separately for each recipe. (Or using something like just --dry-run ... $(just --summary)

It intentionally doesn't retain information about just variables, justfile comments, the dependency graph, etc. It does retain everything needed to reproduce just's execution behavior.

Another use case, for which I'm currently using a similar --dry-run parsing hack to those above, is creating a Dockerfile based on a recipe. This output can remain constant even if the recipe c changes, which avoids needless cache invalidation.

[tgross35]

As a result of discovering #737 I've learned that if one uses the --verbose flag twice (who knew?, not sure when that got added) shebang recipes get printed with just variables substituted. Combine that with --dry-run and, at least with shellcheck capable scripts, a command line like

just -vv -n [recipe] 2>&1 | grep -v '===' > ${TMPDIR}$$ && shellcheck ${TMPDIR}/$$ || rm ${TMPDIR}$$

Slight improvement; shellcheck allows reading from stdin with - so there isn't any need to save to a file. The tee >(...) part is optional to also print the script to stdout.

just -vv -n $recipe 2>&1 | grep -v '===>' | tee >(shellcheck -)

Thank you for discovering this, it has been wonderfully helpful :)