[ShellScript] Add ZSH

[deathaxe]

Fixes #1358 Fixes #2635 Resolves #3826 Fixes #4033

This PR refactors Bash as foundation for adding Zsh.

Used syntax specifications:

• https://www.gnu.org/software/bash/manual/bash.html
• https://zsh.sourceforge.io/Doc/Release/

Syntax follows officiel specifications as close as possible to reliably support most language features.

Bash highlights

• tokenize shell words according to official specifications
• improve brace expansions (actually add support for them, everywhere)
• improve glob patterns
• improve parameter expansions
• improve tilde expansions
• add support for arithmetic highlighting shell words
• add support for command highlighting shell words
• drop generic built-in command specific contexts
• add syntax based folding
• improve indentation rules
• improve symbol definitions
• add completions for keywords, built-in commands and variables
• ...

Various scopes of keywords, punctuation, etc. have been modified to follow latest scope naming best practices.

Those may cause syntax tests of 3rd-party packages to fail. Syntax highlighting itself should however still work.

Zsh highlights

• support for ZSH specific alternative control structures
• support for expansion flags
• support glob modifiers

Benchmarks

Parsing performance is evaluated using syntax test files from current release and various other real world scripts.

This PR does not affect parsing performance.

[michaelblyons]

Oof. Can't with_prototype Shellscript anymore. It hits the sanity limit. I'm gonna have to make some edits to some things.

I'll report back on usage after a bit. First impression: lots of things are strings now.

[deathaxe]

This may be caused by various embed's being used to support shell highlighting in quoted strings, which are expanded to push...with_prototype, if ShellScript itself is used like this.

Well, Makefile, MArkdown and YamlPipelines successfully make use of extended ShellScript with little effort to inject own rules.

push...with_prototype is very likely to create undesirable results with most of current ST syntaxes anyway, due to the many special purpose context switches, they require to implement a language.

[michaelblyons]

Initial impression is good. I think I'm over the shock of so many things being "string" scoped now. There are a couple odd things I've found so far, but most of them are not really decidable. (Is for example awk -F' c-? ' a long option value, or an option with a string value?)

Is it worth making a special match for /dev/null? Is it possible to identify the variable being assigned to in mapfile so that it is a variable, not a string?

[deathaxe]

(Is for example awk -F' c-? ' a long option value, ...

Speaking for the shell, yes. Words are separated by space, parens and pipe operators, only. Thus -f' c-? ' is a single word.

It doesn't say anything about how the command awk interprets the words passed to it as arguments.

Quoting is however important for the shell with regards to character escaping, pattern matching or expansions being applied or not.

Is it worth making a special match for /dev/null?

Possibly. Maybe devices in general (/dev/...)?

Is it possible to identify the variable being assigned to in mapfile so that it is a variable, not a string?

Yes, sure. That's what the syntax does for declare, local, ... already to correctly scope indexed and associated arrays' values.

In general I am still thinking about how to scope l-values in assignments, so we can add them to symbol list or enable goto definition for them.

[michaelblyons]

In a case statement, if there is white space between the patter and the ), is that white space part of string.unquoted or should it be unscoped?

For the most part, I have been extremely pleased with the Bash syntax. I don't use ZSH, so someone else will have to look at that.

[deathaxe]

Case patterns actually are pattern lists, rather than a single pattern with alternation. It means each word is a single pattern pat1 | pat2 | pat3 ). As such whitespace doesn't belong to them and shoudln't receive string scope. Also | is not part of those patterns. It's rather a sequence separator or logical operator.

[michaelblyons]

@okdana @Antibioticss Test drive the ZSH, maybe?

[michaelblyons]

Okay, so there's a test failure from the Groovy changes, but otherwise still 👌🏼 from me.

[MatthiasPortzel]

I have a function alias, r, defined in my .zshrc. With this syntax, the closing bracket is highlighted red, despite working fine. Technically I may be overwriting an obscure ZSH builtin.

[deathaxe]

Thank's for your feedback. It turned out to be a long-living design/priority issue, which Bash on master suffers from as well. Fixed.

[deathaxe]

Great!