Make @sh work with objects; add @shassoc (fix #1947)

[nicowilliams]

trafficstars

• mention eval in @sh docs
• make @sh work on objects, emitting shell variable assignments for ident-like keys
• add @shassoc that emits text that can be used with declare -A x; eval x=($(jq -r ...))

[emanuele6]

The examples are wrong; they are using unquoted expansions that will split the output by whitespace, and then eval will join the splits back collapsing a sequence of multiple spaces into a single space. also globs will be expanded.

declare -A x; eval x=\($(jq -r '@shassoc' <<< '{"foo   bar": "x  /*   y"}')\)

$(jq -r '@shassoc' <<< '{"foo bar": "x /* y"}') will expand into to ['foo bar']='x /* y' since it is unquoted, it will be split into:

<x=(['foo> <bar']='x> </*> <y')>

And again, since it is unquoted, /* will expand, ending up running:

eval 'x=(['\''foo' 'bar'\'']='\''x' '/bin' '/etc' '/...' 'y'\'')'

That will eval x=(['foo bar']='x /bin /etc /... y'), which is wrong: all spaces collapsed into one, and /* is randomly expanded. Also /* may expand to something that injects syntax and runs a command.

Even for simple cases, [foo] is a glob pattern that matches either f or o, using an unquoted expansion will make it expand.

Anyway, sorry, but even if you fix the problem, I don't really like this implementation at all. And I don't think even think this is useful, or that we should add it.

Do you really want to add this feature?

[nicowilliams]

: ; declare -A x
: ; eval x=($(./jq -nr '{"a b":"*"}|@shassoc'))
: ; echo ${#x[@]} "${x[@]}"
1 *
: ; echo ${#x[@]} "${!x[@]}" "${x[@]}"
1 a b *
: ; eval x=($(./jq -nr '{"*":"*"}|@shassoc'))
: ; echo ${#x[@]} "${!x[@]}" "${x[@]}"
1 * *
: ; 

[emanuele6]

@nicowilliams

It only makes sense that x=(['*']='*') doesn't glob expand to the files in your directory, the are no spaces so it is a glob in its entirety. It would expand to files named x=(*='foo') or x='=('foo bar' baz') in your current directory if there were any (and it would expand to nothing if nullglob was in use, like ['foo']='bar' also would since [] is a glob component too). And "a b" does not have sequence of multiple spaces so you don't notice that the sequences of spaces are collapsed into a single space by eval joining them back after splitting.

$ (declare -A x; eval x=($(./jq -nr '{"*":"*"}|@shassoc')); declare -p x)
declare -A x=(["*"]="*" )
$ touch "x=(*='foo' bar')" "x=('='x')"
$ (declare -A x; eval x=($(./jq -nr '{"*":"*"}|@shassoc')); declare -p x)
declare -A x=(["*=foo"]="bar) x=(=x" )
$ (declare -A x; eval "x=($(./jq -nr '{"*":"*"}|@shassoc'))"; declare -p x) #quoted
declare -A x=(["*"]="*" )

You can also check my example:

$ (declare -A x; eval x=\($(./jq -r '@shassoc' <<< '{"foo   bar": "x  /*   y"}')\); declare -p x)
declare -A x=(["foo bar"]="x /bin /boot /dev /etc /home /lib /lib64 /lost+found /mnt /opt /proc /root /run /sbin /srv /sys /tmp /usr /var y" )
$ (declare -A x; eval "x=($(./jq -r '@shassoc' <<< '{"foo   bar": "x  /*   y"}'))"; declare -p x) #quoted
declare -A x=(["foo   bar"]="x  /*   y" )

[nicowilliams]

Oh, I see. Maybe then @shassoc should expand to var[index]=value with the index and value escaped? but whence the associative array variable name? From an empty key's value?

[emanuele6]

You just need to double quote the expansion so it is not splitted and glob expanded: eval "x=($(jq ...))" instead of eval x=($(jq ...)); also note that you don't need to use eval and you can use declare directly in bash: declare -A "x=($(jq ...))".

Anyway, I am still of the opinion that this is not a good feature to add. And also note that bash (but not ksh93) disallows the use of '' as key for an associative array, and you are not handling that case which will result in an assignment error.

[nicowilliams]

Anyway, I am still of the opinion that this is not a good feature to add. And also note that bash (but not ksh93) disallows the use of '' as key for an associative array, and you are not handling that case which will result in an assignment error.

Fair enough. I'll remove it.

[emanuele6]

Can you write `@sh` instead of @sh in your commit message so github does not render it as a user mention ^^

[xPMo]

I recently made a jq module to do some of this, including the functionality that was removed. With the goal of mapping objects to identifiers, sh::param has an optional prefix argument (default is _), so you don't clobber important shell parameters on accident, and can namespace multiple calls to the function.

I'm looking to find the best way to incorporate which $SHELL the user is running without turning the function into a messy 4x nested if-else tree.

Not disparaging a C builtin approach, but I just wanted to add this as option, especially if someone in the future found this PR from the issue search (like I did).

[nicowilliams]

Not disparaging a C builtin approach, but I just wanted to add this as option, especially if someone in the future found this PR from the issue search (like I did).

No disparagement taken :)

By the way, your comment prompted me to see what it would take to add user-defined formatting specifications, and... it can be done right now:

def of($fmt): format($fmt); # original `format`
def format($fmt): if $fmt == "custom" then "CUSTOM: " + . else of($fmt) end;

@custom "hi \("there")"

Now, modules can't quite do this unless they're included instead of imported, but one could always document that the user of a module that defines formats can def format($fmt): the_module::format($fmt); to use them.

[wader]

By the way, your comment prompted me to see what it would take to add user-defined formatting specifications, and... it can be done right now:

There is jaq PR https://github.com/01mf02/jaq/pull/105 discussion add formatters and custom formatters by allowing a function to start with "@" which seems quite nice.

[nicowilliams]

There is jaq PR 01mf02/jaq#105 discussion add formatters and custom formatters by allowing a function to start with "@" which seems quite nice.

I like that, yeah.