bash-completion fails to generate path completions with `$(`

[siteshwar]

Describe the bug

If bash-completion package is installed, bash does not generate completions with $(.

To reproduce

1. Execute touch test.out in current directory.
2. Execute for i in $(cat <TAB>.
3. No completions generated for cat command.

Expected behavior

There should be completions listed for cat command.

Versions (please complete the following information)

• [x] Operating system name/distribution and version: Fedora release 33 (Thirty Three)
• [x] bash version, echo "$BASH_VERSION": 5.0.17(1)-release
• [x] bash-completion version, (IFS=.; echo "${BASH_COMPLETION_VERSINFO[*]}"): 2.8

Additional context

It is caused by a behavior change introduced in bash-4.3 (by this commit to bashline.c).

I discussed this change with Chet over e-mail and this is what he mentioned:

It looks like the completer for `for' doesn't know how to handle getting the command substitution as a single word (since that's what it is). I guess it must be trying to parse that out on its own, using COMP_WORDS, since you get this when you run an instrumented function as the completer:

touch /tmp/test.txt for f in $(cat /tmp/te[TAB]

args: for /tmp/te in declare -a COMP_WORDS=([0]="for" [1]="f" [2]="in" [3]="$(cat /tmp/te") COMP_CWORD=3

Note that the word readline is trying to complete is always passed to the completion function as $2, and that is /tmp/te. But the words on the line are passed as COMP_WORDS, and ${COMP_WORDS[$COMP_CWORD]} is as above.

Bash-4.2 looked `into' the command substitution and passed

args: cat /tmp/te cat declare -a COMP_WORDS='([0]="cat" [1]="/tmp/te")' COMP_CWORD=1

and so ran the completion for `cat'.

But that was fragile, and didn't take into account other command substitutions on the line. For instance, when finding the command name to be completed, we want to skip over command substitutions in assignment statements, something bash-4.2 had real problems with. Here's a bug report on that:

https://lists.gnu.org/archive/html/bug-bash/2011-12/msg00053.html

So bash-4.3 changed this. The actual change was 12/16/2011 to bashline.c: find_cmd_start(), which now considers command substitutions as single words, or parts of single words, as the rest of the shell does. So the words that a completer gets are closer to the words that the parser will eventually produce. That is why bash-4.2 thinks the command name is cat' and bash-4.3 thinks it's for'.

If bash-completion has a function that understands how to complete words beginning with `$(', possibly recursively, that could work. Or the function could look at $2 to see the word that readline thinks should be completed (readline's pretty generic, and just goes back to the whitespace).