akinomyoga
`--` breaks on some commands
ble version: 0.4.0-nightly+32f290d Bash version: 5.2.37(1)-release (aarch64-apple-darwin23.4.0)
On some commands I noticed that typing -- will break the current input and leak other output.
eg
brew list --
As soon as I've typed the -- this happens:
$ brew list -/usr/bin/awk: towc: multibyte conversion failure on: 's API for new formulae or cask data every'
-- INSERT --
input record number 4, file
source line number 29
--/usr/bin/awk: towc: multibyte conversion failure on: 's API for new formulae or cask data every'
input record number 4, file
source line number 29
/usr/bin/awk: towc: multibyte conversion failure on: 's API for new formulae or cask data every'
input record number 4, file
source line number 29
/usr/bin/awk: towc: multibyte conversion failure on: 's API for new formulae or cask data every'
input record number 4, file
source line number 29
/usr/bin/awk: towc: multibyte conversion failure on: 's API for new formulae or cask data every'
Thanks for the report. This type of error message is specific to macOS awk, but I don't have macOS so cannot test it. MacOS awk produces error messages when it sees data that is not compatible with the current encoding. As far as I search GitHub for the data in the error message, it seems to come from the man page of brew.
- Q1: What are the results of the following commands in your environment?
$ man brew | grep 'API for new formulae' | cat -v
$ man brew | grep 'API for new formulae' | /usr/bin/awk '{print NF;}'
Thanks for the prompt follow-up @akinomyoga much appreciated. Also really appreciate the great work on this awesome project. ✨
As for your question, indeed I see some weird encoding in the first command output:
$ man brew | grep 'API for new formulae' | cat -v
Check HomebrewM-^@M-^Ys API for new formulae or cask data every
$ man brew | grep 'API for new formulae' | /usr/bin/awk '{print NF;}'
10
Thanks for the result.
indeed I see some weird encoding in the first command output:
The weird encoding you see is actually what is intended by cat -v.
$ man brew | grep 'API for new formulae' | cat -v Check HomebrewM-^@M-^Ys API for new formulae or cask data every
I actually expect M-bM-^@M-^Y if the code \u2019 is properly encoded, but the beginning byte (which is supposed to be converted to M-b by cat -v) seems to be missing in your output. It might be a characteristic of macOS cat, so I'd like to confirm the output of the following command:
- Q2: What is the result of the following command in macOS?
$ echo $'\u2019' | cat -v
$ man brew | grep 'API for new formulae' | /usr/bin/awk '{print NF;}' 10
I thought macOS /usr/bin/awk would have a problem with processing the line containing "s API for new formulae or cask data every", but the above result seems to suggest that it actually works. Maybe it is related to the locale.
- Q3: What are the results of the following commands?
$ ble/widget/display-shell-version
$ locale
$ man brew | grep 'API for new formulae' | LC_ALL=C /usr/bin/awk '{print NF;}'
Thank you
Q2
$ echo $'\u2019' | cat -v
M-^@M-^Y
Q3
$ ble/widget/display-shell-version
-bash: `ble/bin/diff': not a valid identifier
-bash: ble/bin/diff: No such file or directory
GNU bash, version 5.2.37(1)-release (aarch64-apple-darwin23.4.0)
ble.sh, version 0.4.0-nightly+31f264a (noarch) [git 2.46.2, GNU Make 4.3, GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)]
locale: LANG=en_US.UTF-8
terminal: TERM=tmux-256color wcwidth=15.1-west/16.0-2+ri, tmux:0 (84;0;0)
options:
$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
$ man brew | grep 'API for new formulae' | LC_ALL=C /usr/bin/awk '{print NF;}'
this one does not seem to produce any output, its not exiting either
$ man brew | grep 'API for new formulae' | LC_ALL=C /usr/bin/awk '{print NF;}'this one does not seem to produce any output, its not exiting either
Thank you. Hmm, this is interesting.
The above result seems to contain an empty line. Is that a part of the output of the command? (I.e., does the command output an empty line? or does it output nothing?)
Anyway, as far as the output of man brew | grep 'API for new formulae' is not changed since the last time, this means that /usr/bin/awk doesn't function as expected when the locale is forced to C by LC_ALL=C.
- Q4: Could you provide the results of the following commands?
$ echo $'\u2019' | od -t x1
$ man brew | grep 'API for new formulae' | od -t x1
$ echo $'\u2019' | /usr/bin/awk '{print NF;}'
$ echo $'\u2019' | LC_ALL=C /usr/bin/awk '{print NF;}'
$ ble/widget/display-shell-version -bash: `ble/bin/diff': not a valid identifier -bash: ble/bin/diff: No such file or directory
This suggests another problem with the test for the existence of the command diff.
- Q5: Could you also check the output of the following command?
$ type -a diff
$ (set -x; ble/bin#freeze-utility-path diff; echo "exit status: $?"; type -a ble/bin/diff)
Ah, actually I recently set nvim to be my man pager, I guess this is the reason piping the man page does not produce output. Resetting it to less does produce output:
$ MANPAGER=less
$ man brew | grep 'API for new formulae' | LC_ALL=C /usr/bin/awk '{print NF;}'
10
Q4
$ echo $'\u2019' | od -t x1
0000000 e2 80 99 0a
0000004
$ MANPAGER=less
$ man brew | grep 'API for new formulae' | od -t x1
0000000 20 20 20 20 20 20 20 20 20 20 20 20 20 20 43 68
0000020 65 63 6b 20 48 6f 6d 65 62 72 65 77 e2 80 99 73
0000040 20 41 50 49 20 66 6f 72 20 6e 65 77 20 66 6f 72
0000060 6d 75 6c 61 65 20 6f 72 20 63 61 73 6b 20 64 61
0000100 74 61 20 65 76 65 72 79 0a
0000111
$ echo $'\u2019' | /usr/bin/awk '{print NF;}'
1
$ echo $'\u2019' | LC_ALL=C /usr/bin/awk '{print NF;}'
1
Q5
$ type -a diff
diff is /usr/bin/diff
diff is /opt/homebrew/bin/diff
$ (set -x; ble/bin#freeze-utility-path diff; echo "exit status: $?"; type -a ble/bin/diff)
>>>> ble/bin#freeze-utility-path diff
>>>> local cmd path 'q='\''' 'Q='\''\'\'''\''' fail= flags=
>>>> for cmd in "$@"
>>>> [[ diff == -n ]]
>>>> [[ '' == *n* ]]
>>>> ble/bin#has ble/bin/.frozen:diff
>>>> builtin type -t -- ble/bin/.frozen:diff
>>>> ble/bin#get-path diff
>>>> local cmd=diff
>>>> ble/util/assign path 'builtin type -P -- "$cmd" 2>/dev/null'
>>>> local _ble_local_tmpfile
>>>> ble/util/assign/mktmp
>>>> _ble_local_tmpfile=/tmp/blesh/502/82534.util.assign.tmp.0
>>>> (( BASH_SUBSHELL ))
>>>> _ble_local_tmpfile=/tmp/blesh/502/82534.util.assign.tmp.0.83911
>>>> builtin eval -- 'builtin type -P -- "$cmd" 2>/dev/null'
>>>>> builtin type -P -- diff
>>>> local _ble_local_ret=0 _ble_local_arr=
>>>> mapfile -t _ble_local_arr
>>>> ble/util/assign/rmtmp
>>>> (( _ble_util_assign_level-- ))
>>>> (( BASH_SUBSHELL ))
>>>> printf 'caller %s\n' ble/util/assign/rmtmp ble/util/assign ble/bin#get-path ble/bin#freeze-utility-path
>>>> IFS='
'
>>>> builtin eval 'path="${_ble_local_arr[*]}"'
>>>>> path=/usr/bin/diff
>>>> return 0
>>>> [[ -n /usr/bin/diff ]]
>>>> [[ /usr/bin/diff == ./* ]]
>>>> [[ /usr/bin/diff == ../* ]]
>>>> builtin eval 'function ble/bin/diff { '\''/usr/bin/diff'\'' "$@"; }'
-bash: `ble/bin/diff': not a valid identifier
>>>> (( !fail ))
>>>> echo 'exit status: 0'
exit status: 0
>>>> type -a ble/bin/diff
-bash: type: ble/bin/diff: not found
last one did exit with exit code 1
If it is of any value, here is another example of the behavior:
$ lsof -i --/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
-- INSERT --
input record number 4, file
source line number 29
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 29
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 29
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 29
--
Okay, seems like it's not necessarly breaking on --, but on - and then on the next character the output is slipping through, on any subsequent input the same thing is happening.
In this video you can see my typing lsof - and as soon as I type a the output is produced and then as soon as I type a again the behavior is repeated as well.
https://github.com/user-attachments/assets/b8677ecc-a08b-474d-8a55-0494e55bce75
$ type -a lsof
lsof is /usr/sbin/lsof
So it might be something that is not limited to brew.
Thank you for the answers and additional information.
Ah, actually I recently set
nvimto be my man pager, I guess this is the reason piping the man page does not produce output. Resetting it tolessdoes produce output:
OK, the new result seems consistent with the previous one, yet the original error message is not reproduced. Maybe there are other conditions for the error message to be output.
$ echo $'\u2019' | od -t x1 0000000 e2 80 99 0a
I suspected the possibility of an unexpected behavior of $'\u2019', but it seems OK according to the above result.
$ man brew | grep 'API for new formulae' | od -t x1 0000000 20 20 20 20 20 20 20 20 20 20 20 20 20 20 43 68 0000020 65 63 6b 20 48 6f 6d 65 62 72 65 77 e2 80 99 73 0000040 20 41 50 49 20 66 6f 72 20 6e 65 77 20 66 6f 72 0000060 6d 75 6c 61 65 20 6f 72 20 63 61 73 6b 20 64 61 0000100 74 61 20 65 76 65 72 79 0a
Seems fine. The only non-7-bit characters are "80 99 73" (which is the UTF-8 representation of \u2019). Anyway, /usr/bin/awk does not produce the error message with this data, so this may not contain a hint.
$ echo $'\u2019' | /usr/bin/awk '{print NF;}' 1 $ echo $'\u2019' | LC_ALL=C /usr/bin/awk '{print NF;}' 1
These are also consistent with the previous results.
Okay, seems like it's not necessarly breaking on
--, but on-and then on the next character the output is slipping through, on any subsequent input the same thing is happening.
I suspected that the man page of brew contains a misencoded character or some characters that /usr/bin/awk cannot handle, but yeah, other man pages can also contain similar data on which /usr/bin/awk would produce the error message.
In this video you can see my typing
lsof -and as soon as I typeathe output is produced and then as soon as I typeaagain the behavior is repeated as well.
Thank you for the additional explanation. I'm sorry that I haven't explained the details to you, but I knew that the reported error is related to the extraction of the option description from the man page in the auto-complete feature of ble.sh, and that repeated error is normal when the extraction is broken. The auto-complete feature attempts to generate the candidate words based on the programmable completion setting on every character insertion, so the error message is repeated. When the current word starts with -, ble.sh tries to extract the options and descriptions from the man page of the current command, so the error message appears with -<something>. ble.sh uses awk to process the man page content. The error message is output in this process and seems to be related to the specific content of the man page of brew, lsof, and others.
I'd like to find out anything common in the affected parts of the man pages in brew and lsof.
- Q6: What is the result of the following command?
$ MANPAGER=less
$ man lsof | grep ' AIX:' | cat -v
>>>> builtin eval 'function ble/bin/diff { '\''/usr/bin/diff'\'' "$@"; }' -bash: `ble/bin/diff': not a valid identifier
Hmm, OK. Do you set set -o posix? If so, I can add a workaround for this issue.
- Q5a: What is the result of the following command?
$ declare -p POSIXLY_CORRECT
I think I need to ask you to run a patched version of ble.sh to record some internal data. I'll prepare it and attach .zip later.
Appreciate the explantations very much. 🙏
I think I need to ask you to run a patched version of ble.sh to record some internal data. I'll prepare it and attach .zip later.
Sounds good, thank you.
Q6
$ MANPAGER=less
$ man lsof | grep ' AIX:' | cat -v
# nothing
$ man lsof | grep ' AIX:' | od -t x1
# nothing
$ man lsof | grep 'AIX:' | cat -v
AIX:
$ man lsof | grep 'AIX:' | od -t x1
0000000 20 20 20 20 20 20 20 c2 a0 c2 a0 c2 a0 c2 a0 41
0000020 49 58 3a 0a
0000024
Q5a
Yes, set -o posix is part of my bashrc.
$ declare -p POSIXLY_CORRECT
declare -- POSIXLY_CORRECT="y"
- Q7: Can you try
core-complete.tar.gz?- Download the tar archive from the above link.
- Extract the file
core-complete.shby runningtar xzf core-complete.tar.gz. - Overwrite
lib/core-complete.shin the directory where ble.sh is installed with the extracted file. You can runmv -f core-complete.sh "$_ble_base/lib/core-complete.sh"in a ble.sh session. - Start a new interactive Bash session. For example, you can run
bashor create another tmux pane. Or you can open a new terminal window. - Input
lsof -and press TAB to produce an error message. - If some data that cannot be handled by
/usr/bin/awkis detected, the data will be saved in~/blesh-debug-mandb-lsof.tar.gz. Could you attach the file~/blesh-debug-mandb-lsof.tar.gzif it is created?
I actually seem to hit this case:
Error not found in the intermediate data of the man page analysis.
OK, thank you! I'm now preparing the next version to check another place.
- Q7a: Could you try this:
core-complete-v2.tar.gz? The test procedure is the same as Q7, but the created file will be~/blesh-debug-mandb-lsof-v2.tar.gz.
Hm I can't get it to produce the debug file for some reason.
https://github.com/user-attachments/assets/d1e7bb24-460b-4c25-9752-5edfdc3cbc51
Hm I can't get it to produce the debug file for some reason.
Did you instead see the message debug (check2): Error not found.?
Just did a search on the output against this, but could not find any match. Including the full output here for reference:
$ lsof -aa/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
-- INSERT --
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
-aaa/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
-aaaa/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
-aaaa
Also just checked the patch again to confirm I'm running the latest patch:
$ grep "check2" $_ble_base/lib/core-complete.sh
function __blesh_debug_mandb_check2__ {
echo 'debug (check2): Error not found.' >/dev/tty
__blesh_debug_mandb_check2__ "${subcaches[@]}"
Thank you.
- Q7b: Could you try this?
core-complete-v3.tar.gz. This time a new file will not be created. I'd like to narrow down the possible places where the error is produced. Could you copy the output in the terminal and paste it here?
Sounds good, ty.
$ lsof -able/complete/mandb/generate-cache: step1 (caller ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook)
ble/complete/mandb/generate-cache: step3a (caller ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook)
awk: ble/function#advice/before:ble/bin/awk ble/function#try ble/function#advice/.proc ble/bin/awk ble/complete/mandb/.generate-cache-from-man ble/complete/mandb/generate-cache ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook / 1 5 12 1 16 52 3 9 19 35 13 21 18 15 16 20 1 38 3 10 70 1
awk: ble/function#advice/before:ble/bin/awk ble/function#try ble/function#advice/.proc ble/bin/awk ble/complete/mandb/.generate-cache-from-man ble/complete/mandb/generate-cache ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook / 1 5 12 1 341 52 3 9 19 35 13 21 18 15 16 20 1 38 3 10 70 1
/usr/bin/awk: towc: multibyte conversion failure on: ' AIX:'
input record number 4, file
source line number 35
Error not found in the intermediate data of the man page analysis.
ble/complete/mandb/generate-cache: step3b (caller ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook)
ble/complete/mandb/generate-cache: step4 (caller ble/complete/mandb/load-cache ble/complete/source:option/generate-for-command ble/complete/source:option ble/complete/source:argument/generate ble/complete/source:argument ble/complete/candidates/generate-with-filter ble/complete/candidates/generate ble/complete/auto-complete/source:syntax ble/complete/auto-complete.impl ble/complete/auto-complete.idle ble/util/idle.do/.call-task ble/util/idle.do ble-edit/bind/.tail ble-decode/EPILOGUE _ble_decode_hook)
-a
Thank you for your quick responses. So, the error seems to be produced inside ble/complete/mandb/.generate-cache-from-man for sure.
- Q7c: Could you try this:
core-complete-v4.tar.gz? This time, the file~/blesh-debug-mandb-lsof.tar.gzwill be unconditionally created.
- Q8: What is the result of the following command?
$ /usr/bin/awk -W version || /usr/bin/awk --version
$ /usr/bin/awk -W version
/usr/bin/awk: unknown option -W ignored
$ /usr/bin/awk --version
awk version 20200816
- Q8a: How about this?
$ path=/usr/bin/awk
$ ble/util/assign version '"$path" -W version || "$path" --version' 2>/dev/null </dev/null
$ declare -p version
$ path=/usr/bin/awk
$ ble/util/assign version '"$path" -W version || "$path" --version' 2>/dev/null </dev/null
$ declare -p version
declare -- version=""
Strange...
- Q8b: How about these?
$ { /usr/bin/awk -W version || /usr/bin/awk --version; } 2>/dev/null </dev/null
$ /usr/bin/awk --version 2>/dev/null
$ /usr/bin/awk --version >/dev/null
$ { /usr/bin/awk -W version || /usr/bin/awk --version; } 2>/dev/null </dev/null
# nothing
$ /usr/bin/awk --version 2>/dev/null
awk version 20200816
$ /usr/bin/awk --version >/dev/null
# nothing
Thank you for the results. Ah, OK. /usr/bin/awk -W version </dev/null seems to succeed even though it produces an error?
- Q8c: What are the results of the following commands?
$ /usr/bin/awk -W version < /dev/null; echo "$?"
$ /usr/bin/awk --version 2>/dev/null </dev/null
- Q7d: Could you try this:
core-complete-v5.tar.gz? The test procedure is the same as Q7. (edit: I want the newblesh-debug-mandb-lsof.tar.gz.)
A8c
/usr/bin/awk -W version </dev/null seems to succeed even though it produces an error?
Exactly, kind of weird, its printing this error message, that seems to be more of a warning, as its not exiting with a non-0 exit code.
$ /usr/bin/awk -W version < /dev/null; echo "$?"
/usr/bin/awk: unknown option -W ignored
0
$ /usr/bin/awk --version 2>/dev/null </dev/null
awk version 20200816
A7d
/usr/bin/awk -W version </dev/null seems to succeed even though it produces an error?
Exactly, kind of weird, its printing this error message, that seems to be more of a warning, as its not exiting with a non-0 exit code.
Thanks! I'll add a fix for this. However, this is probably an issue independent of the original error message (though I initially suspected this).
Thank you for this new version. Hmm, the error is not recorded somehow. I have a question.
- Q7d': With the above version of
core-complete.sh(the one in Q7d), do you see the error messagetowc: multibyte conversion failure on: '...'?