argcomplete
argcomplete copied to clipboard
"Unable to open fd 8 for writing, quitting" on Linux
Relevant older issue: https://github.com/kislyuk/argcomplete/issues/142 where I believe users also reported it happening on Linux but it was closed without any resolution (e.g. via suggested use of temporary files).
In my case I have a stock Debian (testing/unstable mix), python 3.8.3rc1 and I observe the same error with our DataLad which recently was fixed up to work (again) with argcomplete and it works for everyone else on the team but (un)lucky me. In the session you below can see that I am getting the exception, and the process has no 8 fd among its fds:
$> _ARC_DEBUG=1 _ARGCOMPLETE=1 datalad --pdb -l 1 --help
[DEBUG ] Command line args 1st pass. Parsed: Namespace() Unparsed: ['--pdb', '--help']
[DEBUG ] Discovering plugins
[DEBUG ] Loading entrypoints
[DEBUG ] Generating detailed description for the parser
[Level 5] Finished setup_parser
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(188)__call__()
-> try:
(Pdb) l
183 except:
184 debug_stream = sys.stderr
185
186 if output_stream is None:
187 import pdb; pdb.set_trace()
188 -> try:
189 output_stream = os.fdopen(8, "wb")
190 except:
191 debug("Unable to open fd 8 for writing, quitting")
192 exit_method(1)
193
(Pdb) os.getpid()
1092474
(Pdb)
[3] + 1092474 suspended _ARC_DEBUG=1 _ARGCOMPLETE=1 datalad --pdb -l 1 --help
$> ls -l /proc/1092474/fd
total 0
lrwx------ 1 yoh yoh 64 May 19 13:48 0 -> /dev/pts/16
lrwx------ 1 yoh yoh 64 May 19 13:48 1 -> /dev/pts/16
lrwx------ 1 yoh yoh 64 May 19 13:48 2 -> /dev/pts/16
lr-x------ 1 yoh yoh 64 May 19 13:48 3 -> pipe:[41232747]
lr-x------ 1 yoh yoh 64 May 19 13:48 5 -> pipe:[41232748]
$> fg
[3] 1092474 continued _ARC_DEBUG=1 _ARGCOMPLETE=1 datalad --pdb -l 1 --help
l
l
194 # print("", stream=debug_stream)
195 # for v in "COMP_CWORD COMP_LINE COMP_POINT COMP_TYPE COMP_KEY _ARGCOMPLETE_COMP_WORDBREAKS COMP_WORDS".split():
196 # print(v, os.environ[v], stream=debug_stream)
197
198 ifs = os.environ.get("_ARGCOMPLETE_IFS", "\013")
199 if len(ifs) != 1:
200 debug("Invalid value for IFS, quitting [{v}]".format(v=ifs))
201 exit_method(1)
202
203 comp_line = os.environ["COMP_LINE"]
204 comp_point = int(os.environ["COMP_POINT"])
(Pdb) n
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(189)__call__()
-> output_stream = os.fdopen(8, "wb")
(Pdb)
OSError: [Errno 9] Bad file descriptor
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(189)__call__()
-> output_stream = os.fdopen(8, "wb")
(Pdb)
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(190)__call__()
-> except:
(Pdb) fg
*** NameError: name 'fg' is not defined
(Pdb)
Traceback (most recent call last):
File "/home/yoh/proj/misc/argcomplete/argcomplete/__init__.py", line 189, in __call__
output_stream = os.fdopen(8, "wb")
File "/usr/lib/python3.8/os.py", line 1023, in fdopen
return io.open(fd, *args, **kwargs)
OSError: [Errno 9] Bad file descriptor
in the shell I do get calls which do not explicitly create fd 8 failing:
$> python -c 'import os; os.fdopen(8)'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.8/os.py", line 1023, in fdopen
return io.open(fd, *args, **kwargs)
OSError: [Errno 9] Bad file descriptor
$> python -c 'import os; os.fdopen(8)' 8>&1
$>
This is expected; you cannot open a file descriptor if it doesn't exist. The shell wrappers explicitly create and read from file descriptor 8. We do this because argcomplete lives within arbitrary Python programs, so rather than trying to suppress all output to stdout, we simply redirect the stream to /dev/null and communicate on a different one.
Your actual problem likely has nothing to do with this. Can you set _ARC_DEBUG=1
and paste the output from an actual tab completion attempt? Please also provide your shell version and what you did to install and activate argcomplete.
The shell wrappers explicitly create and read from file descriptor 8
Should it then be open when I source the shell wrapper? May be the problem is in it on my end.
Here is the wrapper I use - may be it is the source of trouble:
$> cat tools/cmdline-completion
# Universal completion script for DataLad with the core autogenerated by
# python-argcomplete and only slighly umproved to work for ZSH if sourced under
# ZSH
if [ "${ZSH_VERSION:-}" != "" ]; then
echo "I: Enabling support of bash completions for zsh"
autoload -U compinit && compinit
autoload -U bashcompinit && bashcompinit
fi
_python_argcomplete() {
local IFS='
'
COMPREPLY=( $(IFS="$IFS" COMP_LINE="$COMP_LINE" COMP_POINT="$COMP_POINT" _ARGCOMPLETE_COMP_WORDBREAKS="$COMP_WORDBREAKS" _ARGCOMPLETE=1 "$1" 8>&1 9>&2 1>/dev/null 2>/dev/null) )
if [[ $? != 0 ]]; then
unset COMPREPLY
fi
}
complete -o nospace -o default -F _python_argcomplete "datalad"
But even after sourcing it I do not see fd 8 open
lena:~/proj/datalad/datalad-maint
$> source tools/cmdline-completion
$> ls -l /proc/$$/fd/
total 0
0 lrwx------ 1 yoh yoh 64 May 21 09:49 0 -> /dev/pts/25
0 lrwx------ 1 yoh yoh 64 May 21 09:49 1 -> /dev/pts/25
0 lrwx------ 1 yoh yoh 64 May 21 09:49 2 -> /dev/pts/25
0 lrwx------ 1 yoh yoh 64 May 21 09:49 255 -> /dev/pts/25
Should fd 8 be open at that point already or that would happen later? (upon attempt to complete?)
may be there is some shell setting which is in a way...?(btw -- tried with both bash and zsh)
Can you set _ARC_DEBUG=1 and paste the output from an actual tab completion attempt?
Note that in the initial report message I already have that env var set. And that is how I actually arrived to this specific issue -- by use of this env var... But here is plain attempt to complete which is not informative really:
(git)lena:~datalad/datalad-maint[maint]
$> source venv*/dev3/bin/activate
$> export _ARC_DEBUG=1
$> source tools/cmdline-completion
I: Enabling support of bash completions for zsh
$> datalad ins<TAB>
no matches for: `file' or `corrections'
so nothing is shown.
I have tried with a freshly regenerated wrapper which targets specific full path datalad entry point script where I explicitly added that marker on top -- the same silence
$> source /tmp/datalad-argcomplete
$> cat /tmp/datalad-argcomplete
_python_argcomplete() {
local IFS=$'\013'
local SUPPRESS_SPACE=0
if compopt +o nospace 2> /dev/null; then
SUPPRESS_SPACE=1
fi
COMPREPLY=( $(IFS="$IFS" \
COMP_LINE="$COMP_LINE" \
COMP_POINT="$COMP_POINT" \
COMP_TYPE="$COMP_TYPE" \
_ARGCOMPLETE_COMP_WORDBREAKS="$COMP_WORDBREAKS" \
_ARGCOMPLETE=1 \
_ARGCOMPLETE_SUPPRESS_SPACE=$SUPPRESS_SPACE \
"$1" 8>&1 9>&2 1>/dev/null 2>/dev/null) )
if [[ $? != 0 ]]; then
unset COMPREPLY
elif [[ $SUPPRESS_SPACE == 1 ]] && [[ "$COMPREPLY" =~ [=/:]$ ]]; then
compopt -o nospace
fi
}
complete -o nospace -o default -F _python_argcomplete "/home/yoh/proj/datalad/datalad-maint/venvs/dev3/bin/datalad"
$> cat /home/yoh/proj/datalad/datalad-maint/venvs/dev3/bin/datalad
#!/home/yoh/proj/datalad/datalad-maint/venvs/dev3/bin/python
# PYTHON_ARGCOMPLETE_OK
#
# Custom simplistic runner for DataLad. Assumes datalad module
# being available. Generated by monkey patching monkey patched
# setuptools.
#
from datalad.cmdline.main import main
main()
$> /home/yoh/proj/datalad/datalad-maint/venvs/dev3/bin/datalad ins<TAB>
It appears that the library/executable you are using is complex and performs a lot of initialization before calling argcomplete.autocomplete()
. Some of these initialization steps may be interfering with the argcomplete protocol by using fd 8 or otherwise. The solution to your issue is to disable these initialization steps that your executable is running between the entry point and calling argcomplete, until argcomplete starts working.
That looks like an old wrapper. What version of argcomplete are you running? How did you generate it?
You can try manually removing 1>/dev/null 2>/dev/null
from the line where it runs your program to see if it outputs anything when you attempt to tab complete it.
Have you been able to get any other simple test programs working with argcomplete?
THANK YOU @evanunderscore for the guidance -- I am ready to declare "success" in troubleshooting !
What version of argcomplete are you running?
since it is argcomplete which generates it, may be it should embed argcomplete version which generated it somewhere in the comment
TL;DR summary: it is interaction between
- argcomplete setting
IFS=$'\013'
which is later 'exported' into the underlying process:
_python_argcomplete() {
local IFS=$'\013'
local SUPPRESS_SPACE=0
if compopt +o nospace 2> /dev/null; then
SUPPRESS_SPACE=1
fi
COMPREPLY=( $(IFS="$IFS" \
COMP_LINE="$COMP_LINE" \
COMP_POINT="$COMP_POINT" \
COMP_TYPE="$COMP_TYPE" \
_ARGCOMPLETE_COMP_WORDBREAKS="$COMP_WORDBREAKS" \
_ARGCOMPLETE=1 \
_ARGCOMPLETE_SUPPRESS_SPACE=$SUPPRESS_SPACE \
__python_argcomplete_run "$1") )
-
git-annex
standalone (filed fresh bug report) to @joeyh build shim showing intolerance to thatIFS
causing it to segfault.
$> IFS=$'\013' /usr/lib/git-annex.linux/git version
[1] 1039928 segmentation fault (core dumped) IFS=$'\013' /usr/lib/git-annex.linux/git version
That also explains why other @datalad/developers didn't experience this -- since they typically do not use standalone git-annex build.
I guess there is nothing wrong done on argcomplete side, so feel free to close this issue, unless you see that you could avoid setting IFS and set some other "argcomplete specific" variable so it might not cause disturbance in some weak tools underneath?
Some additional notes which are no longer pertinent probably
How did you generate it?
re-doing everything now after pip install --upgrade argcomplete
to 1.12.0:
$> venvs/dev3/bin/register-python-argcomplete --shell bash `which datalad` >| /tmp/datalad-argcomplete
$> bash # since I was in zsh
$> which datalad
$> source /tmp/datalad-argcomplete
and verified that it still doesn't work by trying to complete datalad inst<TAB>
You can try manually removing 1>/dev/null 2>/dev/null from the line ...
*$> sed -e 's, 1>/dev/null 2>&1,,g' /tmp/datalad-argcomplete > /tmp/datalad-argcomplete-noswallow
$> diff /tmp/datalad-argcomplete /tmp/datalad-argcomplete-noswallow
20c20
< "$@" 8>&1 9>&2 1>/dev/null 2>&1
---
> "$@" 8>&1 9>&2
Thanks for the detailed notes @yarikoptic. Glad you managed to find the source of your problem!
Embedding the argcomplete version in the generated wrapper is a good idea, though currently argcomplete doesn't have a __version__
and keeping one in sync with the package version number is a workflow problem that would need to be left to @kislyuk.