Completions (mainly company-mode) in Inferior Python shell causes bug while reading stdin (mainly with input)

[aksharvarma]

Summary

I have been having issues with input() throwing NameError inside the inferior python shell. After some time looking around, I found that having company-mode running in the Inferior Python shell causes both input() and sys.stdin.readline() (and possibly others) to misinterpret user input and try to eval it instead of returning it as a string. This causes a NameError to be thrown which is definitely not the expected behavior (as tested in a terminal).

Disabling company-mode locally via the inferior-python-mode-hook avoids most scenarios in which the bug is triggered. However, even native completion causes a weird unexpected behavior if a specific input is typed in.

Steps to reproduce

1. Run Emacs using emacs -Q --eval '(load "reproduce-elpy-bug.el")' with the minimal reproduce-elpy-bug.el configuration given below.
2. Open the reproduce-elpy-bug.py python script below.
3. Send buffer to inferior python shell using C-c C-c.
4. Type something (not too fast) when prompted for input.
5. It should throw an error.
6. Inspect value of variable to see more weirdness. This value (copied in the python script below) suggests that elpy/inferior-python shell uses some temporary files to take in the input but it doesn't do it correctly resulting in the unexpected behavior.
7. To see that it is primarily company-mode that causes this, try one of the fixes mentioned in the emacs configuration code. Redo the above steps and see that there is no error.

Additional comments

• Expected behavior is what happens when running ipython -i --simple-prompt reproduce-elpy-bug.py. Note that both ipython and jupyter interpreters cause this error but the basic python interpreter doesn't as mentioned in the emacs configurations.
• Every bug can be replicated by directly typing things into the shell, it isn't necessary to send a script.
• Typing fast enough to avoid triggering company will avoid the issue. Can be verified by increasing company-idle-delay to a very large value (10 seconds, for example).
• Typing only a single character also doesn't trigger the error. Probably because company-minimum-prefix-length is generally more than 1. Should be verifiable by increasing this variable's value and ensuring that even larger length inputs do not cause the bug.

Disabling company-mode still leaves completion related bug

1. Do not enable company. Comment out (require 'company) and (global-company-mode) from emacs config.
2. Use same script.
3. As the input, typing part of an existing variable ("asd", for example) then press TAB.
4. This also causes a weird bug and not the expected behavior.
5. Thus this version of the bug isn't handled by the two potential fixes mentioned in the emacs config that address the company-mode issue.

My Configuration

Note: I had someone else try it out on their machine, also an Arch Linux running similar versions of Python/IPython/Jupyter and so on. I also updated recently without this bug going away. So this bug probably persists in the recent few versions.

OS

Arch Linux (5.13.5-arch1-1 x86_64)

(elpy-config)

From bare bones reproduce-elpy-bug.el config.

Elpy Configuration

Emacs.............: 27.2
Elpy..............: 1.35.0
Virtualenv........: None
Interactive Python: ipython 7.25.0 (/usr/bin/ipython)
RPC virtualenv....: rpc-venv (/home/akshar/.emacs.d/elpy/rpc-venv)
 Python...........: python3 3.9.6 (/home/akshar/.emacs.d/elpy/rpc-venv/bin/python3)
 Jedi.............: Not found (0.18.0 available)
 Autopep8.........: Not found (1.5.7 available)
 Yapf.............: Not found (0.31.0 available)
 Black............: Not found (21.7b0 available)
Syntax checker....: Not found (flake8)

Warnings

You have not activated a virtual env. It is not mandatory but often a
good idea to work inside a virtual env. You can use `M-x
pyvenv-activate` or `M-x pyvenv-workon` to activate one.

The directory ~/.local/bin/ is not in your PATH. As there is no active
virtualenv, installing Python packages locally will place executables
in that directory, so Emacs won't find them. If you are missing some
commands, do add this directory to your PATH -- and then do
`elpy-rpc-restart'.

The Jedi package is not currently installed. This package is needed
for code completion, code navigation and access to documentation.

[Install jedi]

No autoformatting package is currently installed. At least one is
needed (Autopep8, Yapf or Black) to perform autoformatting (`C-c C-r
f` in a python buffer).

[Install autopep8]
[Install yapf]
[Install black]

The configured syntax checker (flake8) could not be found. Elpy uses
this program to provide syntax checks of your code. You can either
install it, or select another one using `elpy-syntax-check-command`.

[Install flake8]

Note that setting up jedi doesn't solve the bug, because my normal configuration has jedi and it still causes the bug.

Emacs configuration to reproduce this bug

This is the reproduce-elpy-bug.el file.

(require 'package)
(package-initialize)
(require 'company)
(global-company-mode)			;possible culprit
(require 'elpy)
(elpy-enable)
;; python3, so input() should return the string, not try to eval.
(setq elpy-rpc-python-command "python3")
;; The following doesn't cause any error.
;; (setq python-shell-interpreter "python"
;;       python-shell-interpreter-args "-i")
;; The following does
(setq python-shell-interpreter "ipython"
      python-shell-interpreter-args "-i --simple-prompt")
;; As does the following
;; (setq python-shell-interpreter "jupyter"
;;       python-shell-interpreter-args "console --simple-prompt"
;;       python-shell-prompt-detect-failure-warning nil)
;; (add-to-list 'python-shell-completion-native-disabled-interpreters
;;              "jupyter")

Possible partial fix

This is the fixes part of the reproduce-elpy-bug.el file. Note that neither of these fixes help with the non company-mode bug with native completion. Also, this seems more like a band-aid that makes input() usable without addressing the underlying issue.

;; This fixes the issue. So the bug is based on company interacting with
;; the inferior python shell that elpy makes.
;; (add-hook 'inferior-python-mode-hook (lambda () (company-mode -1)))

;; Another fix is to set company-idle-delay to a predicate function
;; this will disables company idle behavior in inferior-python-mode.
;; (defun ajv/company-idle-delay ()
;;   (if (eq major-mode
;; 	  'inferior-python-mode)
;;       nil 0.1))
;; (setq company-idle-delay 'ajv/company-idle-delay)

Sample python script to use

The reproduce-elpy-bug.py python script that can be used to reproduce the bug. It includes many variations in the comments and more details.

# Something for company-mode to have to auto-complete
# Try typing in `asd` then pressing tab.
# This erroneous behavior will persist even after "disabling" company.
asdfjkl = "foo"

# Variant 1 of code that will trigger the error
a = input("Enter any multi-character input (not too fast, maybe even with a pause):")
print(a)

# Variant 2
# This shows that there is a bug in the way elpy/inferior python shell
# is interpreting input/output.
# print("Enter any multi-character input (not too fast):")
# a = input()
# print(a)

# Variant 3
# Not merely a problem of `input()`. It is a general IO problem.
# import sys
# print("salt")
# a = sys.stdin.readline()

# If you type slowly, you might notice a lag.
# Regardless, on pressing enter you should get a
# NameError: name 'whatever you typed' is not defined
#
# If you then inspect the value of the variable a,
# you will get something like this (tmp file name changes, of course):
# "import codecs, os;__pyfile = codecs.open('''/tmp/pyNgGSdy''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/tmp/pyNgGSdy''');exec(compile(__code, '''/tmp/pyNgGSdy''', 'exec'));"