Elpy seems partially incompatible with Emacs 25's 'native completion' feature

[cpitclaudel]

In Emacs 25, when starting Python with C-c C-c, I get the following message:

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "ipython3" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list. Native completions have been disabled locally.

On the other hand, if I disable elpy, start the repl, and re-enable it, everything works fine. Is elpy introducing a process filter for the python process, or something similar? That warning comes from this bit of python.el:

(defun python-shell-completion-native-setup ()
  "Try to setup native completion, return non-nil on success."

which deep down relies on explicitly doing an accept-process-output, which sounds rather bad.

In any case, I'm enclined to think that python.el is doing something wrong and elpy doing something right, but I'd love to get a bit of help pinpointing the bad interaction before opening an Emacs ticket, if we determine this to be the right course of action; especially since Emacs 25 is in a pretty deep freeze, so there's little chance that this will get fixed on the Emacs side before the release.

Let me know if I can help :)

[jorgenschaefer]

Hello, and thanks for the report. What's the value of python-shell-interpreter with and without Elpy enabled? Are you using a virtualenv in either case?

[cpitclaudel]

No virtualenv in any case. The bug happens regardless of the value I give to python-shell-interpreter in my .emacs (python3 or ipython3). Enabling or disabling elpy doesn't seem to change that value.

[jorgenschaefer]

Could you copy the definition of the function python-shell-completion-native-setup into your *scratch* buffer and edit it so that the part before pytho.el: native completion setup failed reads like this:

        print ('python.el: native completion setup loaded')                     
    except:                                                                     
        raise
        print ('python.el: native completion setup failed')                     

i.e. add a raise to the except branch so we can see the error, hopefully. Evaluate the resulting function and try C-c C-c in Elpy again.

[cpitclaudel]

Sorry, I don't think I explained this clearly. There is no error; the python repl buffer contains this after the warning is shown:

Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
Type "copyright", "credits" or "license" for more information.

IPython 1.2.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 
In [1]: python.el: native completion setup loaded

The reason the warning is shown seems to be that the accept-process-output call that python.el does times out without returning data; strangely enough, though, it doesn't happen reliably...

[jorgenschaefer]

There's nothing Elpy does here that should interfere with that code. I have no idea what's going on there. :-(

[cpitclaudel]

I'll close this then, and I'll try to investigate it myself next time it pops up :)

[rthoma]

I receive the same warning as cpitclaudel running Emacs 25.1.50 (9.0) built with homebrew on my mac.

[ghost]

I also have been getting this same warning, I ended up just setting python-shell-completion-native to nil, as I just couldn't get it to work.

I do have working tab-completion in python shell buffers with that setting with ipython as my interpreter though.

[cpitclaudel]

Looks like this is happening again on master :/

[cpitclaudel]

Interestingly, even setting elpy-modules to nil doesn't suppress the problem. What does elpy do that's not in a module?

[jorgenschaefer]

Interestingly, even setting elpy-modules to nil doesn't suppress the problem. What does elpy do that's not in a module?

Not much. This is a problem with python-mode in Emacs, not Elpy specifically.

I suspect this is yet another problem with the newer ipython stuff. As a wild shot, could you add (setenv "IPY_TEST_SIMPLE_PROMPT" "1") and/or (setenv "JUPYTER_CONSOLE_TEST" "1") to your config file and see if that fixes this problem?

[cpitclaudel]

Hmm. I'm not using IPython (I use the regular CPython); besides, everything works fine in both emacs -Q and in my regular Emacs, as long as I don't load Elpy.

[jorgenschaefer]

But the output you showed above was from ipython?

[cpitclaudel]

Indeed; that's what I guess I was testing out at the time. This happens with 2.7 and 3.5 as well.

[jorgenschaefer]

Could you copy the definition of the function python-shell-completion-native-setup into your *scratch* buffer and edit it so that the part before pytho.el: native completion setup failed reads like this:

        print ('python.el: native completion setup loaded')                     
    except:                                                                     
        raise
        print ('python.el: native completion setup failed')                     

i.e. add a raise to the except branch so we can see the error, hopefully. Evaluate the resulting function and try C-c C-c in Elpy again.

[cpitclaudel]

@jorgenschaefer I'm confused. Didn't you ask me to do this before? :)

I made the requested change, but there is no error to see:

Python 3.5.2 (default, Jul  5 2016, 12:43:10) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> python.el: native completion setup loaded
>>> 

The except path is never taken.

[jorgenschaefer]

Yes I did, but that was with ipython which might have had different code paths :-) When trying to debug some weird behavior, it's confusing when the environment changes randomly.

What does M-: (python-shell-completion-native-try) return for you?

I'm still confused why anything that Elpy does would affect this. If you add (setq elpy-modules nil) before elpy-enable, does this still happen?

[cpitclaudel]

I'll try this. I'll also try to debug this on my own a bit, because this is getting complex and annoying, and I don't think I'm being very helpful. Let me create a minimal test case, and see if that helps me; otherwise, I'll at least have stuff to report here :)

[ghost]

I too am facing same problem as @cpitclaudel. When I do M-: (python-shell-completion-native-try), I get error

Debugger entered--Lisp error: (wrong-type-argument processp nil)
  process-buffer(nil)
  python-shell-completion-native-get-completions(nil nil "")
  python-shell-completion-native-try()
  eval((python-shell-completion-native-try) nil)
  eval-expression((python-shell-completion-native-try) nil)
  funcall-interactively(eval-expression (python-shell-completion-native-try) nil)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

Even with (setq elpy-modules nil) I still get the "Native completions have been disabled locally" warning and completions do not work. Seems to be due to: bug#22897

[ghost]

I have a question. How can I elpy-shell-send-region-or-buffer to an external Python interpreter process (not running in Emacs), but attached to a terminal emulator?

[cpitclaudel]

Closing this: after a while I ended up opening an Emacs bug report, and was pointed to another, existing one. See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24401

Sorry for the confusion, and thanks for the help and the hard work on Elpy!

[jorgenschaefer]

Thank you for finding that upstream bug! That solves one of my headaches here :-D

[chase-dwelle]

I'm having this issue as well. What's the best way to suppress this error until the fix is pushed out through emacs?

[ChillarAnand]

You can try this

(setq python-shell-prompt-detect-failure-warning nil)

[cpitclaudel]

I pushed a fix to emacs-25, so the problem will go away in the next Emacs release :) In the meantime, you can redefine the affected function in your .emacs, as shown in https://github.com/emacs-mirror/emacs/commit/dbb341022870ecad4c9177485a6770a355633cc0

[jorgenschaefer]

Is it possible to reliably detect the problem? I would not mind having the fix in elpy.el's "backwards compatibility" section …

[cpitclaudel]

I'm not sure about reliably detecting it; but since the issue affects only one version I think it would work to just redefine the affected function in that version.

[chaturv3di]

I just stumbled upon this solution to the problem that I'm facing with Emacs25.1.1 (x86_64-w64-mingw32) on Windows7. I added the function definition in my .emacs.d/init.el file, but I continue to get the warning and the accompanying error in the Python interpreter.

It seems to me that python.el is being loaded after my emacs profile, hence the problem persists. I'm attaching a screenshot for reference. Am I doing something wrong?

[cpitclaudel]

You could wrap this redefinition in (with-eval-after-load 'python …)

[chaturv3di]

Thanks a lot. This worked. I'm attaching the screenshot of the code for whoever might be interested in it.