elpy
elpy copied to clipboard
jorgenschaefer
[emacs27] virtual env is sometime reset to rpc-env
Summary
Using emacs27, the current active virtual environment is lost and is reset to rpc-env environment.
Steps to reproduce
- activate a virtual environment with
pyvenv-workon - open a python buffer
- navigate the buffer
- check if the selected virtualenv is still active, sometimes elpy rpc-env will be active instead
My configuration
Result of (elpy-config)
Elpy Configuration
Emacs.............: 27.0.91
Elpy..............: 1.33.0
Virtualenv........: myenv (/home/gaetano/.virtualenvs/myenv/)
Interactive Python: python 3.7.5 (/home/gaetano/.virtualenvs/myenv/bin/python)
RPC virtualenv....: rpc-venv (/home/gaetano/.emacs.d/elpy/rpc-venv)
Python...........: python 3.8.3 (/home/gaetano/.emacs.d/elpy/rpc-venv/bin/python)
Jedi.............: 0.17.0
Rope.............: 0.17.0
Autopep8.........: 1.5.2
Yapf.............: 0.30.0
Black............: 19.10b0
Syntax checker....: flake8 (/home/gaetano/.virtualenvs/myenvbin/flake8)
Elpy temporarily activates the rpc-virtualenv when starting a RPC process (that is, when opening a new python buffer).
If you check what is the active virtualenv at this time, it would indeed be the rpc one. However, it should go reactivate your virtualenv quickly after that. Is it what is happening for you ?
(or it could be related to emacs27, Elpy does not support it officially).
The active virtualenv change is permanent.
It seems to me that with-elpy-rpc-virtualenv-activated sometimes doesn't revert the virtualenv change.
This happens not only when starting the RPC process, but also when using elpy rpc ,for instance when jumping to symbol definition.
Currently I'm running instead on emacs27 with (setq elpy-rpc-virtualenv-path 'current) and everything seems ok.
I am not sure why it stays activated.
Maybe if you can run this instrumented version of with-elpy-rpc-virtualenv-activated, it may shed some light on what's going wrong...
(defmacro with-elpy-rpc-virtualenv-activated (&rest body)
"Run BODY with Elpy's RPC virtualenv activated.
During the execution of BODY the following variables are available:
- `current-environment': current environment path.
- `current-environment-binaries': current environment python binaries path.
- `current-environment-is-deactivated': non-nil if the current
environment has been deactivated (it is not if the RPC environment and
the current environment are the same)."
`(if (not (executable-find elpy-rpc-python-command))
(error "Cannot find executable '%s', please set 'elpy-rpc-python-command' to an existing executable." elpy-rpc-python-command)
(message "Calling `with-elpy-rpc-virtualenv-activated'")
(let* ((pyvenv-post-activate-hooks (remq 'elpy-rpc--disconnect
pyvenv-post-activate-hooks))
(pyvenv-post-deactivate-hooks (remq 'elpy-rpc--disconnect
pyvenv-post-deactivate-hooks))
(venv-was-activated pyvenv-virtual-env)
(current-environment-binaries (executable-find
elpy-rpc-python-command))
(current-environment (directory-file-name (file-name-directory (directory-file-name (file-name-directory current-environment-binaries)))))
;; No need to change of venv if they are the same
(same-venv (or (string= current-environment
(elpy-rpc-get-virtualenv-path))
(file-equal-p current-environment
(elpy-rpc-get-virtualenv-path))))
current-environment-is-deactivated)
(message "venv-was-activated: %s" venv-was-activated)
(message "same-venv: %s" same-venv)
(message "current-environment: %s" current-environment)
;; If different than the current one, try to activate the RPC virtualenv
(unless same-venv
(condition-case err
(pyvenv-activate (elpy-rpc-get-or-create-virtualenv))
((error quit)
(if venv-was-activated
(pyvenv-activate venv-was-activated)
(pyvenv-deactivate))
(error err)))
(setq current-environment-is-deactivated t)
(message "RPC venv activated"))
(let (venv-err result)
;; Run BODY and catch errors and quit to avoid keeping the RPC
;; virtualenv activated
(condition-case err
(progn
(message "Running BODY")
(setq result (progn ,@body)))
(error (setq venv-err
(if (stringp err)
err
(car (cdr err)))))
(quit nil))
;; Reactivate the previous environment if necessary
(unless same-venv
(if venv-was-activated
(progn
(message "Reactivating old venv: %s" venv-was-activated)
(pyvenv-activate venv-was-activated))
(message "Deactivating the RPC venv")
(pyvenv-deactivate)))
;; Raise errors that could have happened in BODY
(when venv-err
(error venv-err))
result))))
Here's the output when opening a file and then moving around:
venv-was-activated: /home/gaetano/.virtualenvs/media-director/
same-venv: nil
current-environment: /home/gaetano/.virtualenvs/media-director
Mark set
Calling ‘with-elpy-rpc-virtualenv-activated’
venv-was-activated: /home/gaetano/.virtualenvs/media-director/
same-venv: nil
current-environment: /home/gaetano
RPC venv activated
Running BODY
Reactivating old venv: /home/gaetano/.virtualenvs/media-director/
Waiting for process to die...done
Parsing test_track.py (LALR)...
Calling ‘with-elpy-rpc-virtualenv-activated’
venv-was-activated: /home/gaetano/.emacs.d/elpy/rpc-venv/
same-venv: nil
current-environment: /home/gaetano
RPC venv activated
Running BODY
Reactivating old venv: /home/gaetano/.emacs.d/elpy/rpc-venv/
current $VIRTUAL_ENV /home/gaetano/.emacs.d/elpy/rpc-venv/
Parsing test_track.py (LALR)...
Quit
Mark saved where search started
Parsing test_track.py (LALR)...done
Mark set
Mark activated
Line "Waiting processs to die..." seems suspicious.
I don't know which process exactly get killed, but it seems to prevent the RPC virtualenv to get deactivated properly.
Can you still deactivate it with (pyvenv-deactivate) after that ?
If not, can you get a backtrace (after a toggle-debug-on-error) ?
Thank you for help in debugging it.
rpc-env can be successfully deactivated with (pyvenv-deactivate)
When the old venv reactivation fails with "Waiting for process to die...done" I get no backtrace, after enabling toggle-debug-on-error
I have this same issue.
Emacs.............: 28.0.50
Elpy..............: 1.34.0
Virtualenv........: runsomething (/home/at145/.virtualenvs/runsomething/)
Interactive Python: python 3.6.7 (/home/at145/.virtualenvs/runsomething/bin/python)
RPC virtualenv....: rpc-venv (/home/at145/.emacs.d/elpy/rpc-venv)
Python...........: python3.7 3.7.5 (/home/at145/.emacs.d/elpy/rpc-venv/bin/python3.7)
Jedi.............: 0.17.1 (0.17.2 available)
Rope.............: Not found (0.17.0 available)
Autopep8.........: 1.5.3
Yapf.............: 0.30.0
Black............: 19.10b0
Syntax checker....: Not found (flake8)
Sorry it took me a bit of time to get back to you.
It looks like something is failing during (pyvenv-activate YOUR-VENV). which is weird knowing you managed to activate it in the first place... Anyway, there is an instrumented version of pyvenv-activate below. The output will at least tell us where Emacs is waiting.
@Xparx A temporary workaround is to not use Elpy RPC (with (setq elpy-rpc-virtualenv-path 'current).)
(defun pyvenv-activate (directory)
"Activate the virtual environment in DIRECTORY."
(interactive (list (read-directory-name "Activate venv: " nil nil nil
pyvenv-default-virtual-env-name)))
(setq directory (expand-file-name directory))
(message "Deactivating previous venv: %s" pyvenv-virtual-env-name)
(pyvenv-deactivate)
(message "Activating new venv: %s" directory)
(setq pyvenv-virtual-env (file-name-as-directory directory)
pyvenv-virtual-env-name (file-name-nondirectory
(directory-file-name directory))
python-shell-virtualenv-path directory
python-shell-virtualenv-root directory)
;; Set venv name as parent directory for generic directories or for
;; the user's default venv name
(when (or (member pyvenv-virtual-env-name '("venv" ".venv" "env" ".env"))
(and pyvenv-default-virtual-env-name
(string= pyvenv-default-virtual-env-name
pyvenv-virtual-env-name)))
(setq pyvenv-virtual-env-name
(file-name-nondirectory
(directory-file-name
(file-name-directory
(directory-file-name directory))))))
(message "pyvenv-virtual-env-name: %s" pyvenv-virtual-env-name)
;; Preserve variables from being overwritten.
(let ((old-exec-path exec-path)
(old-eshell-path eshell-path-env)
(old-process-environment process-environment))
(message "Running 'pre_activate' hooks")
(unwind-protect
(pyvenv-run-virtualenvwrapper-hook "pre_activate" pyvenv-virtual-env)
(setq exec-path old-exec-path
eshell-path-env old-eshell-path
process-environment old-process-environment)))
(message "Running pyvenv pre activate hooks: %s" pyvenv-pre-activate-hooks)
(run-hooks 'pyvenv-pre-activate-hooks)
(message "Updating pyvenv variables to new venv")
(let ((new-directories (append
;; Unix
(when (file-exists-p (format "%s/bin" directory))
(list (format "%s/bin" directory)))
;; Windows
(when (file-exists-p (format "%s/Scripts" directory))
(list (format "%s/Scripts" directory)
;; Apparently, some virtualenv
;; versions on windows put the
;; python.exe in the virtualenv root
;; for some reason?
directory)))))
(setq pyvenv-old-exec-path exec-path
pyvenv-old-eshell-path eshell-path-env
pyvenv-old-process-environment process-environment
;; For some reason, Emacs adds some directories to `exec-path'
;; but not to `process-environment'?
exec-path (append new-directories exec-path)
;; set eshell path to same as exec-path
eshell-path-env (mapconcat 'identity exec-path ":")
process-environment (append
(list
(format "VIRTUAL_ENV=%s" directory)
(format "PATH=%s"
(mapconcat 'identity
(append new-directories
(split-string (getenv "PATH")
path-separator))
path-separator))
;; No "=" means to unset
"PYTHONHOME")
process-environment)
))
(message "Running 'post_activate' hooks")
(pyvenv-run-virtualenvwrapper-hook "post_activate")
(message "Running pyvenv post activate hooks: %s" pyvenv-post-activate-hooks)
(run-hooks 'pyvenv-post-activate-hooks))
Thanks for the answer. I have set this parameter and it seems to workaround the problem.
Having a similar problem w/ emacs 28 HEAD. rpc-env gets activated, but it fails to restore the venv afterward.
Using the instrumented with-elpy-rpc-virtualenv-activated:
with-elpy-rpc-virtualenv-activated
Deactivating previous venv: rpc-venv
Activating new venv: /Users/tmiller/.local/share/virtualenvs/pynet-analysis
pyvenv-virtual-env-name: pynet-analysis
Running ’pre_activate’ hooks
Running pyvenv pre activate hooks: nil
Updating pyvenv variables to new venv
Running ’post_activate’ hooks
Running pyvenv post activate hooks: (elpy-rpc--disconnect)
C-x C-g is undefined
Note: standard-indent, tab-width adjusted to 8
Deactivating previous venv: pynet-analysis
Activating new venv: /Users/tmiller/.emacs.d/elpy/rpc-venv
pyvenv-virtual-env-name: rpc-venv
Running ’pre_activate’ hooks
Running pyvenv pre activate hooks: nil
Updating pyvenv variables to new venv
Running ’post_activate’ hooks
Running pyvenv post activate hooks: nil
Deactivating previous venv: rpc-venv
Waiting for process to die...done
Using the instrumented pyvenv-activate (alone) above:
pyvenv-activate
Quit
Deactivating previous venv: nil
Activating new venv: /Users/tmiller/.local/share/virtualenvs/pynet-analysis
pyvenv-virtual-env-name: pynet-analysis
Running ’pre_activate’ hooks
Running pyvenv pre activate hooks: nil
Updating pyvenv variables to new venv
Running ’post_activate’ hooks
Running pyvenv post activate hooks: (elpy-rpc--disconnect)
Quit
Deactivating previous venv: pynet-analysis
Activating new venv: /Users/tmiller/.emacs.d/elpy/rpc-venv
pyvenv-virtual-env-name: rpc-venv
Running ’pre_activate’ hooks
Running pyvenv pre activate hooks: nil
Updating pyvenv variables to new venv
Running ’post_activate’ hooks
Waiting for process to die...done
Quit
Note: standard-indent, tab-width adjusted to 8
Config:
Emacs.............: 28.0.50
Elpy..............: 1.34.0
Virtualenv........: rpc-venv (/Users/tmiller/.emacs.d/elpy/rpc-venv)
Interactive Python: python 3.7.3 (/Users/tmiller/.emacs.d/elpy/rpc-venv/bin/python)
RPC virtualenv....: rpc-venv (/Users/tmiller/.emacs.d/elpy/rpc-venv)
Python...........: python3 3.7.3 (/Users/tmiller/.emacs.d/elpy/rpc-venv/bin/python3)
Jedi.............: 0.17.2
Rope.............: 0.17.0
Autopep8.........: 1.5.4
Yapf.............: 0.30.0
Black............: 19.10b0 (20.8b1 available)
Syntax checker....: flake8 (/Users/tmiller/.emacs.d/elpy/rpc-venv/bin/flake8)
As suspected, Emacs get stuck in pyvenv-activate (in pyvenv-run-virtualenvwrapper-hook to be more precise).
Do you have the package virtualenvwrapper installed in your system python ?
Do you have postactivate and preactivate scripts in ~/.emacs/elpy/rpc-venv/bin ?
If so, you can try reinstalling the rpc virtualenv with M-x elpy-rpc-reinstall-virtualenv.
If not, it could be that pyvenv is mistakenly thinking that the rpc-venv has been made with virtualenvwrapper. It then tries to run hooks that does not exist. If is is the case, using the following advice may help:
(defadvice pyvenv-virtualenvwrapper-supported (around not-for-elpy-rpc-venv activate)
"Ensure no virtualenvwrapper support for the RPC virtualenv"
(if (string= pyvenv-virtual-env-name "rpc-venv")
nil
ad-do-it))
If the issue comes from this, it will be a good idea to report the issue upstream to Pyvenv.
Swapped back to elpy after suffering w/ python-mode + eglot for a while.
On a fresh reinstall, seems to be to be tracking OK now even w/o the advice. There's no post/preactivate scripts in (newly created) rpc-venv/bin:
❯ ls -l
total 152
-rw-r--r-- 1 tmiller staff 8471 Dec 11 15:44 Activate.ps1
-rw-r--r-- 1 tmiller staff 2225 Dec 11 15:44 activate
-rw-r--r-- 1 tmiller staff 1277 Dec 11 15:44 activate.csh
-rw-r--r-- 1 tmiller staff 2429 Dec 11 15:44 activate.fish
-rwxr-xr-x 1 tmiller staff 420 Dec 11 15:44 autopep8
-rwxr-xr-x 1 tmiller staff 256 Dec 11 15:44 black
-rwxr-xr-x 1 tmiller staff 251 Dec 11 15:44 black-primer
-rwxr-xr-x 1 tmiller staff 257 Dec 11 15:44 blackd
-rwxr-xr-x 1 tmiller staff 266 Dec 11 15:44 easy_install
-rwxr-xr-x 1 tmiller staff 266 Dec 11 15:44 easy_install-3.8
-rwxr-xr-x 1 tmiller staff 250 Dec 11 15:44 flake8
-rwxr-xr-x 1 tmiller staff 248 Dec 11 15:44 pip
-rwxr-xr-x 1 tmiller staff 248 Dec 11 15:44 pip3
-rwxr-xr-x 1 tmiller staff 248 Dec 11 15:44 pip3.8
-rwxr-xr-x 1 tmiller staff 248 Dec 11 15:44 pycodestyle
-rwxr-xr-x 1 tmiller staff 247 Dec 11 15:44 pyflakes
lrwxr-xr-x 1 tmiller staff 7 Dec 11 15:44 python -> python3
lrwxr-xr-x 1 tmiller staff 51 Dec 11 15:44 python3 -> /Library/Developer/CommandLineTools/usr/bin/python3
-rwxr-xr-x 1 tmiller staff 247 Dec 11 15:44 yapf
However, I'm getting a virtualenvwrapper hook output buffer popping up every time I load a python file (w/ empty post_activate/post_deactivate output). Something's off w/ pyvenv but I'm not doing any special config here.
I sorted out the hook buffer (see https://github.com/jorgenschaefer/pyvenv/issues/102 for a fix).
I experimented a bit more and I think the venv switching is a problem with both auto-virtualenv and auto-virtualenvwrapper packages. I've been using auto-virtualenvwrapper, so I swapped that out for auto-virtualenv and it was also having problems keeping the activated venv straight.
I switched over to setting pyvenv-workon in dir-locals and on emacs 27.1/Linux the problem seems resolved. I'll test on macOS and emacs:HEAD later.
On macOS I also had issues w/ rpc-venv stickiness w/ pyvenv-tracking-mode enabled. Turned that off.
Because of brew, I also had to fix elpy-rpc-python-command as a fully-qualified path. Apparently I'd set this to just python3 and that seemed to cause problems.
I also wiped the rpc-venv and had elpy rebuild it before loading any projects or activating another venv. That seemed to settle things.
Everything seems stable now.
Extended editing sessions were still sticking, so I added the defadvice above and that seems to be working. I'll open an issue under pyvenv.
So I can't say for certain that it's the same issue I'm seeing, but I've managed to figure out, at least in one case, why this would happen. @galaunay would love to hear your thoughts on this. Let me know if I need to clarify anytthing :smile:
My steps to reproduce it are as follows, but not that it _requires a slow elpy-rpc "init" execution, basically longer than the eldoc delay.
M-x revert-bufferor(pyvenv-activate "/path/to/venv")- Move around and observe the eldoc changing a couple of times. Notice that the venv is now
[rpc-venv].
I was able to reliably reproduce this with a small change, adding a (sleep-for 5) inside elpy-rpc-init.
(defun elpy-rpc-init (library-root environment-binaries &optional success error)
"Initialize the backend.
This has to be called as the first method, else Elpy won't be
able to respond to other calls.
+LIBRARY-ROOT is the current project root,
+ENVIRONMENT-BINARIES is the path to the python binaries of the environment to work in."
(message "%S sleeping 5" (format-time-string "%Y-%m-%dT%H:%M:%S"))
(sleep-for 5)
(message "%S awake" (format-time-string "%Y-%m-%dT%H:%M:%S"))
(elpy-rpc "init"
;; This uses a vector because otherwise, json-encode in
;; older Emacsen gets seriously confused, especially when
;; backend is nil.
(vector `((project_root . ,(expand-file-name library-root))
(environment . ,(when environment-binaries
(expand-file-name
environment-binaries)))))
success error))
My observation leading to this theory is that each time (elpy-rpc "get_calltip_or_oneline_docstring" ...) is called, it calls elpy-rpc-init if the RPC buffer hasn't already been created (for example, if it's taking time to do so). When this happens, it does a nested (elpy-rpc "init" ...) call which also saves the current venv, but this time, the current venv is actually rpc-venv instead of my active venv (or no venv, as it were). Then when the init RPC finishes, it "restores" the venv to the RPC venv.
(elpy-rpc "get_calltip_or_oneline_docstring")
(elpy-rpc--call "get_calltip_or_oneline_docstring")
(elpy-rpc--get-rpc-buffer)
(elpy-rpc-open)
(elpy-rpc-init)
(elpy-rpc "init")
And so for the first call to (elpy-rpc "get_calltip_or_oneline_docstring"), venv-was-activated is correct, but when it's called a second time, venv-was-activate is rpc-venv because the first call is still executing the (sleeping) (elpy-rpc-init).
Hopefully that makes sense. I don't have a good idea for a fix just yet, though!
I just reproduced this on emacs 27.2 after installing only elpy and pyvenv from melpa.
Elpy Configuration
Emacs.............: 27.2
Elpy..............: 1.35.0
Virtualenv........: rpc-venv (/home/wade/.emacs.d/elpy/rpc-venv)
Interactive Python: python 3.8.10 (/home/wade/.emacs.d/elpy/rpc-venv/bin/python)
RPC virtualenv....: rpc-venv (/home/wade/.emacs.d/elpy/rpc-venv)
Python...........: python 3.8.10 (/home/wade/.emacs.d/elpy/rpc-venv/bin/python)
Jedi.............: 0.18.0
Autopep8.........: 1.6.0
Yapf.............: 0.31.0
Black............: 21.10b0
Syntax checker....: flake8 (/home/wade/.emacs.d/elpy/rpc-venv/bin/flake8)
Even adding a small delay like (sleep-for 0 100) ;; 100 ms in elpy-rpc-init can reproduce it with (setq eldoc-idle-delay 0) and some quick C-n C-n to get a couple calls in there, i.e.
(setq eldoc-idle-delay 0)(pyvenv-activate "/path/to/venv")C-n C-n
@galaunay just wondering if you had a chance to look at my analysis of the problem in this earlier post? I would submit a PR if I had some idea how to fix it, but alas, hoping for an expert to weigh in here.