No clear way to use `lsp-mode` with Python backend and `pyvenv`; users are left to stumble along on their own, despite the popularity of the workflow

[posita]

Thank you for the bug report

• [X] I am using the latest version of lsp-mode related packages.
• [X] I checked FAQ and Troubleshooting sections
• [ ] You may also try reproduce the issue using clean environment using the following command: M-x lsp-start-plain

Bug description

Python developers properly lean on virtual environments for development, but those are difficult to integrate with lsp-mode with no clear documentation or recommendation of best practices.

When using pyvenv, the key realization (often buried in the materials) is that pyvenv-workon must be executed before lsp. This is difficult to achieve because the argument to pyvenv-workon is often dynamic such as programmatically determining a project root/virtual environment mapping, performing setup inside .dir-locals.el, etc.

Without this ordering, lsp may start with virtualenv A, but when opening a buffer that should use virtualenv B, an lsp server is started using virtualenv B before switching to virtualenv B. This happens silently and is difficult to diagnose.

#393 and #458 describe confusion around integrating pyvenv with lsp-mode, but the presented work-arounds are out-of-date. This post is fairly user-specific. This post suggests it can be done, but doesn't go into details.

What is needed is a clear, coherent example on how to use pyvenv with lsp without running into latent problems.

Steps to reproduce

See above.

Expected behavior

See above.

Which Language Server did you use?

python-lsp-server

OS

MacOS

Error callstack

No response

Anything else?

No response

[posita]

For example, is creating a .dir-locals.el such as the following a good way? Is it the only way? This works for me, but it was hard to come to

;; .dir-locals.el
(
 (python-mode
  (eval pyvenv-workon "myproj")
  (eval lsp)
  (eval add-hook 'before-save-hook 'lsp-format-buffer nil t))
 )

What about the save hook? Where should that live?

If you have that entry, and you have, e.g., a repo that acts as an exception, you will likely have to explicitly remove that hook, which is counter-intuitive:

;; .dir-locals.el
(
 (python-mode
  (eval pyvenv-workon "someotherproj")
  (eval lsp)
  (eval remove-hook 'before-save-hook 'lsp-format-buffer nil t))
 )

When switching back-and-forth between files in each project, this can be error prone and get pretty confusing.

What about lsp-deferred (see #3360)? Does that help in this situation?

[JulienPalard]

I'm experimenting with direnv and emacs-direnv to achieve this.

But it often needs a lsp-workspace-restart to pick up the right path, far from ideal.

[CsBigDataHub]

I'm experimenting with direnv and emacs-direnv to achieve this.

But it often needs a lsp-workspace-restart to pick up the right path, far from ideal.

same with pyvenv and venv

[spenczar]

A slight improvement (maybe?) is to use pyvenv's post-activate hooks. I use pyvenv-activate, but should work just the same with pyvenv-workon:

((python-mode . ((pyvenv-activate . "~/path/to/some/venv")
		 (pyvenv-post-activate-hooks . (lsp)))))

[code-SL]

I was able to get this to work, but it depends on your lsp backend for python. Instructions below are for vanilla emacs using pyright python backend.

When opening a python file, if emacs boots up your lsp backend (in my case that is pyright), emacs boots it up with the system python3 interpreter rather than the venv one. So in emacs one would need to 1) switch to the venv via pyvenv, 2) tell your lsp backend where your venv is by setting the variable and 3) then reboot the lsp backend so that it then uses the venv in the variable you set. If you are starting emacs after having already activated the venv in your shell (via source name_of_your_env/bin/activate then running emacs) then when I open up a python file and emacs spins up the lsp pyright, it selects the venv interpreter instead because it determines which interpreter to use by determining the location from running python. It may help to read your lsp backend's documentation on how it figures out what interpreter to use.

What worked for me is to set up hooks after activating a pyvenv environment (M-x pyvenv-activate) and then separate hooks by deactivating (M-x pyvenv-deactivate) (modifying F.Meyer's instructions here). This is for a vanilla emacs setup, again, using pyright:

; Hooks for pyvenv when you activate a venv. You can do this via :hook as well, presumably.
(use-package pyvenv
  :config
  (pyvenv-mode t)
  (setq pyvenv-post-activate-hooks
        (list (lambda ()
		(setq lsp-pyright-venv-path pyvenv-virtual-env)
		(lsp-restart-workspace)
		))
	)

  (setq pyvenv-post-deactivate-hooks
        (list (lambda ()
		(setq lsp-pyright-venv-path nil)
		(lsp-restart-workspace)
		)))
  )

For reference, the configs for the other relevant packages here:

; lsp mode
(use-package lsp-mode
  :init
  (setq lsp-keymap-prefix "C-c l")
  :hook (
	 (python-mode . lsp)
	 )
  :commands lsp
  )

;; This is the backend
(use-package lsp-pyright
)