Using with virtualenv created with `--system-site-packages`, system libraries are unresolved

[WJCFerguson]

Working with a virtualenv (Py2.7 in this case) build --system-site-packages and activated via direnv, system libraries such as os, are not found by lsp and are marked as unresolved-import. Interactive python can import them fine, and VS Code also seems happy, shows documentation etc..

To reproduce: (for reference I put a testing emacs minimal lsp init .el I used below)

• Create virtualenv with system-site-packages, e.g.:

(mkdir -p ~/.virtualenvs && cd ~/virtualenvs && virtualenv --python=python2 --system-site-packages py2env`)
source ~/virtualenvs/py2env/bin/activate
cat > foo.py <<EOF
import os

os.path.join('foo')
EOF
emacs -q -l minimal-lsp-init.el foo.py

• observe unresolved-import 'os' messages, though sometimes they disappear until I make an edit to the file

Emacs version: 27.1 lsp-python-ms version 0.7.1

I tried to dig in a bit but didn't get far. I can tell you that lsp-python-ms--get-python-ver-and-syspath returned paths including /usr/lib/python2.7 etc. where os lives, so that seemed in order to me.

---

For testing to isolate from my emacs config, I made this minimal use-package init.el and ran emacs -q -l init.el foo.py:

;; ============================ bootstrap packaging ============================
(require 'package)
(add-to-list 'package-archives
             '("MELPA" . "https://melpa.org/packages/"))
(package-initialize)

;; Bootstrap `use-package'
(unless (package-installed-p 'use-package)
  (package-refresh-contents)
  (package-install 'use-package))
(eval-when-compile
  (require 'use-package))
(setq use-package-always-ensure 't)

;; =============================================================================
(use-package lsp-mode
  :ensure t
  :diminish lsp-mode
  :commands lsp
  :init

  ;; https://emacs-lsp.github.io/lsp-mode/page/performance/ suggestions:
  (setq gc-cons-threshold 100000000
        read-process-output-max (* 1024 1024))

  (use-package lsp-python-ms
    :ensure t
    :init (setq lsp-python-ms-auto-install-server t)))

Thank you for this package, and please let me know what investigation I can do or how I can help.

[WJCFerguson]

~~I've discovered that this seems to only happen if I activate the virtualenv via direnv (whether managed via direnv.el or not).~~

~~As a result, this is at most an esoteric bug, and probably a bug in direnv, or in my config of direnv. As such I'll close it.~~

Edit: I was wrong. See next comment

[WJCFerguson]

Reopening as there's a real issue here. Though it may well be in the MS LSP.

I've poked around this for a while. Thinking my environment was at fault I abandoned direnv for pyvenv.el and got something that sometimes worked, though lsp-restart-workspace often failed. Further debugging showed me my environment was correct when the LSP server is launched, and something else is at play.

I found errors in *mspyls::stderr*:

Traceback (most recent call last):
  File "/home/james.ferguson/.emacs.d/.cache/lsp/mspyls/get_search_paths.py", line 31, in <module>
    import site
  File "/home/james.ferguson/.virtualenvs/py2env/lib/python2.7/site.py", line 166, in <module>
    main()
  File "/home/james.ferguson/.virtualenvs/py2env/lib/python2.7/site.py", line 19, in main
    load_host_site()
  File "/home/james.ferguson/.virtualenvs/py2env/lib/python2.7/site.py", line 54, in load_host_site
    full_path = os.path.abspath(os.path.join(here, path.encode("utf-8")))
LookupError: no codec search functions registered: can't find encoding

And yet running python interactively etc., this code executes fine. Through some debug print statements I found this function called multiple times, sometimes successfully, sometimes not. When unsuccessful, the Python interpreter was weirdly stripped-down - no sys.argv for instance, and codec handling fails. I haven't yet worked out what's calling it or how. I'll continue to investigate when I have time.

Simply manually stripping out the .encode("utf-8") from that file is a functioning, if ugly, workaround.

[seagle0128]

@WJCFerguson so that means it's an issue of mspyls?

[WJCFerguson]

That does have to be a strong assumption but it's hard for me to tell until/unless I can debug more deeply. Though the only reference to that error with mspyls is one page in Chinese may well be discussing the same thing w.r.t. lsp-python-ms: https://emacs-china.org/t/lsp-python-ms-python2/12300.