Goto definition stop working after diving into python internals with virtualenvwrapper

[humrochagf]

Hello,

I use this extension in a setup where I have pyenv to control python versions and on top of it virtualvenvwrapper to keep my venvs organized in a common folder and it works much like the anaconda scenario but with a WORKON_HOME envvar to tell were the virtualenvs are (~/.virtualenvs in my case), and a VIRTUAL_ENV envvar that once activated tells the location of the activated venv.

I noticed that sometimes while diving into python internals definitions the goto definition stops working and keeps showing the following message:

[coc.nvim] definition not found

The only way I can get them back working is by closing vim or nvim and opening it again. It was hard to find a pattern but today I found a consistent way of reproducing it, you just need to create a folder to your code then a setup.py file with the following content:

from setuptools import setup

setup()

The next steps to reproduce it are:

• Open the file
• Jump into setup() definition
• Jump into _install_setup_requires(attrs) that is in the same file to see everything works
• Then jump into setup from the line bellow _install_setup_requires that has this content return distutils.core.setup(**attrs)
• At this point you will notice that you leave virtualenv space to reach pyenv distutils core code
• Navigate back to your setup.py code without closing the editor
• Jump into setup() definition
• Jump into _install_setup_requires(attrs) and this time it's not working anymore

Checking the outputs I got this:

What's the output of :CocCommand pyright.version

[coc.nvim] coc-pyright 1.1.243 with Pyright 1.1.244

What's the output of :CocCommand workspace.showOutput Pyright

Workspace: /home/humrochagf/Projects/example
Using python from /home/humrochagf/.virtualenvs/test/bin/python

[Info  - 5:16:10 PM] Pyright language server 1.1.244 starting
[Info  - 5:16:10 PM] Server root directory: /home/humrochagf/.config/coc/extensions/node_modules/coc-pyright/node_modules/pyright/dist/
[Info  - 5:16:10 PM] No configuration file found.
[Info  - 5:16:10 PM] No pyproject.toml file found.
[Info  - 5:16:10 PM] Setting pythonPath for service "example": "/home/humrochagf/.virtualenvs/test/bin/python"
[Warn  - 5:16:10 PM] stubPath /home/humrochagf/Projects/example/typings is not a valid directory.
[Info  - 5:16:10 PM] Assuming Python version 3.10
[Info  - 5:16:10 PM] Assuming Python platform Linux
[Info  - 5:16:10 PM] Searching for source files
[Info  - 5:16:10 PM] Found 1 source file
[Info  - 5:16:15 PM] No configuration file found.
[Info  - 5:16:15 PM] No pyproject.toml file found.
[Info  - 5:16:15 PM] Setting pythonPath for service ".pyenv": "/home/humrochagf/.virtualenvs/test/bin/python"
[Warn  - 5:16:15 PM] stubPath /home/humrochagf/.pyenv/typings is not a valid directory.
[Info  - 5:16:15 PM] Assuming Python version 3.10
[Info  - 5:16:15 PM] Assuming Python platform Linux
[Info  - 5:16:15 PM] Searching for source files
[Info  - 5:16:16 PM] Found 7370 source files
[Info  - 5:17:29 PM] No pyproject.toml file found.
[Info  - 5:17:29 PM] Setting pythonPath for service "<default>": "/home/humrochagf/.virtualenvs/test/bin/python"
[Warn  - 5:17:29 PM] stubPath typings is not a valid directory.
[Info  - 5:17:29 PM] Assuming Python version 3.10
[Info  - 5:17:29 PM] Assuming Python platform Linux
[Info  - 5:17:29 PM] Searching for source files
[Info  - 5:17:29 PM] No source files found.

I Tried to compare with the same setup of having pyenv but instead of using virtualenvwrapper creating a venv inside the code folder and in that case, everything works properly.

And checking the logs I noticed that I don't get the Setting pythonPath for service "<default>" log chunk.

This is my venv test for comparison:

Workspace: /home/humrochagf/Projects/example
Using python from /home/humrochagf/Projects/example/.venv/bin/python

[Info  - 5:45:21 PM] Pyright language server 1.1.244 starting
[Info  - 5:45:21 PM] Server root directory: /home/humrochagf/.config/coc/extensions/node_modules/coc-pyright/node_modules/pyright/dist/
[Info  - 5:45:21 PM] No configuration file found.
[Info  - 5:45:21 PM] No pyproject.toml file found.
[Info  - 5:45:21 PM] Setting pythonPath for service "example": "/home/humrochagf/Projects/example/.venv/bin/python"
[Warn  - 5:45:21 PM] stubPath /home/humrochagf/Projects/example/typings is not a valid directory.
[Info  - 5:45:21 PM] Assuming Python version 3.10
[Info  - 5:45:21 PM] Assuming Python platform Linux
[Info  - 5:45:21 PM] Searching for source files
[Info  - 5:45:21 PM] Found 1 source file
[Info  - 5:45:36 PM] No configuration file found.
[Info  - 5:45:36 PM] No pyproject.toml file found.
[Info  - 5:45:36 PM] Setting pythonPath for service ".pyenv": "/home/humrochagf/Projects/example/.venv/bin/python"
[Warn  - 5:45:36 PM] stubPath /home/humrochagf/.pyenv/typings is not a valid directory.
[Info  - 5:45:36 PM] Assuming Python version 3.10
[Info  - 5:45:36 PM] Assuming Python platform Linux
[Info  - 5:45:36 PM] Searching for source files
[Info  - 5:45:37 PM] Found 7370 source files

Another info that may be pertinent is that the Setting pythonPath for service "<default>" log chunk happens at the second time I jump from my code into the setup() call.

Thanks for taking the time of reading my issue and also thanks a lot for this awesome extension :)

[fannheyward]

First of all, very clear issue steps 👍 .

I'm using project-level venv in my projects, as your 2nd test, didn't encounter the problem.

I'd like to debug the issue with pyenv/virtualvenvwrapper, can you clear me how do you setup pyenv to keep all venvs in a common folder? Sorry I didn't use pyenv ever.

[humrochagf]

Hey, sure to use pyenv with virtualenvwrapper plugin you first need to clone the pyenv repo into your local home:

 git clone https://github.com/pyenv/pyenv.git ~/.pyenv

I use this installation option since I'm in Linux but if you have a different setup you can check their installation docs page

Next step is to install pyenv-virtualenvwrapper plugin also by cloning their repo:

git clone https://github.com/pyenv/pyenv-virtualenvwrapper.git ~/.pyenv/plugins/pyenv-virtualenvwrapper

You can also check for additional info on their installation page.

With all installed now it's time to add their configs into your .bashrc / .zshrc / others. My config is for zsh and is this one, but you can also check on the installation guides if any adjustment is required for your setup:

# virtualenvwrapper
export WORKON_HOME=~/.virtualenvs
export PROJECT_HOME=~/Projects
export PYTHON_CONFIGURE_OPTS="--enable-shared"

# pyenv
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
pyenv virtualenvwrapper

You may leave the last line pyenv virtualenvwrapper commented for the moment, so you can install and configure the python versions that you want first.

Launch a new session of the terminal to get the new configs and install the python versions of your choice (I'll use 3.9 and 3.10 as example):

pyenv install 3.9.13
pyenv install 3.10.4

Now define the order of priority for your python installations (I usually keep system python last so you never pollute it with dependencies other than system specific):

pyenv global 3.10.4 3.9.13 system

You can run python --version to check if the one you chose to be the priority one is correct, then install in it virtualenvwrapper so the pyenv-virtualenvwrapper plugin can use it:

pip install virtualenvwrapper

After this long journey now its time to uncomment that pyenv virtualenvwrapper line in your rc file and launch a new terminal session and voilá :laughing:

You can create virtualenvs with the command:

mkvirtualenv <envname>

Also with other python versions as ref:

mkvirtualenv -p python3.9 <envname>

All envs will be created at the place you defined by the envvar WORKON_HOME and you can list them with:

workon

Or activate one of them with:

workon <envname>

As a plus, after activating one of them you can use the command setvirtualenvproject inside your code project folder to remember it as home of your env and every time you workon into this env it will redirect you to your code project folder.

And I think that covers the whole setup, if you have any other question just let me know :smiley:

[fannheyward]

Setting pythonPath for service "<default>": "/Users/fannheyward/.virtualenvs/t1/bin/python"

Reproduced, working on this.

[fannheyward]

Try :CocInstall [email protected], does it work ?

The fix in v1.1.251 may cause another issues, I've reverted the fix in v1.1.252.

[humrochagf]

Yup, it worked using version 1.1.251 :)

[humrochagf]

Took a better look into the logs using the version 1.1.251 and yeah it works for the internals but it will not work for any dependency that you installed because it sets the env as being the pyenv python path instead of being the virtualenv one:

Workspace: /home/humrochagf/Projects/example
Using python from /home/humrochagf/.pyenv/versions/3.10.4/bin/python3.10

[Info  - 11:16:44 AM] Pyright language server 1.1.251 starting
[Info  - 11:16:44 AM] Server root directory: /home/humrochagf/.config/coc/extensions/node_modules/coc-pyright/node_modules/pyright/dist/
[Info  - 11:16:44 AM] No configuration file found.
[Info  - 11:16:44 AM] No pyproject.toml file found.
[Info  - 11:16:44 AM] Setting pythonPath for service "example": "/home/humrochagf/.pyenv/versions/3.10.4/bin/python3.10"
[Warn  - 11:16:44 AM] stubPath /home/humrochagf/Projects/example/typings is not a valid directory.
[Info  - 11:16:44 AM] Assuming Python version 3.10
[Info  - 11:16:44 AM] Assuming Python platform Linux
[Info  - 11:16:44 AM] Searching for source files
[Info  - 11:16:44 AM] Found 1 source file
[Info  - 11:17:06 AM] No configuration file found.
[Info  - 11:17:06 AM] No pyproject.toml file found.
[Info  - 11:17:06 AM] Setting pythonPath for service ".pyenv": "/home/humrochagf/.pyenv/versions/3.10.4/bin/python3.10"
[Warn  - 11:17:06 AM] stubPath /home/humrochagf/.pyenv/typings is not a valid directory.
[Info  - 11:17:06 AM] Assuming Python version 3.10
[Info  - 11:17:06 AM] Assuming Python platform Linux
[Info  - 11:17:06 AM] Searching for source files
[Info  - 11:17:07 AM] Found 7370 source files

[fannheyward]

it works for the internals but it will not work for any dependency that you installed because it sets the env as being the pyenv python path instead of being the virtualenv one

Yes, I've reverted the fix, and looking into any other fix.

[fannheyward]

@humrochagf sorry for the delay, I'm still working on this, but didn't find a solution by now.

[humrochagf]

Oh no worries, when I have some spare time I'll also try to investigate more to see what can be done :)

[ytyt-yt]

Any update on this? I got the similiar issue with pyenv + virtualvenvwrapper.

I am developing a python library with pip install -e .. Everything works good under the project directory, but pyright failed to resolve the developing library path when open files outside the directory (and goto definition stop working).

Not sure if it's exactly the same issue, or it might be caused by the develope mode (-e for pip install)

[ytyt-yt]

Any update on this? I got the similiar issue with pyenv + virtualvenvwrapper.

I am developing a python library with pip install -e .. Everything works good under the project directory, but pyright failed to resolve the developing library path when open files outside the directory (and goto definition stop working).

Not sure if it's exactly the same issue, or it might be caused by the develope mode (-e for pip install)

I found a discussion: microsoft/pyright/issues/1473, and the pyrightconfig.json workaround is working.

[fannheyward]

Looks like Pyright had improved for this issue, with Pyright 1.1.280+, I can't reproduce the Setting pythonPath for service "<default>" logs, and goto definition works as expected.

@humrochagf can you try with the latest Pyright?