Jupyter notebook IntelliSense doesn't autocomplete modules in workspace subfolders when `"python.pylanceLspNotebooksEnabled": true`

[torext]

Environment data

• Language Server version: 2022.8.20
• OS and version: linux x64
• Python version (and distribution if applicable, e.g. Anaconda): 3.9.10, Conda
• python.analysis.indexing: null
• python.analysis.typeCheckingMode: off

Repro Steps

1. Create a new workspace folder myworkspace with the following structure:

myworkspace/
    subfolder/
        myclasses.py

2. Put the following inside myclasses.py:

class myclass:
    pass

3. Make sure the user setting "python.pylanceLspNotebooksEnabled": true is set.
4. Open a new blank Jupyter notebook using Create: New Jupyter Notebook from the command palette
5. Start typing from subfolder.myclas

Expected behavior

I should get an autocompletion suggestion for from subfolder.myclasses.

Actual behavior

No suggestions are given, even after pressing Ctrl + Space to force suggestions. Finishing typing the command

from subfolder.myclasses import myclass

doesn't give autocomplete suggestions at any point, but executes fine when the cell is run.

Logs

Python output (sensitive info modified):

LSP Notebooks experiment is enabled
LSP Notebooks interactive window support is enabled
> conda info --json
Python interpreter path: ~/anaconda3/envs/myenv/bin/python
> ~/.anaconda/bin/conda info --json
> ~/.conda/bin/conda info --json
> ~/anaconda3/bin/conda info --json
> /bin/python3 ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py
> /bin/python3.7 ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py
> /usr/bin/python3 ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py
> /usr/bin/python3.7 ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py
> /usr/include/python3.7 ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py
[ERROR 2022-7-11 11:7:10.145]: Error: Command failed: /usr/include/python3.7 /home/user/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py /home/user/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py
/bin/sh: 1: /usr/include/python3.7: Permission denied

    at ChildProcess.exithandler (node:child_process:406:12)
    at ChildProcess.emit (node:events:390:28)
    at maybeClose (node:internal/child_process:1064:16)
    at Socket.<anonymous> (node:internal/child_process:450:11)
    at Socket.emit (node:events:390:28)
    at Pipe.<anonymous> (node:net:687:12) {
  killed: false,
  code: 126,
  signal: null,
  cmd: '/usr/include/python3.7 /home/user/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py /home/user/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py'
}
> ~/anaconda3/bin/conda run -n myenv --no-capture-output python ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/printEnvVariables.py
> ~/anaconda3/bin/python ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py
> ~/anaconda3/envs/gst/bin/python ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py
> ~/anaconda3/envs/myenv/bin/python ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py
> ~/anaconda3/envs/myenv/bin/python ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py
Starting Pylance language server.
Jupyter provided interpreter path override: /home/user/anaconda3/envs/qcenv2/bin/python
Jupyter provided interpreter path override: /home/user/anaconda3/envs/qcenv2/bin/python
Jupyter provided interpreter path override: /home/user/anaconda3/envs/qcenv2/bin/python
Jupyter provided interpreter path override: /home/user/anaconda3/envs/qcenv2/bin/python
Jupyter provided interpreter path override: /home/user/anaconda3/envs/qcenv2/bin/python

Python language server output (sensitive info modified):

[Info  - 11:07:11 AM] (15656) Pylance language server 2022.8.20 (pyright 888eb243) starting
[Info  - 11:07:11 AM] (15656) Server root directory: /home/user/.vscode/extensions/ms-python.vscode-pylance-2022.8.20/dist
[Info  - 11:07:11 AM] (15656) Starting service instance "myworkspace"
[Info  - 11:07:11 AM] (15656) Notebook support: LSP
[Info  - 11:07:11 AM] (15656) Interactive window support: LSP
[Info  - 11:07:11 AM] (15656) No configuration file found.
[Info  - 11:07:11 AM] (15656) No pyproject.toml file found.
[Info  - 11:07:11 AM] (15656) Setting pythonPath for service "myworkspace": "/home/user/anaconda3/envs/myenv/bin/python"
[Warn  - 11:07:11 AM] (15656) stubPath /home/user/repositories/myworkspace/typings is not a valid directory.
[Info  - 11:07:11 AM] (15656) Assuming Python version 3.9
[Info  - 11:07:11 AM] (15656) Assuming Python platform Linux
[Info  - 11:07:11 AM] (15656) Searching for source files
[Info  - 11:07:11 AM] (15656) Found 1 source file
[Info  - 11:07:11 AM] (15656) Background analysis(1) root directory: /home/user/.vscode/extensions/ms-python.vscode-pylance-2022.8.20/dist
[Info  - 11:07:11 AM] (15656) Background analysis(1) started
[Info  - 11:07:14 AM] (15656) virtual workspace: untitled:Untitled-1.ipynb?jupyter-notebook
[Info  - 11:07:14 AM] (15656) Starting service instance "Untitled-1.ipynb"
[Info  - 11:07:14 AM] (15656) virtual workspace: untitled:Untitled-1.ipynb?jupyter-notebook
[Info  - 11:07:14 AM] (15656) Starting service instance "Untitled-1.ipynb"
[Info  - 11:07:14 AM] (15656) No pyproject.toml file found.
[Info  - 11:07:14 AM] (15656) Setting pythonPath for service "Untitled-1.ipynb": "/home/user/anaconda3/envs/myenv/bin/python"
[Warn  - 11:07:14 AM] (15656) stubPath typings is not a valid directory.
[Info  - 11:07:14 AM] (15656) Assuming Python version 3.9
[Info  - 11:07:14 AM] (15656) Assuming Python platform Linux
[Info  - 11:07:14 AM] (15656) Searching for source files
[Info  - 11:07:14 AM] (15656) No source files found.
[Info  - 11:07:14 AM] (15656) No pyproject.toml file found.
[Info  - 11:07:14 AM] (15656) Setting pythonPath for service "Untitled-1.ipynb": "/home/user/anaconda3/envs/myenv/bin/python"
[Warn  - 11:07:14 AM] (15656) stubPath typings is not a valid directory.
[Info  - 11:07:14 AM] (15656) Assuming Python version 3.9
[Info  - 11:07:14 AM] (15656) Assuming Python platform Linux
[Info  - 11:07:14 AM] (15656) Searching for source files
[Info  - 11:07:14 AM] (15656) No source files found.
[Info  - 11:07:15 AM] (15656) No pyproject.toml file found.
[Info  - 11:07:15 AM] (15656) Setting pythonPath for service "Untitled-1.ipynb": "/home/user/anaconda3/envs/myenv/bin/python"
[Warn  - 11:07:15 AM] (15656) stubPath typings is not a valid directory.
[Info  - 11:07:15 AM] (15656) Assuming Python version 3.9
[Info  - 11:07:15 AM] (15656) Assuming Python platform Linux
[Info  - 11:07:15 AM] (15656) Searching for source files
[Info  - 11:07:15 AM] (15656) No source files found.
[Info  - 11:07:43 AM] (15656) No pyproject.toml file found.
[Info  - 11:07:43 AM] (15656) Setting pythonPath for service "Untitled-1.ipynb": "/home/user/anaconda3/envs/myenv/bin/python"
[Warn  - 11:07:43 AM] (15656) stubPath typings is not a valid directory.
[Info  - 11:07:43 AM] (15656) Assuming Python version 3.9
[Info  - 11:07:43 AM] (15656) Assuming Python platform Linux
[Info  - 11:07:43 AM] (15656) Searching for source files
[Info  - 11:07:43 AM] (15656) No source files found.
[Info  - 11:07:43 AM] (15656) No pyproject.toml file found.
[Info  - 11:07:43 AM] (15656) Setting pythonPath for service "Untitled-1.ipynb": "/home/user/anaconda3/envs/myenv/bin/python"
[Warn  - 11:07:43 AM] (15656) stubPath typings is not a valid directory.
[Info  - 11:07:43 AM] (15656) Assuming Python version 3.9
[Info  - 11:07:43 AM] (15656) Assuming Python platform Linux
[Info  - 11:07:43 AM] (15656) Searching for source files
[Info  - 11:07:43 AM] (15656) No source files found.

[bschnurr]

might need __init__.py file in your subfolder

[torext]

might need __init__.py file in your subfolder

I'd disagree with that. First, before the recent LSP changes autocomplete suggestions would work fine without __init__.py inside the subfolder. Second, namespace packages don't require __init__.py since Python 3.3, and surely should be treated with the same respect as regular packages by the language server :)

I should've mentioned of course that the setup on my side is such that running

import sys
print(sys.path)

inside a notebook cell does show myworkspace to be in the Python path; I've implicitly said so when I mentioned that from subfolder.myclasses import myclass runs fine even if the suggestions aren't showing up. Otherwise of course this wouldn't be a bug...

[bschnurr]

this error is a little suspicious

could be do to extra space after 3.7 in the path? Error: Command failed: /usr/include/python3.7 /

[ERROR 2022-7-11 11:7:10.145]: Error: Command failed: /usr/include/python3.7 /home/user/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py /home/user/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py
/bin/sh: 1: /usr/include/python3.7: Permission denied

    at ChildProcess.exithandler (node:child_process:406:12)
    at ChildProcess.emit (node:events:390:28)
    at maybeClose (node:internal/child_process:1064:16)
    at Socket.<anonymous> (node:internal/child_process:450:11)
    at Socket.emit (node:events:390:28)
    at Pipe.<anonymous> (node:net:687:12) {
  killed: false,
  code: 126,
  signal: null,
  cmd: '/usr/include/python3.7 /home/user/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/get_output_via_markers.py /home/user/.vscode/extensions/ms-python.python-2022.12.0/pythonFiles/interpreterInfo.py'
}

[debonte]

@bschnurr, I think the issue here is that unsaved notebooks are limited workspaces. Once the notebook is saved, the import suggestions work as expected.

Btw, somewhat related, given that the cell executes fine, it seems that Jupyter treats unsaved notebooks as if they reside at the root of the workspace, since from subfolder.myclasses import myclass is only valid there, not in subfolder.

[torext]

@bschnurr, I think the issue here is that unsaved notebooks are limited workspaces. Once the notebook is saved, the import suggestions work as expected.

Btw, somewhat related, given that the cell executes fine, it seems that Jupyter treats unsaved notebooks as if they reside at the root of the workspace, since from subfolder.myclasses import myclass is only valid there, not in subfolder.

I agree that for saved notebooks things work as expected; it's just for the unsaved ones where I see the issue. Are there any updates on looking into this? I already posted about this on other threads related to the same issue for saved notebooks (see e.g. https://github.com/microsoft/pylance-release/issues/3017#issuecomment-1248097940), but the issue for unsaved notebooks is still outstanding.

It would be great if unsaved notebooks worked exactly like saved ones, as I frequently find it useful to prototype something real quick in an unsaved notebook without cluttering my local repo checkout.

[torext]

Are there any updates on looking into this?

[bschnurr]

appears to be working in Pylance v2023.3.30