"No definition found for" property

[ibobak]

Type: Bug

Behaviour

when I am trying to navigate (Ctrl+B in my case, this is "Go to definition") on a property, it tells me "no definition found for [name_of_property]".

Steps to reproduce:

1. install this library https://pypi.org/project/shap/ as an example of the one which has properties (I believe it should work with others too)
2. start a notebook, make an instance of the class with properties, and try to navigate. It won't work:

Diagnostic data

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

Extension version: 2024.8.0 VS Code version: Code 1.90.0 (89de5a8d4d6205e5b11647eb6a74844ca23d2573, 2024-06-04T19:43:07.605Z) OS version: Linux x64 6.4.6-060406-generic Modes:

• Python version (& distribution if applicable, e.g. Anaconda): 3.10.14
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Conda
• Value of the python.languageServer setting: Pylance

User Settings

languageServer: "Pylance"

Installed Extensions

Extension Name	Extension Id	Version
.NET Install Tool	ms-dotnettools.vscode-dotnet-runtime	2.0.5
autoDocstring - Python Docstring Generator	njpwerner.autodocstring	0.6.1
Ayu Monokai	lakshits11.ayu-monokai	0.0.3
Black Formatter	ms-python.black-formatter	2024.2.0
C#	ms-dotnettools.csharp	2.31.19
C# Dev Kit	ms-dotnettools.csdevkit	1.6.8
Checkpoints	micnil.vscode-checkpoints	1.3.3
Code Runner	formulahendry.code-runner	0.12.2
Conventional Commits	vivaxy.vscode-conventional-commits	1.25.0
Data Wrangler	ms-toolsai.datawrangler	1.2.0
Dev Containers	ms-vscode-remote.remote-containers	0.369.0
Django	batisteo.vscode-django	1.15.0
Docker	ms-azuretools.vscode-docker	1.29.1
Docs View	bierner.docs-view	0.1.0
Edit csv	janisdd.vscode-edit-csv	0.9.1
Git Extension Pack	donjayamanne.git-extension-pack	0.1.3
Git Extension Pack (GPack)	SeyyedKhandon.gpack	2.0.0
Git File History	pomber.git-file-history	1.0.1
Git History	donjayamanne.githistory	0.6.20
Github Light Theme	Hyzeta.vscode-theme-github-light	7.14.2
GitHub Plus Theme	thenikso.github-plus-theme	1.4.3
GitHub Theme	GitHub.github-vscode-theme	6.3.4
gitignore	codezombiech.gitignore	0.9.0
GitLens — Git supercharged	eamodio.gitlens	15.1.0
Grey Light+ Pro	elanzalaco.grey-light-plus-pro	0.0.1
Inline SQL	qufiwefefwoyn.inline-sql-syntax	2.16.0
IntelliCode	VisualStudioExptTeam.vscodeintellicode	1.3.1
IntelliCode API Usage Examples	VisualStudioExptTeam.intellicode-api-usage-examples	0.2.8
IntelliCode for C# Dev Kit	ms-dotnettools.vscodeintellicode-csharp	2.1.11
JavaScript Debugger	ms-vscode.js-debug	1.90.0
JavaScript Debugger Companion Extension	ms-vscode.js-debug-companion	1.1.2
Jinja	wholroyd.jinja	0.0.8
julia light	JuliaLight.julia-light	0.4.0
Jupyter	ms-toolsai.jupyter	2024.5.0
Jupyter Cell Tags	ms-toolsai.vscode-jupyter-cell-tags	0.1.9
Jupyter Keymap	ms-toolsai.jupyter-keymap	1.1.2
Jupyter Notebook Renderers	ms-toolsai.jupyter-renderers	1.0.18
Jupyter Slide Show	ms-toolsai.vscode-jupyter-slideshow	0.1.6
Jupyter Theme	sam-the-programmer.jupyter-theme	0.1.3
JupyterLab Light Theme	MiguelCorralJr.jupyterlab-light-theme	0.1.1
Light Pink Theme	mgwg.light-pink-theme	0.6.0
Markdown All in One	yzhang.markdown-all-in-one	3.6.2
Markdown Preview Enhanced	shd101wyy.markdown-preview-enhanced	0.8.13
MZ Light Grey Theme	Malaz-YI.mz-light-grey	2.1.1
Night Owl	sdras.night-owl	2.0.1
Notepad++ color themes	ShayanAhmedKhan.notepadpp-color-theme	0.2.0
Numbered Bookmarks	alefragnani.numbered-bookmarks	8.4.0
One Dark Pro	zhuangtongfa.material-theme	3.17.2
One Dark Theme	mskelton.one-dark-theme	1.14.2
Open in GitHub, Bitbucket, Gitlab, VisualStudio.com !	ziyasal.vscode-open-in-github	1.3.6
Path Intellisense	christian-kohler.path-intellisense	2.9.0
Project Manager	alefragnani.project-manager	12.8.0
Pylance	ms-python.vscode-pylance	2024.6.1
Pylint	ms-python.pylint	2023.10.1
Python	ms-python.python	2024.8.0
Python Debugger	ms-python.debugpy	2024.6.0
Python Environment Manager	donjayamanne.python-environment-manager	1.2.4
Python Extension Pack	donjayamanne.python-extension-pack	1.7.0
Python Indent	KevinRose.vsc-python-indent	1.18.0
Remote - SSH	ms-vscode-remote.remote-ssh	0.110.1
Remote - SSH: Editing Configuration Files	ms-vscode-remote.remote-ssh-edit	0.86.0
Remote - Tunnels	ms-vscode.remote-server	1.5.1
Remote Development	ms-vscode-remote.vscode-remote-extensionpack	0.25.0
Remote Explorer	ms-vscode.remote-explorer	0.4.3
Studio Icons	jtlowe.vscode-icon-theme	1.6.6
Table Visualizer for JavaScript Profiles	ms-vscode.vscode-js-profile-table	1.0.9
Visual Studio Keymap	ms-vscode.vs-keybindings	0.2.1
WSL	ms-vscode-remote.remote-wsl	0.88.2
Zoomer	anthonyattard.zoomer	0.3.1

System Info

Item	Value
CPUs	Intel(R) Xeon(R) CPU E5-2696 v4 @ 2.20GHz (88 x 1197)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	2, 5, 6
Memory (System)	251.76GB (156.92GB free)
Process Argv	--crash-reporter-id 27d42247-63fb-4d9e-9cb0-87d9974843dc
Screen Reader	no
VM	50%
DESKTOP_SESSION	ubuntu-xorg
XDG_CURRENT_DESKTOP	Unity
XDG_SESSION_DESKTOP	ubuntu-xorg
XDG_SESSION_TYPE	x11

A/B Experiments

vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscoreces:30445986
vscod805:30301674
binariesv615:30325510
vsaa593:30376534
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
2i9eh265:30646982
962ge761:30959799
pythongtdpath:30769146
welcomedialog:30910333
pythonidxpt:30866567
pythonnoceb:30805159
asynctok:30898717
pythontestfixt:30902429
pythonregdiag2:30936856
pythonmypyd1:30879173
pythoncet0:30885854
h48ei257:31000450
pythontbext0:30879054
accentitlementst:30995554
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
jchc7451:31067544
chatpanelt:31048053
dsvsc021:30996838
9c06g630:31013171
pythoncenvpt:31062603
a69g1124:31058053
dvdeprecation:31068756
pythonprt:31056678
dwnewjupyter:31046869
26j00206:31048877

[debonte]

install this library https://pypi.org/project/shap/ as an example of the one which has properties (I believe it should work with others too)

It's not a general problem with properties, as go to def on d.year below works fine:

import datetime

d = datetime.date(1, 1, 1)
d.year

Here's a simplified repro:

import shap

explainer = shap.Explainer(1)
shap_values = explainer()
reveal_type(shap_values)  # Type of "shap_values" is "Unknown"
shap_values.data

We are having trouble inferring the return type of shap.Explainer.__call__. Since shap_values is seen as type Unknown (equivalent to Any), there's no way for us to go to def on shap_values.data.

I'm not sure what the specific issue is. If I start commenting out chunks of Explainer.__call__, we eventually report the return type as Unknown | list[Unknown]. But I think that's the best type we could expect here given that their return value (out) is not annotated and they use append to populate it. And if that's the case, go to def is never going to work here.

Here's a simplified example of a function that returns a list populated via append:

def func():
    out = []
    out.append(1)
    return out

result = func()
reveal_type(result) # Type of "result" is "list[Unknown]"

I'd suggest filing an issue on shap and seeing if they'd be willing to add type annotations, in particular on Explainer.__call__ which appears to return Explanation | List[Explanation].

Btw, @ibobak the images in your bugs are great and very helpful for understanding the scenario. Thanks for that! But providing code as text (also) would make it much easier for us to repro it.

[debonte]

I'd suggest filing an issue on shap and seeing if they'd be willing to add type annotations, in particular on Explainer.__call__ which appears to return Explanation | List[Explanation].

Oh, also, you should be able to improve your own experience by adding this annotation on shap_values

[ibobak]

Erik, thank you very much for your answer.

It is a commercial project, and I got this just in the middle of my work. I know submitting code would be easier for you, but to do this, I would have to reproduce this in a separate notebook, etc. (to avoid inclusion of the commercial code). I just did not have time at that moment, sorry for that.

As to what I understood from your answer: pylance is analyzing the code statically, and this is the biggest issue. We (data scientist) are working in the notebooks, and we run cell after cell. At a certain moment of time, after execution of the cell, we already know EXACTLY what is the type of the variable. Just EXACTLY. And we don't need to analyze anything: if the cursor is standing on the word "data" of expression "shap_values.data", and we already know what is the value of "shap_values", then we can just get this type and navigate to the property of "data" of this object, that's it.

No library maintainers (shap and others) will do any code changes just to fulfill the requirements of good static code: those libraries are what they are. Python is a dynamic programming language, work of a data scientist is a dynamic work: we are working in REPL mode, running cell by cell.

Therefore, functionality of code navigation, finding types, etc. should be seriously redesigned. And the main feature of this redesign should be finding out what is the type of variable live (at run-time of the notebook).

[ibobak]

To summarize, this refactoring should be made as to the "pylance in jupyter notebooks":

• if the variable is already initialized (because the code which initialized it was running) - just USE THIS TYPE
• if the variable is not initialized (just because you scrolled to the code but yet did not run it) - use previous functionality of static type finding

[debonte]

At a certain moment of time, after execution of the cell, we already know EXACTLY what is the type of the variable. Just EXACTLY. And we don't need to analyze anything: if the cursor is standing on the word "data" of expression "shap_values.data", and we already know what is the value of "shap_values", then we can just get this type and navigate to the property of "data" of this object, that's it.

@rebornix, what do you think about this? Does the Jupyter extension already have type info for all variables in a cell after it is executed (ex. for the Jupyter variables pane)? Could you provide us with the full dotted name of a particular variable's type so we could implement go to def in this scenario? Or maybe there's a better approach?

[rebornix]

@debonte we have api in both VS Code and Jupyter to fetch variables, and for the top levels, we have both the type and also value/summary. So technically you can use that one. I'm not sure if Pylance should use VS Code api or Jupyter, or it should be Jupyter contributing this info to Pylance.

[debonte]

Moving this issue to discussions as an enhancement request for comments and upvotes.