pylance crash: Allocation failed - JavaScript heap out of memory

[dibalavs]

Environment data

Pylance v2024.5.1 VScode: Version: 1.89.1 (user setup) Commit: dc96b837cf6bb4af9cd736aa3af08cf8279f7685 Date: 2024-05-07T05:13:33.891Z Electron: 28.2.8 ElectronBuildId: 27744544 Chromium: 120.0.6099.291 Node.js: 18.18.2 V8: 12.0.267.19-electron.0 OS: Windows_NT x64 10.0.19045 work over Remote SSH

python version: 3.10 (on remote host)

Repro Steps

1. open very-very large workspace (you even not need to open .py file in editor)
2. wait couple of minutes
3. pylance will crashed

Expected behavior

no crashes

Actual behavior

pylance get more than 4.5 Gb resident memory and crashed. after restart, it will crashed again and again

Logs

pylance_crash.txt

[heejaechang]

can you provide log? https://github.com/microsoft/pylance-release/wiki/Collecting-data-for-an-investigation.#collecting-trace-logs

please make sure it contains all info necessary.

[dibalavs]

Ok. Please find it in attachment pylance_crash_trace.txt

[rchiodo]

This being in the log likely means there's just too many files being scanned:

2024-05-28 13:10:49.670 [info] [Error - 1:10:49 PM] (1091826) Enumeration of workspace source files is taking longer than 10 seconds.
This may be because:
* You have opened your home directory or entire hard drive as a workspace
* Your workspace contains a very large number of directories and files
* Your workspace contains a symlink to a directory with many files
* Your workspace is remote, and file enumeration is slow
To reduce this time, open a workspace directory with fewer files or add a pyrightconfig.json configuration file with an "exclude" section to exclude subdirectories from your workspace. For more details, refer to https://github.com/microsoft/pyright/blob/main/docs/configuration.md.

You can alleviate some of this following the suggestions here: https://github.com/microsoft/pylance-release/blob/main/TROUBLESHOOTING.md#pylance-is-crashing

Ideally we'd be able to handle any number of python files, but at the moment we can't.

[dibalavs]

Is it possible to run indexing only when I open .py flie? I do not need to work with py files each time I open my large workspace. For example other language servers (clangd) not started until I open .c or .cpp file.

Is it possible index only opened files not all 26000 files?

Why python.analysis.exclude does not work? I try to set up excludes:

"python.analysis.exclude": [
                                        "**/dist"
                                ],

or

"python.analysis.exclude": [
                                        "${workspaceFolder}/../dist"
                                ],

But pylance ignore both of them

[heejaechang]

Is it possible to run indexing only when I open .py flie? I do not need to work with py files each time I open my large workspace. For example other language servers (clangd) not started until I open .c or .cpp file.

it already does.

Is it possible index only opened files not all 26000 files?

you can get that by python.analysis.indexing: false, we index only open files by default, that option tell us to index closed files as well.

Why python.analysis.exclude does not work?

I see you are using multi root workspace, and dist is the root of one of your workspace. so **/dist won't work.

what you need is creating .vscode/settings.json in your dist folder (one of your root), and put "python.analysis.exclude": ["**"] or something like that there.

[dibalavs]

it already does.

I launch workspace, and currently opened is .mk file but pylance is started and indexing all of possible files

you can get that by python.analysis.indexing: false, we index only open files by default, that option tell us to index closed files as well.

Thanks. I will try.

I see you are using multi root workspace, and dist is the root of one of your workspace. so **/dist won't work.

In case of clangd/ms intellisence multi root workspaces are working correctly. It is good idea to make it work for pylance also.

Also I found settettings python.analysis.userFileIndexingLimit I tried to set up 200 files, but it looks like this setting does not work. pylance also crashed.

[dibalavs]

You can get that by python.analysis.indexing: false,

I tried this option. It looks like pylance ignores it. Also after restart VScode, pylance tries to re-index all files and also crashes when it get more than 4.5Gb RAM

[heejaechang]

I launch workspace, and currently opened is .mk file but pylance is started and indexing all of possible files

ya, looks like we only delay 3rd party lib but not user files.

I tried this option. It looks like pylance ignores it. Also after restart VScode, pylance tries to re-index all files and also crashes when it get more than 4.5Gb RAM

where did you put the option? I just tried it and it worked as expected. since vscode supports nested settings machine settings -> user settings -> workspace settings -> folder setting, make sure indexing is not enabled in one setting but not in the other.

Also I found settettings python.analysis.userFileIndexingLimit I tried to set up 200 files, but it looks like this setting does not work. pylance also crashed.

same for this.

In case of clangd/ms intellisence multi root workspaces are working correctly. It is good idea to make it work for pylance also.

interesting. so, their exclude include root itself?

[dibalavs]

where did you put the option? I just tried it and it worked as expected. since vscode supports nested settings machine settings -> user settings -> workspace settings -> folder setting, make sure indexing is not enabled in one setting but not in the other.

I use Workspace settings:

[heejaechang]

are you using remote? if so, you need to set option at the remote, local setting has no impact.

[dibalavs]

My workspace file: prod_boost.code-workspace.zip

[dibalavs]

are you using remote? if so, you need to set option at the remote, local setting has no impact.

My workspace file is on the remote host, and pylance is running also on remote host.

[heejaechang]

I just tested it on WSL and SSH, and all works as expected. can you do this? https://github.com/microsoft/pylance-release/wiki/Collecting-data-for-an-investigation.#collecting-trace-logs-using-commands

and provide log file generated at the remote side? this will tell us whether indexing is really turned off.

[dibalavs]

Hello. I did next settings in workspace file (on remote side):

"python.analysis.logLevel": "Trace",
				"python.languageServer": "Default",
				"python.analysis.userFileIndexingLimit": 200,
				"python.analysis.indexing": false,
				"python.analysis.logTypeEvaluationTime": true,
				"python.analysis.typeEvaluationTimeThreshold": 500,
				"python.analysis.diagnosticSeverityOverrides": {

				}

b.t.w. next settings are unable to set up via settings and they greyout in json file:

				"python.analysis.logTypeEvaluationTime": true,
				"python.analysis.typeEvaluationTimeThreshold": 500,

My logs (also on remote side (~/.vscode-server/data/logs/20240530T083057/exthost2/ms-python.python)

pylance.tar.gz

[dibalavs]

B.T.W. My local host is Win and my remote host is Linux

It is strange to see Win-specific paths in logs on remote linux host 2024-05-30 08:35:15.456 [info] [Info - 8:35:15 AM] (4181204) Setting environmentName for service "dist": "C:\Users\dybala\AppData\Local\Programs\Python\Python310\python.exe"

[heejaechang]

I see no indexing going on. are you sure that python.analysis.indexing not working?

"indexing": false,
"userFileIndexingLimit": 200

and it is set correctly.

for this

2024-05-30 08:35:15.456 [info] [Info - 8:35:15 AM] (4181204) Setting environmentName for service "dist": "C:\Users\dybala\AppData\Local\Programs\Python\Python310\python.exe"

did you select interpreter at the remote side? or select one at the local?

...

that said, crash still happening, so probably not related to indexing.

have you tried creating .vscode/settings.json in side of dist folder and put

{
    "python.analysis.exclude = ["**"]
}

this should make dist not scan whole folder.

also, do you have cyclic symlink somewhere? that could be also cause of this.

[dibalavs]

I see no indexing going on. are you sure that python.analysis.indexing not working?

How can I check it? Currently I am seeing than node process with pylance do hard working and eventually crashed when it get 4.5Gb memory.

did you select interpreter at the remote side? or select one at the local?

I did not select. I even do not work with .py files and I do not have suggestion to select interpreter currently pylance get 4.7Gb memory and crashed.

have you tried creating .vscode/settings.json in side of dist folder and put

Tried with no effect

cat dist/.vscode/settings.json 
{
    "python.analysis.exclude" : [
        "**"
    ]
}

Crash was gone when I create in the root of project pyrightconfig.json file:

 cat ./pyrightconfig.json
{
  "exclude": [
    "**"
  ]
}

But It looks ugly. I prefere to keep all settings in single .code-workspace file.

[heejaechang]

we added some code to handle cyclic symlinks recently that will be released this week. let's try that bits to see whether it can mitigate your issue.

that said, let me see what we can do on this exclude issue.

[heejaechang]

I confirmed userFileIndexLimit works as expected.

[heejaechang]

This issue has been fixed in prerelease version 2024.7.100, which we've just released. You can find the changelog here: CHANGELOG.md