pylance-release icon indicating copy to clipboard operation
pylance-release copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

Pylance using a lot of memory

Open ale-dg opened this issue 1 year ago • 23 comments

Hi,

I was checking the activity monitor in my Mac these days and it always shows "Code Helper (plugin)" with over 1 GB of memory usage. When checking through the Terminal which process is it, it turns out it is Pylance the one which is using this much memory. This sometimes causes a lag in VSCode and, in my case, sometimes it crashes it.

Is there a workaround for it?

Thanks!

Best

Screenshot 2024-03-09 at 10 47 33 PM

ale-dg avatar Mar 10 '24 04:03 ale-dg

Same as you, I notice that when i update vscode 1.87, memory of process Code Help(Plugin) keep increasing when vscode lunched. Search pid by ps -ef|grep $pid, find the pylance (spwaned by Code Help(Plugin)), is eating my memory.

Dragon-GCS avatar Mar 11 '24 09:03 Dragon-GCS

Thanks for logging an issue. Please see https://github.com/microsoft/pylance-release/blob/main/TROUBLESHOOTING.md#filing-an-issue For how to catching the log file.

If you can share a code sample or repo that reproduces the problem that would be best.

bschnurr avatar Mar 11 '24 21:03 bschnurr

Hi @bschnurr ,

Thank you for your answer.

Not sure how to catch that log as it is not something I caught in VSCode but rather in the Activity Monitor from MacOS. As @Dragon-GCS mentioned, just catch the PID from the activity monitor and in the terminal shows which plug in is stealing so much memory and gets the machine hot as an oven, which in this case is Pylance.

I am not sure how often have you checked the main VSCode repo, but there have been a lot of issues reported where the he typing begins to lag (see https://github.com/microsoft/vscode-jupyter/issues/15351, https://github.com/microsoft/vscode/issues/206119, https://github.com/microsoft/vscode/issues/206865,). Even though some of them point to Jupyter, the one that seems to not "dump" memory is Pylance (Code Helper (Plugin)), which always stays over 1GB.

Let me know if I can help with something else.

Thank you again for your help.

Best,

ale-dg avatar Mar 11 '24 22:03 ale-dg

if your goal is reducing memory footprint over having more functionality available, then you can try these to turn off some expensive features in settings.json

"python.analysis.indexing": false,
"python.analysis.typeCheckingMode": "off",
"python.analysis.enablePytestSupport": false,
"python.analysis.inlayHints.pytestParameters": false,
"python.analysis.inlayHints.callArgumentNames": "off",
"python.analysis.inlayHints.functionReturnTypes": false,
"python.analysis.inlayHints.variableTypes": false,
"python.analysis.autoImportCompletions": false,
"python.analysis.useLibraryCodeForTypes": false,
"editor.semanticHighlighting.enabled": false,

this will turn off us from digging into third party packages to figure out (infer) types when the packages don't support type annotations. usually that's the one that cause us to load a lot of third-party modules into memory.

since your main scenario is notebook, some of this info could still be fetched from kernel if you have run your notebooks.

heejaechang avatar Mar 11 '24 23:03 heejaechang

also for the logs you need to copy the output window. "Select "View: Toggle Output" from the command palette (Ctrl+Shift+P on Windows/Linux, Command+Shift+P on macOS), then select "Python Language Server" in the dropdown on the right."

bschnurr avatar Mar 11 '24 23:03 bschnurr

Hi @heejaechang ,

Thank you for your help. I understand I can turn all those assistances off for the sake of memory, but it does not explain the lagging nor the amount of hints it pops when using it. Yet again I could just uninstall it and call it quits.

In addition to me, I see others filing similar issues here, and even more dramatic problems than mine (https://github.com/microsoft/pylance-release/issues/5531)

This morning I switched back to VSCode 1.82.3, Jupyter v2023.8.1002501831, Pylance v2023.8.30 and it runs as smooth as it should. Pylance now "only" uses 712 MB (from 1.2GB last time I used the latest version) and uses only 19 threads (or 8.1% of the CPU, whereas before it was around 20%). I also see another user did this and it helped (https://github.com/microsoft/pylance-release/issues/5377). I have even disabled the automatic updates since now I am afraid something will break if it updates and I have been delayed for far too long troubleshooting this.

@Dragon-GCS would you be able to help us retrieving the logs as @bschnurr suggests so they can analyse the problem, please? I would gladly do it, but I have downgraded all the packages. 😜

Thanks

Best

ale-dg avatar Mar 11 '24 23:03 ale-dg

@ale-dg, it's good to know that with VSCode 1.82.3, Jupyter v2023.8.1002501831, Pylance v2023.8.30, things work smoothly. Can you also tell us which python extension version you are using?

By the way, I am not saying we are not trying to address the performance issue. We are trying to figure it out, but so far, we are not getting enough information to reproduce it ourselves. This kind of performance issue is best addressed with a live reproduction so we can dig deeper. (We are fixing the ones that we could reproduce or investigate if users provide us with their project).

For your case as well, if you can provide us with the repository, such as your workspace (or a demo project that shows the symptoms) that we can open ourselves and investigate, then we might be able to help you much better.

heejaechang avatar Mar 12 '24 22:03 heejaechang

by the way, this (https://github.com/microsoft/pylance-release/issues/5531) is different issue. that one is something is crashing (it could be us but not 100% sure from the info provided by the user) and creating core dump in the directory where pylance extension is installed.

if we are crashed, then we would expect core dump to be created in the executable folder (the folder vscode exe exists) not our extension folder where js files exist.

heejaechang avatar Mar 12 '24 22:03 heejaechang

Hi @heejaechang

My apologies if I sounded too blunt, it wasn't my intention. Just trying to imply that if I turn off all the assistances, then there is not a case on holding to Pylance and just uninstalling it as soon as possible.

As of now, I can't share the notebook I am working on since it is a work-in-progress and it is not finished yet, and even so the file is quite heavy (38MB), so GitHub has its restrictions on it. What I can tell you is that the notebook has around 185 active coding cells and several markdowns (it's a deliverable, so the markdowns must be in there). As it's a data science project, naturally it has some DataFrames using pandas, some graphs using Matplotlib, Seaborn, Plotly and GeoPandas and I just began with some "light" machine learning using scikit-learn, everything with the latest releases as it is a new environment (see attached).

The Python extension version is v2024.2.1

Best

environment.yml.zip

ale-dg avatar Mar 12 '24 23:03 ale-dg

hmm.. the enviroment.yml.zip link doesnt work. 404 error.

heejaechang avatar Mar 12 '24 23:03 heejaechang

... weird... I just dropped it... now I've attached it... environment.yml.zip

ale-dg avatar Mar 13 '24 00:03 ale-dg

Hi @heejaechang ,

Thank you for your help. I understand I can turn all those assistances off for the sake of memory, but it does not explain the lagging nor the amount of hints it pops when using it. Yet again I could just uninstall it and call it quits.

In addition to me, I see others filing similar issues here, and even more dramatic problems than mine (#5531)

This morning I switched back to VSCode 1.82.3, Jupyter v2023.8.1002501831, Pylance v2023.8.30 and it runs as smooth as it should. Pylance now "only" uses 712 MB (from 1.2GB last time I used the latest version) and uses only 19 threads (or 8.1% of the CPU, whereas before it was around 20%). I also see another user did this and it helped (#5377). I have even disabled the automatic updates since now I am afraid something will break if it updates and I have been delayed for far too long troubleshooting this.

@Dragon-GCS would you be able to help us retrieving the logs as @bschnurr suggests so they can analyse the problem, please? I would gladly do it, but I have downgraded all the packages. 😜

Thanks

Best

Just saw it， I’ll try to reproduce this issue tomorrow on my Macbook

Dragon-GCS avatar Mar 13 '24 14:03 Dragon-GCS

Hi @heejaechang @bschnurr

Just trying to keep all in the same loop. I opened another issue in VSCode related to the battery drainage and a lot of people jumped in it too. Another fellow loaded some info there (see https://github.com/microsoft/vscode-jupyter/issues/14459#issuecomment-1986677050) and does he does not have Pylance installed (according to him).

So, honestly now I am at odds on what is going on because apparently everything is draining memory and battery.

Best

ale-dg avatar Mar 13 '24 22:03 ale-dg

Just for update... I tried to run the log of Pylance today with the latest updates as I was getting the logs for the Jupyter extension... and just a couple of things... just by opening VSCode, automatically logs 31,087 lines and I have no idea when does Pylance finishes. Then like 10,000 of them are just parsing and analysing cells . Even so, I attach it below, without running anything.

Pylance.log

ale-dg avatar Mar 14 '24 00:03 ale-dg

Bellowing test with a new vscode profile

Test Repo: https://github.com/Dragon-GCS/apscheduler-webui (or any workspace with a python file)

versions

  • Vscode 1.87.2

Extensions：(All of them is the latest with specified vscode version)

  • Python v2024.3.1
  • Pylance v2024.2.1
  • Copilot v1.41.0(latest is 0.141.0)
  • Copilot Chat 0.11.1(latest is 0.13.1)

Maybe problem is Copilot Chat, all works well when I update CopilotChat to latest version

Why I use this verion of copilot?

Because I have to different account to use copilit discussion

What I found

  • This copilot version working with vscode 1.87 will take a lot cpu and will cause vscode to crash , maybe because of bellowing logs
  • Copilot Chat won't take a lot memory, only with Pylance, the memory is increasing.
  • This suggestion icon is keep blinking image

Logs

Copilot Chat (Repeating and not stoping until program broken down):

2024-03-14 10:38:41.496 [info] [auth] Logged in as Dragon-GCS
2024-03-14 10:38:41.496 [info] [chat] copilot token chat_enabled: true
2024-03-14 10:38:41.496 [info] [extension] Could not register explain command: Error: command 'github.copilot.interactiveEditor.explain' already exists
    at l.registerCommand (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:150:183445)
    at Object.registerCommand (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:160:20857)
    at o1 (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1322:152)
    at khe (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1322:5173)
    at xv.registerProviders (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1324:7474)
    at xv.set activated [as activated] (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1324:6716)
    at xv.set enabled [as enabled] (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1324:6472)
    at Ho.<anonymous> (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1326:178)
    at Ho.emit (node:events:526:35)
    at Ho.emit (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:184:22301)
    at H2 (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:188:12269)
    at kGe (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1245:2854)
    at OGe (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1245:3192)
    at PP.getCopilotToken (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1245:4503)
    at xq.register (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1258:1612)
2024-03-14 10:38:41.496 [info] [chat] Registration of chat agents failed:,Error: command 'github.copilot.interactiveSession.feedback' already exists
2024-03-14 10:38:41.574 [info] [auth] Logged in as Dragon-GCS
2024-03-14 10:38:41.574 [info] [chat] copilot token chat_enabled: true
2024-03-14 10:38:41.575 [info] [extension] Could not register explain command: Error: command 'github.copilot.interactiveEditor.explain' already exists
    at l.registerCommand (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:150:183445)
    at Object.registerCommand (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:160:20857)
    at o1 (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1322:152)
    at khe (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1322:5173)
    at xv.registerProviders (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1324:7474)
    at xv.set activated [as activated] (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1324:6716)
    at xv.set enabled [as enabled] (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1324:6472)
    at Ho.<anonymous> (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1326:178)
    at Ho.emit (node:events:526:35)
    at Ho.emit (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:184:22301)
    at H2 (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:188:12269)
    at kGe (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1245:2854)
    at OGe (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1245:3192)
    at PP.getCopilotToken (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1245:4503)
    at xq.register (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1258:1612)
2024-03-14 10:38:41.575 [info] [chat] Registration of chat agents failed:,Error: command 'github.copilot.interactiveSession.feedback' already exists
2024-03-14 10:38:41.634 [info] [auth] Logged in as Dragon-GCS
2024-03-14 10:38:41.635 [info] [chat] copilot token chat_enabled: true
2024-03-14 10:38:41.635 [info] [extension] Could not register explain command: Error: command 'github.copilot.interactiveEditor.explain' already exists
    at l.registerCommand (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:150:183445)
    at Object.registerCommand (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:160:20857)
    at o1 (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1322:152)
    at khe (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1322:5173)
    at xv.registerProviders (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1324:7474)
    at xv.set activated [as activated] (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1324:6716)
    at xv.set enabled [as enabled] (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1324:6472)
    at Ho.<anonymous> (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1326:178)
    at Ho.emit (node:events:526:35)
    at Ho.emit (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:184:22301)
    at H2 (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:188:12269)
    at kGe (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1245:2854)
    at OGe (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1245:3192)
    at PP.getCopilotToken (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1245:4503)
    at xq.register (/Users/dragon/.vscode/extensions/github.copilot-chat-0.11.1/dist/extension.js:1258:1612)
2024-03-14 10:38:41.635 [info] [chat] Registration of chat agents failed:,Error: command 'github.copilot.interactiveSession.feedback' already exists

Pylance log Python Language Server.log

Maybe these can help figure out why pylance is taking up so much memory @bschnurr

@ale-dg Are you using this copilot too?

Dragon-GCS avatar Mar 14 '24 02:03 Dragon-GCS

@Dragon-GCS nope, I have even just ran a "clean" VSCode with minimal extensions.

Best

ale-dg avatar Mar 14 '24 03:03 ale-dg

@ale-dg I see this error at the bottom of the logs. "Workspace indexing has hit its upper limit: 2000 files"

did you try turning off indexing? see below

"python.analysis.indexing": false,
"python.analysis.typeCheckingMode": "off",
"python.analysis.enablePytestSupport": false,
"python.analysis.inlayHints.pytestParameters": false,
"python.analysis.inlayHints.callArgumentNames": "off",
"python.analysis.inlayHints.functionReturnTypes": false,
"python.analysis.inlayHints.variableTypes": false,
"python.analysis.autoImportCompletions": false,
"python.analysis.useLibraryCodeForTypes": false,
"editor.semanticHighlighting.enabled": false,

Are you using "openFileOnly" mode "python.analysis.diagnosticMode": "openFilesOnly",

You can all add directories to exclude "python.analysis.exclude": ["cdk.out"],

If none of that works try turning off type inference "python.analysis.useLibraryCodeForTypes": false,

bschnurr avatar Mar 14 '24 17:03 bschnurr

@bschnurr nope... I just turned on the logging as suggested above 😅, let me try setting all of those and see what it logs

ale-dg avatar Mar 14 '24 19:03 ale-dg

@bschnurr I have activated all the settings above. Let me know if it helps.

Best

Pylance-2.log

ale-dg avatar Mar 14 '24 19:03 ale-dg

oh sorry. i mean you have to exclude any folders yourself, ["cdk.out"] was just an example.

 "python.analysis.exclude": [" your folder here"],

but does it seem fast now? after startup finishes the first analysis it should feel faster since more types are now cached.

bschnurr avatar Mar 14 '24 20:03 bschnurr

oh sorry. i mean you have to exclude any folders yourself, ["cdk.out"] was just an example.

 "python.analysis.exclude": [" your folder here"],

but does it seem fast now? after startup finishes the first analysis it should feel faster since more types are now cached.

Well naturally if I turn off all the assistances it will feel faster, but when I turn them back on, everything begins to slowdown. @DonJayamanne opened an issue yesterday https://github.com/microsoft/vscode/issues/210528 for this. @heejaechang asked me to pull some cpuprofiles which I have uploaded there as well.

Would it be necessary to get the logs again? I'm asking for downgrading to the stable release again.

Best

ale-dg avatar Mar 14 '24 22:03 ale-dg

This is affecting me too with Pylance's server (~/.vscode-server/extensions/ms-python.vscode-pylance-2024.4.1/dist/server.bundle.js) using over 1.8GB of memory in a workspace with 26 folders, only 5 of which actually have Python scripts in them with a total count of 13 Python files.

This is quite frustrating as the VSC extensions used by the other part of my job - maintaining a very large PHP+React application - tend to scale their memory usage up and down as they do stuff, which means that during "busy" spikes, my WSL instance runs out of memory, causing VSC to slow to a crawl then crash.

This has been an issue for a very long time, and the fact that a FAQ entry has been written to detail how to work around out-of-memory issues strongly implies that the development team has basically given up on this problem.

Link for those who are curious: https://github.com/microsoft/pylance-release/blob/main/TROUBLESHOOTING.md#pylance-is-crashing

Could you please spend some development time exploring options to reduce the memory usage of this extension.

Some ideas:

  • Ignoring folders or projects that don't have python scripts in them - this will help users with multi-folder workspaces.
  • Pushing data to the frontend that doesn't need to be kept in the backend - this will help users with dev containers or WSL remotes
  • Purging data that isn't needed "now", e.g. analysing all cached data and balancing how much effort is required to reproduce it vs how much storage it uses - this will help everyone

I'm sorry if this is coming across as condescending but this is a very old issue and is seemingly being ignored because the majority of users of this extension seem to be throwing RAM at the issue rather than reporting it. It's doubly frustrating as other extension authors seem to be taking memory usage reasonably seriously.

Just to illustrate how other extensions are managing their memory usage, here are some stats from the VSC remote server for my workspace:

  • A community PHP language server: 1.3GB to manage 8 top-level folders with 66+ thousand PHP files
  • Official Terraform extension: 1GB to manage 3 top-level folders with hundreds of Terraform files in over a hundred configurations
  • The rest of the VSC remote instance including 24+ other extensions - this includes all the TypeScript / Javascript related extensions: ~400MB

JulianCBC avatar Apr 09 '24 07:04 JulianCBC

I have the same problem. Pylance is consistently using 2.5 GB. My laptop is crashing frequently, and I haven't been able to find other causes so far. I'm using vscode with a docker container, and have multiple projects in my workspace. I have tried turning off the settings suggested in https://github.com/microsoft/pylance-release/issues/5614#issuecomment-1989619126.

kjersbry avatar Aug 26 '24 14:08 kjersbry