Jupyter for vscode continues to be slow.

[loftusa]

Every few months I try to use vscode for jupyter because I would really love to just use vscode for everything. Every few months, I am disappointed and switch back to the web version.

There are two reasons for this:

1) Jupyter for vscode continues, stubbornly, to essentially always be more slow than traditional jupyter lab on localhost. Look at the run times in this screenshot. It took me a minute to run imports; when I ran the exact same code on the localhost version, it took 7.7 seconds (pictures attached). This is an extremely consistent theme in vscode jupyter. Cells will sometimes randomly take minutes to run, and will sometimes not even run at all until you press 'shift-enter' on them twice. This has been true for me across multiple computers, in many different dev environments.

Cells also just randomly take forever to run, for god knows what reason. Here is a screenshot of assigning a string to a variable taking 27.4 seconds:

Note that I am not trying to blame the team here, I am just frustrated because this is so close to being a great product, but this one thing holds it back, and it keeps not being fixed for years on end. The very first thing I would do as a product manager if I were in charge of vscode-jupyter is to pause all current tasks and plan, with the team, a multiple-month effort to speed things up, and get cells to run effectively instantly (or as close to the amount of time the python processing of the code takes as possible), every time.

2) Jupyter for vscode sucks at inline documentation, the equivalent of shift+tab in vscode jupyter. I am aware of the existence of the trigger parameter hints and show hover settings in the keyboard shortcuts. These are extremely unreliable, and actually show documentation when I press the button maybe 1/5 of the time. When they do show documentation, there is a 'loading' tag for awhile. Browser jupyter, on the other hand, is immediate with this. Basically every time. Below is an example.

The other issue with inline documentation is that, as far as I can tell, hover documentation for methods on instantiated variables simply doesn't work. When I am using pandas, for instance, typing df.unique( and then pressing the show hover hotkey while my typing carat is to the right of the parenthesis pops up a documentation window saying exactly nothing. In contrast, in the web version, typing the same thing produces full documentation, as expected.

I don't understand how these two issues aren't your guys's top priority. Everyone I've spoken to who uses jupyter has had exactly the same experience as I have, and everyone I've spoken to who uses jupyter uses the web version exclusively for exactly these issues. Even Kaggle notebooks are better. I love copilot and it'd be great to bring it into my jupyter notebook experience, but it has just never been viable to switch if I don't want a workflow where I have to wait for 30 seconds every time I press command-enter, or I am frustratingly making a new cell above the current one and typing function? just to see documentation.

These issues have been ongoing since vscode jupyter started. They are the only things holding me and everyone else I've spoken to back from using it. Without fixing these issues, the whole thing is unusable, and no other features you guys put in matter. Why are you guys working on anything besides this when they are the only things anyone I know cares about?

I should note that this is all running in a docker container with access to 7 of my 8 cpus and 10gb of RAM. I am on a 2022 macbook air. I realize that this is a rant, so thank you for reading it. Nothing personal, I just think this product has a bunch of potential and I hate to see it unusable for so long.

[DonJayamanne]

Thank you for filling this issue and sorry you are running into these issues

I would like to get to the bottom of these issues and get you unblocked

• please can you disable all extensions except Python , Jupyter and the Notebook renderers extension
• If you have the variable viewer open please hide that
• Run again and check if it’s slow (I if you have the power tools extension that can show things down)
• finally pear could you provide the logs so I can see what’s slow at your end (instructions below)

Please could you enable logging as follows:

• Open settings & go into Jupyter->Logging
• Change the value to verbose
• Reload VS Code,
• Attempt to repro the issue & then copy the output from the Jupyter output panel.
• Use the command ‘Jupyter: Show Output’ to get to the logs

[dschaub95]

I have exactly the same issues. The notebooks get especially slow as they get bigger. But many of the problems already exist in an empty notebook.

[jhancibo]

Same here as @dschaub95 . Could you @DonJayamanne upload a screenshot of Jupyter output panel? I have nothing here. This issue did not present couple of versions of vscode before. It is a recent versions thing.

[jhancibo]

Okay, I found Jupyter output panel. Here is the log @DonJayamanne . Any other information needed? I'm not sure how to command ‘Jupyter: Show Output’ to get to the logs, so I copy all of them from the Jupyter output panel. log.zip

[DonJayamanne]

@jhancibo @dschaub95 @loftusa Does this repro when you close the variable view Please ensure you always hide the variable view completely Let me know of the issue persists even after that

[dschaub95]

In my experience, it is entirely independent of the variable view.

[DonJayamanne]

If you have the Jupyter powertoys extension Please disable that as well

[dschaub95]

I don't use the powertoys extension at all. Maybe it's also important to mention that the problems with jupyter notebooks are even more severe when developing on a remote server (via ssh or Kubernetes). However, they still persist when developing locally.

[DonJayamanne]

@dschaub95 Please could you enable logging as follows:

• Open settings & go into Jupyter->Logging
• Change the value to verbose
• Reload VS Code,
• Attempt to repro the issue & then copy the output from the Jupyter output panel.

please share these logs when you run into perf issues

[dschaub95]

The output is attached. As far as I could see, the output only changed when the cell started executing. The time between me trying to execute and the actual execution seems not to be logged. logs.txt

[jhancibo]

In my experience, it is entirely independent of the variable view.

Same here. And I didn't install powertoys extension.

[jhancibo]

@DonJayamanne The issue persists after close the variable view completely.

[DonJayamanne]

@dschaub95 @jhancibo I’m being sorry this is still unresolved Please can you try the following

• disable all extensions except Python, Jupyter and notebook renders extension
• from the command palette select.Developer: Set log level
• Select extension host and select info
• When you run into this issue
• Please go into the output panel and select Extension host
• Clear the logs
• click the run cell button
• Capture the logs and let me know what you get
• Also let me know how long it took for you to see the output ‘NotebookController[..] EXECUTE Cell’ in the output panel
• Basically was there a delay between you clicking the button and the message getting logged
• Please do sure the logs from extension host when you replicate this issue
• And please do disable all but the about 3 extensions

Once again thank you for your continued patience ands support

[jhancibo]

@DonJayamanne Could you recording a video to do those instruction above? I tried and it is abstract to follow each step. For example, when I type Developer: Set log level in command palette, I see nothing pop up. If you have time to recording the video, I would be very happy to test it. Thanks.

[DonJayamanne]

The time between me trying to execute and the actual execution seems not to be logged.

Please send the logs from Extension Host and Jupyter From the ExtenisonHost logs I'm only interested in the lines that have the text [trace] NotebookController[...] EXECUTE cells

@dschaub95 @jhancibo Please could you send these two logs From what I understand

• Jupyter extension logs do not get updated for a while, meaning Jupyter extension does not recieve messages for execution for while

Please let me know

• How long it takes to see output in the ExtensionHost logs from the time you try to run a cell. @dschaub95 You have already confirmed that Jupyter logs do not get updated, please check the ExtensionHost logs as well.

Here are the instructions again (please clear the both ExtensionHost and Jupyter logs before you start executing cells)

Please can you try the following

• disable all extensions except Python, Jupyter and notebook renders extension
• from the command palette select.Developer: Set log level
• Select extension host and select info
• When you run into this issue
• Please go into the output panel and select Extension host
• Clear the logs
• click the run cell button
• Capture the logs and let me know what you get
• Also let me know how long it took for you to see the output ‘NotebookController[..] EXECUTE Cell’ in the output panel
• Basically was there a delay between you clicking the button and the message getting logged
• Please do sure the logs from extension host when you replicate this issue
• And please do disable all but the about 3 extensions

Once again thank you for your continued patience ands support

[DonJayamanne]

Closing this issue as its been over 4 weeks, since the information was requested. We'll be happy to reopen the issue when the requested information has been provided.

[yuuuxt]

got one example and the logs are in https://github.com/microsoft/pylance-release/issues/5301 (Note that I'm not disabling all other extensions).

UPDATE: having the same issue again, but nothing in Extension Host (in info level). the pending lasted for about one minutes. I tried it again with debug level for Extension Host, the only logs (from middle of pending to after execution) are as follows. (seems unrelated?)

meanwhile, the time shown below is incorrect - the waiting time is more than one minute.

2023-12-22 14:30:10.159 [debug] ExtHostNotebook#$acceptEditorPropertiesChanged 47805538-ec73-4099-90db-736613202be8 {"selections":{"selections":[{"start":34,"end":35}]}}
2023-12-22 14:30:27.062 [debug] ExtHostNotebook#$acceptEditorPropertiesChanged 47805538-ec73-4099-90db-736613202be8 {"selections":{"selections":[{"start":35,"end":36}]}}
2023-12-22 14:30:33.206 [debug] ExtHostNotebook#$acceptEditorPropertiesChanged 47805538-ec73-4099-90db-736613202be8 {"selections":{"selections":[{"start":35,"end":36}]}}
2023-12-22 14:30:48.061 [debug] ExtHostNotebook#$acceptEditorPropertiesChanged 47805538-ec73-4099-90db-736613202be8 {"visibleRanges":{"ranges":[{"start":33,"end":36}]}}
2023-12-22 14:30:51.428 [debug] ExtHostNotebook#$acceptEditorPropertiesChanged 47805538-ec73-4099-90db-736613202be8 {"visibleRanges":{"ranges":[{"start":33,"end":37}]}}
2023-12-22 14:30:52.066 [debug] ExtHostNotebook#$acceptEditorPropertiesChanged 47805538-ec73-4099-90db-736613202be8 {"selections":{"selections":[{"start":35,"end":36}]}}
2023-12-22 14:31:30.497 [debug] ExtHostNotebook#$acceptEditorPropertiesChanged 47805538-ec73-4099-90db-736613202be8 {"selections":{"selections":[{"start":35,"end":36}]}}
2023-12-22 14:32:34.055 [debug] ExtHostNotebook#$acceptEditorPropertiesChanged 47805538-ec73-4099-90db-736613202be8 {"visibleRanges":{"ranges":[{"start":33,"end":36}]}}

and again, pls note that I'm not disabling other extensions during work, which is not consistent with your instruction above.

[suiluj]

Hi, i think this issue should not be closed because it is not solved.

Or does someone have a solution?

When I run notebooks in jupyter lab in the browser everything is instant but in vscode everything runs delayed.

[FlorinAndrei]

Not sure if this is what you're seeing, but I've noticed a regression of an old bug. I have a code cell that should run in a fraction of a second. I run it. It's stuck for about 1 minute. Then all of a sudden it runs.

Very annoying. Because of this and other, numerous bugs, I'm thinking to go back to Jupyter Notebook in a browser.

[suiluj]

I don't know if Windows interferes the "vscode ipykernel" differently than the "kernel type" that is started by running jupyter lab.
But I noticed that on my macbook air m2 vscode restarts kernels and runs cells fast.

Perhaps there are some Windows processes like Windows Defender and proxy and vpn settings that interfere with the ipykernel when starting a kernel from vscode.

Here is an issue that describes some of these things:
https://github.com/jupyter/notebook/issues/1415

[FayzulSaimun]

I am experiencing the same issue. A few days ago, everything was working fine but after updating to the latest version, Visual Studio Code is using almost 90% of my CPU and the Jupyter Notebook is also running slow. Previously, I could use 3-4 notebooks at a time with my current setup but now running even one notebook is not possible. On the other hand, everything is smooth in Jupyter Lab.

"I have attached the cell loading data. It has been stuck for 5 minutes now and is not running."

[JasonGross]

In my experience, jupyter notebooks performance degrades very quickly in the size of the notebook. This is especially true for plotly.express plots, and is independent of whether I am using a .ipynb file or the interactive cell views for a .py file.

Describing the experience for a .py file: when there are no plots and no LaTeX in the interactive window, everything is snappy. But if I have even just a handful of plots (or many lines of rendered LaTeX from Markdown cells), then it takes multiple seconds between when I press Shift+Enter and when the interactive window starts running the command. If I click "clear all", everything is quick again. This seems to largely depend on how many plots are in the interactive window, not how many are currently visible.

Some other observations:

• Making fancy interactive .js plots slows vscode way more than making png plots
• When my notebooks contain many interactive plotly.express plots, the .ipynb files saved from vscode can be ~10x larger (100+MB rather than 10MB) than equivalent notebooks saved from Google Colab

[FlorinAndrei]

@JasonGross All that is true, but the bug where it gets stuck on a cell does not depend on notebook size or plot complexity.

[JasonGross]

Is this behavior related? I sometimes see code execution hanging for multiple minutes on trying to write to the interactive window. Maybe there's a similar blocking IO writing call that is deadlocked or something in the other cases?

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
Cell In[161], line 12
     10             weights[q_tok, max_tok, n_copies_nonmax] = (max_tok - 1) ** n_copies_nonmax * math.comb(model.cfg.n_ctx - 1, n_copies_nonmax)
     11 for _, v in min_gaps_list_nosvd:
---> 12     weighted_histogram(v.flatten().detach().numpy(), weights.flatten().detach().numpy(),labels={"x":"gap", "y":"count * # sequences"}, num_bins=v.max().item()).show(RENDERER)

File ~/guarantees-based-mechanistic-interpretability/.venv/lib/python3.10/site-packages/plotly/basedatatypes.py:3410, in BaseFigure.show(self, *args, **kwargs)
   3377 """
   3378 Show a figure using either the default renderer(s) or the renderer(s)
   3379 specified by the renderer argument
   (...)
   3406 None
   3407 """
   3408 import plotly.io as pio
-> 3410 return pio.show(self, *args, **kwargs)

File ~/guarantees-based-mechanistic-interpretability/.venv/lib/python3.10/site-packages/plotly/io/_renderers.py:386, in show(fig, renderer, validate, **kwargs)
    383 fig_dict = validate_coerce_fig_to_dict(fig, validate)
    385 # Mimetype renderers
--> 386 bundle = renderers._build_mime_bundle(fig_dict, renderers_string=renderer, **kwargs)
    387 if bundle:
    388     if not ipython_display:

File ~/guarantees-based-mechanistic-interpretability/.venv/lib/python3.10/site-packages/plotly/io/_renderers.py:294, in RenderersConfig._build_mime_bundle(self, fig_dict, renderers_string, **kwargs)
    291             if hasattr(renderer, k):
    292                 setattr(renderer, k, v)
--> 294         bundle.update(renderer.to_mimebundle(fig_dict))
    296 return bundle

File ~/guarantees-based-mechanistic-interpretability/.venv/lib/python3.10/site-packages/plotly/io/_base_renderers.py:126, in ImageRenderer.to_mimebundle(self, fig_dict)
    125 def to_mimebundle(self, fig_dict):
--> 126     image_bytes = to_image(
    127         fig_dict,
    128         format=self.format,
    129         width=self.width,
    130         height=self.height,
    131         scale=self.scale,
    132         validate=False,
    133         engine=self.engine,
    134     )
    136     if self.b64_encode:
    137         image_str = base64.b64encode(image_bytes).decode("utf8")

File ~/guarantees-based-mechanistic-interpretability/.venv/lib/python3.10/site-packages/plotly/io/_kaleido.py:143, in to_image(fig, format, width, height, scale, validate, engine)
    140 # Validate figure
    141 # ---------------
    142 fig_dict = validate_coerce_fig_to_dict(fig, validate)
--> 143 img_bytes = scope.transform(
    144     fig_dict, format=format, width=width, height=height, scale=scale
    145 )
    147 return img_bytes

File ~/guarantees-based-mechanistic-interpretability/.venv/lib/python3.10/site-packages/kaleido/scopes/plotly.py:153, in PlotlyScope.transform(self, figure, format, width, height, scale)
    142     raise ValueError(
    143         "Invalid format '{original_format}'.\n"
    144         "    Supported formats: {supported_formats_str}"
   (...)
    148         )
    149     )
    151 # Transform in using _perform_transform rather than superclass so we can access the full
    152 # response dict, including error codes.
--> 153 response = self._perform_transform(
    154     figure, format=format, width=width, height=height, scale=scale
    155 )
    157 # Check for export error, later can customize error messages for plotly Python users
    158 code = response.get("code", 0)

File ~/guarantees-based-mechanistic-interpretability/.venv/lib/python3.10/site-packages/kaleido/scopes/base.py:305, in BaseScope._perform_transform(self, data, **kwargs)
    302 self._std_error = io.BytesIO()
    304 # Write and flush spec
--> 305 self._proc.stdin.write(export_spec)
    306 self._proc.stdin.write("\n".encode('utf-8'))
    307 self._proc.stdin.flush()

KeyboardInterrupt: 

[DonJayamanne]

Is this behavior related? I sometimes see code execution hanging for multiple minutes on trying to write to the interactive

This does not seem to be related to VS Code or the like. Based on the error message, its a stack trace from Python code, ie its the python code that is hanging. I.e. python code you are running is hanging the kernel.

[DonJayamanne]

@FayzulSaimun

A few days ago, everything was working fine but after updating to the latest version, Visual Studio Code is using almost 90% of my CPU and the Jupyter Notebook is also running slow. Previously, I could use 3-4

Most likely unrelated to the Jupyter extension. Please can you follow the notes here and file an issue withe the relevant logs, to help us identify the bottleneck https://github.com/microsoft/vscode/wiki/Performance-Issues

[JasonGross]

Is this behavior related? I sometimes see code execution hanging for multiple minutes on trying to write to the interactive

This does not seem to be related to VS Code or the like. Based on the error message, its a stack trace from Python code, ie its the python code that is hanging. I.e. python code you are running is hanging the kernel.

@DonJayamanne I don't see how this could possibly be unrelated to VS Code. When I run code on the command line, it executes in a couple of seconds and dumps {'image/png': <base64-encoded png>} to stdout. When I execute in a cell in VS Code, it hangs on a stdin.write for over 5 minutes. This indicates to me that VS Code is failing to read from its buffers in a timely manner, resulting in hanging on things that are trying to print to the interactive window.

[DonJayamanne]

@suiluj

Here is an issue that describes some of these things: https://github.com/jupyter/notebook/issues/1415

Looks like this indicates that some of these issues could be caused by VPN issues, But i suspect thats not what you are running into or is it?

When I run notebooks in jupyter lab in the browser everything is instant but in vscode everything runs delayed.

Please can you share a very simple sample of this issue

• Open VS Code
• Clear all of the Jupyter logs (open the command Jupyter: Show Output to get to the logs)
• Close all notebooks
• Open a notebook
• Run a cell in a notebook with a simple print("Hello World") statement
• Share the logs
• If the steps to repro this per issue is more complex, please do share all relevant information such as what packages are installed and what code is executed (please share the full notebook if there's more than just one cell)

Please could you enable logging as follows:

• Open settings & go into Jupyter->Logging
• Change the value to verbose
• Reload VS Code,
• Attempt to repro the issue & then copy the output from the Jupyter output panel.

[vscodenpa]

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

[Animadversio]

In my experience, jupyter notebooks performance degrades very quickly in the size of the notebook. This is especially true for plotly.express plots, and is independent of whether I am using a .ipynb file or the interactive cell views for a .py file.

Describing the experience for a .py file: when there are no plots and no LaTeX in the interactive window, everything is snappy. But if I have even just a handful of plots (or many lines of rendered LaTeX from Markdown cells), then it takes multiple seconds between when I press Shift+Enter and when the interactive window starts running the command. If I click "clear all", everything is quick again. This seems to largely depend on how many plots are in the interactive window, not how many are currently visible.

Some other observations:

• Making fancy interactive .js plots slows vscode way more than making png plots
• When my notebooks contain many interactive plotly.express plots, the .ipynb files saved from vscode can be ~10x larger (100+MB rather than 10MB) than equivalent notebooks saved from Google Colab

I'm experiencing the same issue and I have exactly the same observation as @JasonGross .... which pushed me to switch to web-based version...