%pip install pandas requires a kernel restart before being use - refactor to get_pandas that do the import each time, wrapped in a try-except.

[ronblum]

Positron Version:

Positron Version: 2024.06.1 build 27 Code - OSS Version: 1.90.0 Commit: a893e5b282612ccb2200102957ac38d3c14e5196 Date: 2024-06-26T01:22:29.024Z Electron: 29.4.0 Chromium: 122.0.6261.156 Node.js: 20.9.0 V8: 12.2.281.27-electron.0 OS: Linux x64 6.5.0-41-generic

Steps to reproduce the issue:

EDIT: 2024-07-18 by @jthomasmock

Daniel: I think I know what the issue is.

Pandas is discovered when ipykernel is first loaded, if we install pandas within the current session (eg with %pip install pandas) then even though you are able to use it, the internal ipykernel calls to pandas won't be able to use them as pd_ wasn't found when launching the kernel. > A simple solution is to simply restart the kernel.

Here's what happens when ipykernel is launched the following is evaluated, and pd_ becomes None as pandas is not installed. If you install pandas now with %pip install pandas, you can run import pandas and it will work, but pd_ won't be updated until you restart the kernel.

Wasim: I think we should move away from importing third party packages on startup the way we currently do, and instead have functions like get_pandas that do the import each time, wrapped in a try-except.

---

1. In the Python Console (cribbed from test plan)

import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randint(0, 10, size=(1000, 100)))
column_names = [f"Column_{i+1}" for i in range(100)]
df.columns = column_names

The df object should appear under the "Data" section

2. Click on the checkboard to the right of df to view the data.
   • Sometimes, the data explorer tab doesn't open.
   • No errors are emitted, though, in the output.
   • This doesn't happen consistenty.

What did you expect to happen?

A data explorer tab should open.

Were there any error messages in the output or Developer Tools console?

Not in the output.

[dfalbel]

I think I know what the issue is.

Pandas is discovered when ipykernel is first loaded, if we install pandas within the current session (eg with %pip install pandas) then even though you are able to use it, the internal ipykernel calls to pandas won't be able to use them as pd_ wasn't found when launching the kernel. A simple solution is to simply restart the kernel.

Here's what happens when ipykernel is launched the following is evaluated, and pd_ becomes None as pandas is not installed. If you install pandas now with %pip install pandas, you can run import pandas and it will work, but pd_ won't be updated until you restart the kernel.

https://github.com/posit-dev/positron/blob/ff87538651c017bcd442a2e8eb6c4d41a56c8454/extensions/positron-python/python_files/positron/positron_ipykernel/third_party.py#L64

https://github.com/posit-dev/positron/blob/ff87538651c017bcd442a2e8eb6c4d41a56c8454/extensions/positron-python/python_files/positron/positron_ipykernel/third_party.py#L20-L25

This behavior shouldn't be linux specific

[seeM]

I think we should move away from importing third party packages on startup the way we currently do, and instead have functions like get_pandas that do the import each time, wrapped in a try-except.

It’ll mean that users don't have to restart after installing (most of the time; some packages still require a restart), and it’ll make kernel startup a little bit faster since we’ll only import third party packages when they’re needed.

Alternatively, @dfalbel mentioned on Slack:

I think this makes sense! I think it's also fine having to restart the kernel, but we should at least re-surface the error to the UI, with some tips that restarting might fix.

[testlabauto]

Do we feel 100% confident that we have identified the root cause here? I ask because in smoke tests, I am seeing occasional failures where the test double clicks a variables row and no data explorer opens. These tests pre-install all their dependencies.

[dfalbel]

I don't see any other reasons for that to happen. Can you try in the smoket ests if running %view df returns an error?

[testlabauto]

Sure. I will try that. I also just put in retries in case double clicking on the variables row is the real problem.

[testlabauto]

It looks to me like click retries have solved the smoke test problem! Carry on :)

[jthomasmock]

Should we open an issue for Wasim's suggestion to try-expect and/or surface the error per Daniel?

[seeM]

@jthomasmock I made #3800 but then closed it. I think we can use this issue to track the %pip install pandas bug.

[jthomasmock]

I've relabled this issue and added context to original comment.

[dfalbel]

I ended up approaching this by forwarding the error to the user instead of lazily loading pandas as it's a simpler change and also solves other possible issues during initialization that wouldn't be covered by simply loading pandas lazily. See https://github.com/posit-dev/positron/pull/4122 for more info.

[testlabauto]

Verified Fixed

Positron Version(s) : 2024.07.0-113
OS Version          : OSX

Test scenario(s)

Error message appears for user when trying open data frame post pandas install without restart:

Link(s) to TestRail test cases run or created: N/A