jupyter-xeus
Stack imbalance and `base64enc.so` load errors when running with Playwright and a JupyterLite deployment
Hi! I've been trying to debug a strange issue with initialising the R kernel in a JupyterLite environment, which happens only with Playwright, apparently.
Running some basic R code such as lm(mpg ~ wt + hp + disp + cyl, data=mtcars) in Playwright displays the following logs with jupyterlite-xeus==4.1.2:
Error: .onLoad failed in loadNamespace() for 'methods', details:
call: initMethodDispatch(where)
error: object 'C_R_initMethodDispatch' not found
Warning: stack imbalance in '{', 110 then 78
Warning: stack imbalance in 'if', 108 then 76
Warning: stack imbalance in '{', 104 then 72
Warning: stack imbalance in 'if', 102 then 70
Warning: stack imbalance in '{', 98 then 66
Warning: stack imbalance in '{', 91 then 59
Warning: stack imbalance in 'if', 89 then 57
Warning: stack imbalance in '{', 85 then 53
Warning: stack imbalance in 'if', 80 then 8
Warning: stack imbalance in '{', 76 then 4
Warning: stack imbalance in 'if', 74 then 2
Error in dyn.load(file, DLLpath = DLLpath, ...) :·
unable to load shared object '/lib/R/library/base64enc/libs/base64enc.so':
Could not load dynamic lib: /lib/R/library/base64enc/libs/base64enc.so
Error: bad export type for 'PyByteArray_Type': undefined
Warning: stack imbalance in 'if', 126 then 89
Warning: stack imbalance in '<-', 124 then 87
Warning: stack imbalance in '{', 120 then 83
Warning: stack imbalance in '<-', 115 then 78
Error in getNativeSymbolInfo(symNames, dll, unlist = FALSE, withRegistrationInfo = TRUE) :·
must pass a package name, “DLLInfo” or “DllInfoReference” object
Warning: stack imbalance in 'if', 106 then 89
Warning: stack imbalance in 'if', 104 then 87
Warning: stack imbalance in 'if', 102 then 85
Error in FUN(X[[i]], ...) : object 'pkgName' not found
Error in FUN(X[[i]], ...) : object 'pkgName' not found
Error in FUN(X[[i]], ...) : object 'pkgName' not found
Error in FUN(X[[i]], ...) : object 'pkgName' not found
Warning: stack imbalance in 'lapply', 95 then 83
Warning: stack imbalance in '.Internal', 94 then 82
Warning: stack imbalance in '{', 90 then 78
Warning: stack imbalance in '<-', 85 then 73
Warning: stack imbalance in '{', 98 then 69
Warning: stack imbalance in '<-', 93 then 64
Warning: stack imbalance in '{', 89 then 60
Warning: stack imbalance in 'if', 87 then 58
Warning: stack imbalance in '{', 83 then 54
Warning: stack imbalance in '<-', 78 then 49
Warning: stack imbalance in '{', 111 then 45
Warning: stack imbalance in 'for', 104 then 38
Warning: stack imbalance in '{', 100 then 34
Warning: stack imbalance in 'if', 98 then 32
Warning: stack imbalance in '{', 94 then 28
Warning: stack imbalance in 'is.character', 88 then 37
Warning: stack imbalance in '||', 86 then 35
Warning: stack imbalance in 'if', 84 then 33
Warning: stack imbalance in '{', 80 then 29
Warning: stack imbalance in '<-', 75 then 24
Warning: stack imbalance in '{', 71 then 20
Warning: stack imbalance in 'if', 66 then 15
Warning: stack imbalance in 'if', 64 then 13
Warning: stack imbalance in '{', 60 then 9
and this happens only with Playwright as part of a test we run that uses the R kernel: https://github.com/JupyterEverywhere/jupyterlite-extension/blob/76bcabca965c8e2443a39f46f8bde6eaa5f86a1f/ui-tests/tests/jupytereverywhere.spec.ts#L453-L475.
Running with jupyterlite-xeus==4.0.5 instead of 4.1.2 does not display the above logs at all, but it still fails with very cryptic errors.
The R kernel initialises and runs code well on https://jupyter-xeus.github.io/xeus-r/lab/index.html when testing code outside of Playwright.
I know that this issue does not provide a proper reproducer at all, but I thought I should file this issue in case this is a known problem with newer versions of Xeus or the Xeus-R kernel. It is also likely that this isn't an issue with the R kernel at all, given that this test has been a part of our test suite and only started failing yesterday.
xref: JupyterEverywhere/jupyterlite-extension#176
Hi @agriyakhetarpal !
We haven't done many playwright tests with xeus-r. I wonder if this is an issue with the virtual file system. Would you be able to check if other shared objects can be loaded? I would like to know if base64enc is the only problematic one. These could be some test examples
library("digest")
library("htmltools")
getLoadedDLLs()
I'll try to replicate locally as well.
Hi @IsabelParedes, thanks for your response!
I can check this, though I'd now say that this happens in a very specialised case. jupyterlite-xeus==4.1.2 works well if I only have the xeus-r and xeus-python kernels installed. However, if I add a few compiled Python packages to the mix, that generates these warnings.
Here is a reproducer:
name: my-env
channels:
- https://repo.mamba.pm/emscripten-forge
- https://repo.prefix.dev/conda-forge
dependencies:
- xeus-python
- xeus-r
- numpy
- pandas
- scipy
- seaborn
- matplotlib
- scikit-learn
- r-ggplot2
- r-dplyr
- r-tidyr
- r-readr
- r-purrr
- r-tibble
- r-stringr
- r-forcats
- r-coursekata
- r-neuralnet
I can quickly set up a deployment with this and jupyterlite-xeus==4.1.2 from the template and share it here. Thanks!
I've set up a reproducer at https://github.com/agriyakhetarpal/jupyterlite-xeus-r-stack-imbalance-repro.
If I run the following code snippet by starting up a kernel at https://agriyakhetarp.al/jupyterlite-xeus-r-stack-imbalance-repro/lab/index.html:
lm(mpg ~ wt + hp + disp + cyl, data=mtcars)
I get the above logs about stack imbalances and base64enc.so load errors. Here's the environment file: https://github.com/agriyakhetarpal/jupyterlite-xeus-r-stack-imbalance-repro/blob/8087d9398b9f2145285ef89063605d2eb760c998/environment.yml. We are also facing a host of other problems when adding (compiled) Python packages and using xeus-python and xeus-r together. For example, @krassowski reported in https://github.com/emscripten-forge/recipes/issues/2661 how adding NumPy breaks the stats built-in R package. All of these might share a cause!
I think I know what the problem is. There is a known issue when scipy and r-base are in the same environment.
This is because scipy is built with openblas compiled with f2c, and for r-base we use the internal lapack/blas implementation and compile with flang. This causes conflicts when openblas.so, Rlapack.so, and Rblas.so are loaded together because there are some conflicting symbols.
We don't have a solution for this yet. One idea was to find a way to separate the kernel environments, or to load the shared libraries on demand. But we're open to suggestions.
I don't know why NumPy is breaking R though, that might be a separate issue. I'll look into that one a bit more.
I think I know what the problem is. There is a known issue when
scipyandr-baseare in the same environment.This is because `scipy` is built with `openblas` compiled with `f2c`, and for `r-base` we use the internal `lapack`/`blas` implementation and compile with flang. This causes conflicts when `openblas.so`, `Rlapack.so`, and `Rblas.so` are loaded together because there are some conflicting symbols.
We don't have a solution for this yet. One idea was to find a way to separate the kernel environments, or to load the shared libraries on demand. But we're open to suggestions.
Thanks, that makes sense! I also had the same thoughts. The Pyodide kernel does not face this issue at this time. We also recently added RPATH support in Pyodide 0.28, which means that we no longer need to load all shared libraries in a folder directly, and that speeds up symbol resolution (or it's supposed to, at least). Since the Pyodide kernel and the xeus-r kernel work well together, it could be worth changing the paths where xeus-python and xeus-r store their shared libraries (I speak of this from a higher-level understanding of as I don't fully understand the Xeus architecture or the loading mechanisms for either yet) and mount separate directories for both of them, so that they have no reason to interact with each other. Loading the shared libraries on demand would require RPATH support, IIUC.