IJulia.jl icon indicating copy to clipboard operation
IJulia.jl copied to clipboard

Published 20 hours ago • verified publisher icon JuliaLang

nbconvert with Julia kernel fails inside `Pkg.test`

Open bencottier opened this issue 3 years ago • 5 comments

I have a Julia package with a function that runs nbconvert - it's actually using Weave.notebook but I've since found a MWE.

I want to test the function that runs nbconvert but I hit an error when nbconvert runs the Julia kernel.

MWE setup

For the MWE I created a package/folder IJuliaPkgTest.jl with the following structure:

.
├── Project.toml
├── src
│   └── IJuliaPkgTest.jl
├── test
│   └── runtests.jl
└── test.ipynb

Project.toml:

name = "IJuliaPkgTest"
uuid = "f2c0e409-174f-4158-90ff-9978d98f62c8"

[deps]
IJulia = "7073ff75-c697-5162-941a-fcdaad2a7d2a"

src/IJuliaPkgTest.jl just contains module IJuliaPkgTest end.

test/runtests.jl:

using IJulia

run(`jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --execute ../test.ipynb --debug --output ../test_executed.ipynb`)

Finally, test.ipynb:

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-06T08:04:32.314433Z",
     "iopub.status.busy": "2022-01-06T08:04:31.594744Z",
     "iopub.status.idle": "2022-01-06T08:04:33.800845Z",
     "shell.execute_reply": "2022-01-06T08:04:33.800037Z"
    }
   },
   "outputs": [],
   "source": [
    "println(\"Hello!\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.6.2",
   "language": "julia",
   "name": "julia-1.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

The problem

Here is the error when I run Pkg.test. The key thing is this after the julia kernel starts:

ERROR: LoadError: ArgumentError: Package Revise not found in current path:
- Run `import Pkg; Pkg.add("Revise")` to install the Revise package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:893
in expression starting at /Users/bencottier/.julia/config/startup.jl:1

As you can see I have a startup.jl file, which contains using Revise. I have Revise.jl in my base environment, but the base environment doesn't seem to be available here.

Full logs:

(IJuliaPkgTest) pkg> test
     Testing IJuliaPkgTest
      Status `/private/var/folders/g6/w_cfz8c507j6mbt5mdv7wt_h0000gn/T/jl_b8DcZa/Project.toml`
  [7073ff75] IJulia v1.23.2
  [f2c0e409] IJuliaPkgTest v0.0.0 `~/projects/testing/IJuliaPkgTest`
      Status `/private/var/folders/g6/w_cfz8c507j6mbt5mdv7wt_h0000gn/T/jl_b8DcZa/Manifest.toml`
  [8f4d0f93] Conda v1.6.0
  [7073ff75] IJulia v1.23.2
  [f2c0e409] IJuliaPkgTest v0.0.0 `~/projects/testing/IJuliaPkgTest`
  [692b3bcd] JLLWrappers v1.3.0
  [682c06a0] JSON v0.21.2
  [739be429] MbedTLS v1.0.3
  [69de0a69] Parsers v2.1.3
  [21216c6a] Preferences v1.2.3
  [b85f4697] SoftGlobalScope v1.1.0
  [81def892] VersionParsing v1.2.1
  [c2297ded] ZMQ v1.2.1
  [8f1865be] ZeroMQ_jll v4.3.4+0
  [a9144af2] libsodium_jll v1.0.20+0
  [0dad84c5] ArgTools `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [f43a241f] Downloads `@stdlib/Downloads`
  [7b1f6079] FileWatching `@stdlib/FileWatching`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [b27032c2] LibCURL `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [a63ad114] Mmap `@stdlib/Mmap`
  [ca575930] NetworkOptions `@stdlib/NetworkOptions`
  [44cfe95a] Pkg `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [fa267f1f] TOML `@stdlib/TOML`
  [a4e569a6] Tar `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [deac9b47] LibCURL_jll `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll `@stdlib/MozillaCACerts_jll`
  [83775a58] Zlib_jll `@stdlib/Zlib_jll`
  [8e850ede] nghttp2_jll `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll `@stdlib/p7zip_jll`
     Testing Running tests...
[NbConvertApp] Searching ['/Users/bencottier/projects/testing/IJuliaPkgTest/test', '/Users/bencottier/.jupyter', '/Users/bencottier/.julia/conda/3/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[NbConvertApp] Looking for jupyter_config in /etc/jupyter
[NbConvertApp] Looking for jupyter_config in /usr/local/etc/jupyter
[NbConvertApp] Looking for jupyter_config in /Users/bencottier/.julia/conda/3/etc/jupyter
[NbConvertApp] Looking for jupyter_config in /Users/bencottier/.jupyter
[NbConvertApp] Looking for jupyter_config in /Users/bencottier/projects/testing/IJuliaPkgTest/test
[NbConvertApp] Looking for jupyter_nbconvert_config in /etc/jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /usr/local/etc/jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /Users/bencottier/.julia/conda/3/etc/jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /Users/bencottier/.jupyter
[NbConvertApp] Looking for jupyter_nbconvert_config in /Users/bencottier/projects/testing/IJuliaPkgTest/test
[NbConvertApp] Converting notebook ../test.ipynb to notebook
[NbConvertApp] Notebook name is '../test'
[NbConvertApp] Applying preprocessor: ExecutePreprocessor
[NbConvertApp] Starting kernel (async): ['/Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia', '-i', '--color=yes', '--project=@.', '/Users/bencottier/.julia/packages/IJulia/e8kqU/src/kernel.jl', '/private/var/folders/g6/w_cfz8c507j6mbt5mdv7wt_h0000gn/T/tmpkzynyxq_.json']
[NbConvertApp] Connecting to: tcp://127.0.0.1:65133
[NbConvertApp] connecting shell channel to tcp://127.0.0.1:65130
[NbConvertApp] Connecting to: tcp://127.0.0.1:65130
[NbConvertApp] connecting iopub channel to tcp://127.0.0.1:65131
[NbConvertApp] Connecting to: tcp://127.0.0.1:65131
[NbConvertApp] connecting stdin channel to tcp://127.0.0.1:65132
[NbConvertApp] Connecting to: tcp://127.0.0.1:65132
[NbConvertApp] connecting heartbeat channel to tcp://127.0.0.1:65134
[NbConvertApp] connecting control channel to tcp://127.0.0.1:65133
[NbConvertApp] Connecting to: tcp://127.0.0.1:65133
ERROR: LoadError: ArgumentError: Package Revise not found in current path:
- Run `import Pkg; Pkg.add("Revise")` to install the Revise package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:893
in expression starting at /Users/bencottier/.julia/config/startup.jl:1
Traceback (most recent call last):
  File "/Users/bencottier/.julia/conda/3/bin/jupyter-nbconvert", line 11, in <module>
    sys.exit(main())
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/jupyter_core/application.py", line 254, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/traitlets/config/application.py", line 845, in launch_instance
    app.start()
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 350, in start
    self.convert_notebooks()
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 524, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 489, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 418, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 318, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbconvert/preprocessors/execute.py", line 79, in preprocess
    self.execute()
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/client.py", line 524, in async_execute
    async with self.async_setup_kernel(**kwargs):
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/async_generator/_util.py", line 34, in __aenter__
    return await self._agen.asend(None)
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/client.py", line 483, in async_setup_kernel
    await self.async_start_new_kernel_client()
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/client.py", line 410, in async_start_new_kernel_client
    await ensure_async(self.kc.wait_for_ready(timeout=self.startup_timeout))
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/util.py", line 85, in ensure_async
    result = await obj
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/jupyter_client/asynchronous/client.py", line 139, in wait_for_ready
    raise RuntimeError('Kernel died before replying to kernel_info')
RuntimeError: Kernel died before replying to kernel_info
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Users/bencottier/.julia/conda/3/lib/python3.8/site-packages/nbclient/client.py", line 351, in _async_cleanup_kernel
    assert self.km is not None
AssertionError
ERROR: LoadError: failed process: Process(`jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --execute ../test.ipynb --debug --output ../test.ipynb`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:440
 [3] run(::Cmd)
   @ Base ./process.jl:438
 [4] top-level scope
   @ ~/projects/testing/IJuliaPkgTest/test/runtests.jl:4
 [5] include(fname::String)
   @ Base.MainInclude ./client.jl:444
 [6] top-level scope
   @ none:6
in expression starting at /Users/bencottier/projects/testing/IJuliaPkgTest/test/runtests.jl:4
ERROR: Package IJuliaPkgTest errored during testing

My guess is this has something to do with Pkg.test running julia with --startup-file=no and the IJulia kernel running with the default --startup-file=yes. The Pkg.test command looks like this:

/Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia -Cnative -J/Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib --depwarn=yes --check-bounds=yes -g1 --color=yes --startup-file=no --project=/var/folders/g6/w_cfz8c507j6mbt5mdv7wt_h0000gn/T/jl_GYXYn9 -e <code>

But it's not just that, because this seems to run OK:

/Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia -Cnative -J/Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib --depwarn=yes --check-bounds=yes -g1 --color=yes --startup-file=no --project=/var/folders/g6/w_cfz8c507j6mbt5mdv7wt_h0000gn/T/jl_GYXYn9 -e "run(`/Applications/Julia-1.6.app/Contents/Resources/julia/bin/julia -i --color=yes --project=@. /Users/bencottier/.julia/packages/IJulia/e8kqU/src/kernel.jl`)"

versioninfo

julia> versioninfo()
Julia Version 1.6.2
Commit 1b93d53fc4 (2021-07-14 15:36 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
Environment:
  JULIA_PKG_DEVDIR = /Users/bencottier/JuliaEnvs
  JULIA_EDITOR = code

bencottier avatar Jan 06 '22 08:01 bencottier

I think the reason it doesn't work with Pkg.test in particular is that Pkg.test ultimately runs this to remove the julia base environment from LOAD_PATH.

The reason I made an issue here is that I'd expect the IJulia kernel to inherit the --startup-file option from any parent process, unless there is particular reason not to. This kind of thing can be done using Base.julia_cmd(). But I guess there are special reasons for overriding other julia options, like -i?

bencottier avatar Jan 06 '22 12:01 bencottier

From the git blame of https://github.com/JuliaLang/IJulia.jl/blob/8710cd4bc1adabfacb8d9caec56332e36daa1f90/deps/kspec.jl#L97 I found https://github.com/JuliaLang/Pkg.jl/issues/1815/ which is a reason to not specify --startup-file 😕

bencottier avatar Jan 06 '22 13:01 bencottier

Ah, I now realise the command to start the kernel is normally set one time in installkernel, so that makes it difficult to change the startup-file option.

bencottier avatar Jan 07 '22 08:01 bencottier

@bencottier, are you able to nbconvert within the Julia REPL?

phillip-buelow avatar Jul 19 '22 19:07 phillip-buelow

IIRC, yes I was able to nbconvert within the Julia REPL. (Sorry, I'm no longer working with Julia so this issue is no longer relevant to me.)

bencottier avatar Sep 08 '22 14:09 bencottier