Can't finish build

[tpike3]

Description

I am integrating jupyterlite-sphinx into mesa-geo readthedocs but my build keeps failing with the following error on in read the docs (no issues if doing jupyter lite build on my local machine)

Extension error (jupyterlite_sphinx.jupyterlite_sphinx):
--
94 | Handler <function jupyterlite_build at 0x7f477f0ddab0> for event 'build-finished' threw an exception (exception: Command '['jupyter', 'lite', 'build', '--debug', '--contents', 'tutorials/intro_tutorial.ipynb', '--contents', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/docs/source/_contents', '--output-dir', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/_readthedocs/html/lite', '--apps', 'notebooks', '--apps', 'edit', '--apps', 'lab', '--apps', 'repl', '--apps', 'tree', '--apps', 'consoles', '--lite-dir', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/docs/source', '--XeusAddon.environment_file', 'environment.yml']' returned non-zero exit status 3.)
95 | [jupyterlite-sphinx] Running JupyterLite build

Reproduce

You can see the build at ----

https://app.readthedocs.org/projects/mesa-geo-test/builds/24708569/

Expected behavior

Mesa-Geo docs render with xeus-python kernel

Context

• JupyterLite version: jupyterlite-xeus>0.1.5
• Operating System and version: "ubuntu-22.04"

Thank you for your time. Attached are my conf.txt readthedocs.txt and environment.txt

I feel like this is going to be something very simple but just cant figure out the right combination

[jtpio]

Thanks @tpike3 for reporting :+1:

It may be related to jupyterlite-xeus. There were a few issues recently, maybe @martinRenou or @DerThorsten may no more about this.

In the meantime, you could try to use a different kernel (for example https://github.com/jupyterlite/pyodide-kernel), to see if the build passes.

[tpike3]

Thanks @jtpio will do.

It may also be the directory the build is in. The way I can replicate that error on my local machine if the output directory (jupyterlite_dir) doesn't match.

[martinRenou]

Your logs don't contain much, it looks like the actual error message is being hidden (we don't know why the jupyterlite build command failed)

[tpike3]

Your logs don't contain much, it looks like the actual error message is being hidden (we don't know why the jupyterlite build command failed) Thanks @martinRenou for the help and interest!

Here is the traceback from the readthedocs build ....

writing additional pages... search done
dumping search index in English (code: en)... done
dumping object inventory... done
Updating searchtools for Read the Docs search... 
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/conda/latest/lib/python3.10/site-packages/sphinx/events.py", line 97, in emit
    results.append(listener.handler(self.app, *args))
  File "/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/conda/latest/lib/python3.10/site-packages/jupyterlite_sphinx/jupyterlite_sphinx.py", line 707, in jupyterlite_build
    completed_process: CompletedProcess[bytes] = subprocess.run(
  File "/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/conda/latest/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['jupyter', 'lite', 'build', '--debug', '--contents', 'tutorials/intro_tutorial.ipynb', '--contents', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/docs/source/_contents', '--output-dir', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/_readthedocs/html/lite', '--apps', 'notebooks', '--apps', 'edit', '--apps', 'lab', '--apps', 'repl', '--apps', 'tree', '--apps', 'consoles', '--lite-dir', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/docs/source']' returned non-zero exit status 3.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/conda/latest/lib/python3.10/site-packages/sphinx/cmd/build.py", line 337, in build_main
    app.build(args.force_all, args.filenames)
  File "/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/conda/latest/lib/python3.10/site-packages/sphinx/application.py", line 353, in build
    self.events.emit('build-finished', None)
  File "/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/conda/latest/lib/python3.10/site-packages/sphinx/events.py", line 108, in emit
    raise ExtensionError(
sphinx.errors.ExtensionError: Handler <function jupyterlite_build at 0x7effb8c75360> for event 'build-finished' threw an exception (exception: Command '['jupyter', 'lite', 'build', '--debug', '--contents', 'tutorials/intro_tutorial.ipynb', '--contents', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/docs/source/_contents', '--output-dir', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/_readthedocs/html/lite', '--apps', 'notebooks', '--apps', 'edit', '--apps', 'lab', '--apps', 'repl', '--apps', 'tree', '--apps', 'consoles', '--lite-dir', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/docs/source']' returned non-zero exit status 3.)

Extension error (jupyterlite_sphinx.jupyterlite_sphinx):
Handler <function jupyterlite_build at 0x7effb8c75360> for event 'build-finished' threw an exception (exception: Command '['jupyter', 'lite', 'build', '--debug', '--contents', 'tutorials/intro_tutorial.ipynb', '--contents', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/docs/source/_contents', '--output-dir', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/_readthedocs/html/lite', '--apps', 'notebooks', '--apps', 'edit', '--apps', 'lab', '--apps', 'repl', '--apps', 'tree', '--apps', 'consoles', '--lite-dir', '/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/docs/source']' returned non-zero exit status 3.)
[jupyterlite-sphinx] Running JupyterLite build

Right now I am thinking it is one of 2 things (but could be way off)

1. That it the way my directories are set up as I am using a source directory and a build directory, but even when I change structure to match jupyterlite sphinx its doesnt work and my .jupyterlite.doit.db never ends up where it should be and so never gets cleaned as it should here https://github.com/jupyterlite/jupyterlite-sphinx/blob/d50e2465d5bf67cb03897f68abc9a827658fae69/jupyterlite_sphinx/jupyterlite_sphinx.py#L731

2. Or that need to build an environment in my pyproject.toml like jupyterlite-sphinx does here https://github.com/jupyterlite/jupyterlite-sphinx/blob/d50e2465d5bf67cb03897f68abc9a827658fae69/pyproject.toml#L43C1-L47C60

[tpike3]

Well I should pay closer attention to the documentation I set jupyterlite_silence = False and I am getting good traceback information.

However, it is not immediately obvious to me what the issue is.

RTD build details.txt

[jtpio]

Thanks @tpike3.

These two lines at the top of the logs seem to indicate an issue with jupyterlite-xeus:

ValidationError(ValueError("'emscripten' is not a valid Platform"))
ValidationError(ValueError("'emscripten' is not a valid Platform"))

Which should likely be fixed by https://github.com/conda/conda/pull/13962?

In the meantime, I think the workaround might be to use the latest pre-release of jupyterlite-xeus, and add micromamba to the build environment, similar to https://github.com/voila-dashboards/voici/pull/120 which had the same issue.

[jtpio]

@tpike3 looks like this is indeed an issue with jupyterlite-xeus: https://github.com/jupyterlite/jupyterlite-sphinx/pull/179

@martinRenou @DerThorsten maybe we should document the fix somewhere (for example in the jupyterlite-xeus repo) until it is fixed in conda?

[tpike3]

Thanks @jtpio!!! I am getting closer but now I am getting this error--- error libmamba ZSTD decompression error: Unknown frame descriptor

[tpike3]

Thanks @tpike3 for reporting 👍

It may be related to jupyterlite-xeus. There were a few issues recently, maybe @martinRenou or @DerThorsten may no more about this.

In the meantime, you could try to use a different kernel (for example https://github.com/jupyterlite/pyodide-kernel), to see if the build passes.

@jtpio Apologies should have done this earlier, I did switch to a pyodide-kernel and the build works. Working to see how to pre-install packages now.

[jtpio]

@tpike3 you could also continue using jupyterlite-xeus but update to the latest pre-release (as in https://github.com/jupyterlite/jupyterlite-sphinx/pull/179), which should fix the issue.

[tpike3]

@tpike3 you could also continue using jupyterlite-xeus but update to the latest pre-release (as in #179), which should fix the issue.

@jtpio I still cannot get the xeus-python to build into RTD.

[LiteBuildApp] creating folder /home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/checkouts/latest/_readthedocs/html/lite/static

139 | error    libmamba ZSTD decompression error: Unknown frame descriptor

Of course I realized pyodide won't work with piplite as Mesa doesn't have a pure python wheel

[jtpio]

Normally this issue should not be related to Pyodide since you are using xeus-python, and also because this is happening a build time (when the JupyterLite site is built) and not at runtime (when opening JupyterLite in a web browser).

Looks like this could be due to a malformed package on emscripten forge or conda forge?

[martinRenou]

error libmamba ZSTD decompression error: Unknown frame descriptor

I believe this should not cause the build to fail. You probably have another error somewhere.

[reox]

In the meantime, you could try to use a different kernel (for example https://github.com/jupyterlite/pyodide-kernel), to see if the build passes.

I also have a problem with xeus and found this issue. I tried to switch to pyodide and if I do it build. But when I start the notebook it prompts me to select a kernel but no kernel seems to be available:

Sorry for this super basic question but is there any configuration do to, such that pyodide is added as kernel?

[agriyakhetarpal]

Hi @reox, I think you might need to pip-install jupyterlite-pyodide-kernel if you haven't already, and the Pyodide kernel should then show up.

[reox]

@agriyakhetarpal yes, I did that, removed all build files and caches and built again - but no kernel shows up... I also tried with the --pre version, but that just leads to crashes during the build

[agriyakhetarpal]

I see, could you please paste an error trace after setting jupyterlite_silence = False in conf.py and building, as noted above? That might show some extra information

[reox]

Sure:

[jupyterlite-sphinx] Running JupyterLite build
[LiteBuildApp] Searching ['C:\\Users\\User Name\\git\\gdp2024', 'C:\\Users\\User Name\\.jupyter', 'C:\\Users\\User Name\\AppData\\Roaming\\Python\\etc\\jupyter', 'C:\\Program Files\\Python312\\etc\\jupyter', 'C:\\ProgramData\\jupyter'] for config files
[LiteBuildApp] Looking for jupyter_config in C:\ProgramData\jupyter
[LiteBuildApp] Looking for jupyter_config in C:\Program Files\Python312\etc\jupyter
[LiteBuildApp] Looking for jupyter_config in C:\Users\User Name\AppData\Roaming\Python\etc\jupyter
[LiteBuildApp] Looking for jupyter_config in C:\Users\User Name\.jupyter
[LiteBuildApp] Looking for jupyter_config in C:\Users\User Name\git\gdp2024
[LiteBuildApp] Looking for jupyter_lite_config in C:\ProgramData\jupyter
[LiteBuildApp] Looking for jupyter_lite_config in C:\Program Files\Python312\etc\jupyter
[LiteBuildApp] Looking for jupyter_lite_config in C:\Users\User Name\AppData\Roaming\Python\etc\jupyter
[LiteBuildApp] Looking for jupyter_lite_config in C:\Users\User Name\.jupyter
[LiteBuildApp] Looking for jupyter_lite_config in C:\Users\User Name\git\gdp2024
[LiteBuildApp] [lite] [addon] loading ...
[LiteBuildApp] [lite] [addon] [archive] ... will archive
[LiteBuildApp] [lite] [addon] [archive] ... will status
[LiteBuildApp] [lite] [addon] [contents] ... will build
[LiteBuildApp] [lite] [addon] [contents] ... will check
[LiteBuildApp] [lite] [addon] [contents] ... will post_build
[LiteBuildApp] [lite] [addon] [contents] ... will status
[LiteBuildApp] [lite] [addon] [federated_extensions] ... will post_build
[LiteBuildApp] [lite] [addon] [federated_extensions] ... will post_init
[LiteBuildApp] [lite] [addon] [federated_extensions] ... will pre_build
[LiteBuildApp] [lite] [addon] [icons] ... will build
[LiteBuildApp] [lite] [addon] [icons] ... will status
[LiteBuildApp] [lite] [addon] [jupyterlite-pyodide-kernel-piplite] ... will build
[LiteBuildApp] [lite] [addon] [jupyterlite-pyodide-kernel-piplite] ... will check
[LiteBuildApp] [lite] [addon] [jupyterlite-pyodide-kernel-piplite] ... will post_build
[LiteBuildApp] [lite] [addon] [jupyterlite-pyodide-kernel-piplite] ... will post_init
[LiteBuildApp] [lite] [addon] [jupyterlite-pyodide-kernel-pyodide] ... will build
[LiteBuildApp] [lite] [addon] [jupyterlite-pyodide-kernel-pyodide] ... will check
[LiteBuildApp] [lite] [addon] [jupyterlite-pyodide-kernel-pyodide] ... will post_build
[LiteBuildApp] [lite] [addon] [jupyterlite-pyodide-kernel-pyodide] ... will post_init
[LiteBuildApp] [lite] [addon] [jupyterlite-pyodide-kernel-pyodide] ... will status
[LiteBuildApp] [lite] [addon] [lite] ... will build
[LiteBuildApp] [lite] [addon] [lite] ... will check
[LiteBuildApp] [lite] [addon] [lite] ... will status
[LiteBuildApp] [lite] [addon] [mimetypes] ... will post_build
[LiteBuildApp] [lite] [addon] [mimetypes] ... will status
[LiteBuildApp] [lite] [addon] [report] ... will pre_archive
[LiteBuildApp] [lite] [addon] [serve] ... will serve
[LiteBuildApp] [lite] [addon] [serve] ... will status
[LiteBuildApp] [lite] [addon] [settings] ... will check
[LiteBuildApp] [lite] [addon] [settings] ... will post_build
[LiteBuildApp] [lite] [addon] [settings] ... will pre_build
[LiteBuildApp] [lite] [addon] [settings] ... will status
[LiteBuildApp] [lite] [addon] [static] ... will init
[LiteBuildApp] [lite] [addon] [static] ... will post_init
[LiteBuildApp] [lite] [addon] [static] ... will pre_init
[LiteBuildApp] [lite] [addon] [static] ... will pre_status
[LiteBuildApp] [lite] [addon] [translation] ... will build
[LiteBuildApp] [lite] [addon] [translation] ... will check
[LiteBuildApp] [lite] [addon] [translation] ... will status
[LiteBuildApp] [lite] [addon] [voici] ... will post_build
[LiteBuildApp] [lite] [addon] ... OK 14 addons
[LiteBuildApp] [lite] [tasks] loading ...
[LiteBuildApp] [lite] [tasks] ... OK 18 tasks
static:jupyter-lite.json
.  pre_status:static:jupyter-lite.json
    tarball:         jupyterlite-app-0.3.0.tgz 13MB
    output:          C:\Users\User Name\git\gdp2024\public\lite
    lite dir:        C:\Users\User Name\git\gdp2024
    apps:            notebooks, edit, lab, repl, tree, consoles
    sourcemaps:      True
    unused packages: True
archive:archive
contents:contents
icons:icons
jupyterlite-pyodide-kernel-pyodide:pyodide
lite:jupyter-lite.json
mimetypes:jupyter-lite.json
serve:contents
settings:overrides
translation:translation
.  status:archive:archive
[LiteBuildApp] No archive (yet): gdp2024-jupyterlite.tgz
.  status:contents:contents
[LiteBuildApp] [lite] [contents] All Contents ['C:\\Users\\User Name\\git\\gdp2024\\_contents\\Jupyter1.ipynb',
 'C:\\Users\\User '
 'Name\\git\\gdp2024\\_contents\\Jupyter_Beispiel.ipynb',
 'C:\\Users\\User Name\\git\\gdp2024\\src\\Jupyter1_Beispiel1a.png',
 'C:\\Users\\User Name\\git\\gdp2024\\src\\Jupyter1_Beispiel1.png']
    contents: 4 files
.  status:icons:icons
[LiteBuildApp] [lite] [icons] All favicons []
    favicon files: 0 files
.  status:jupyterlite-pyodide-kernel-pyodide:pyodide
     URL: None
 archive: []
   cache: 0 files
   local: 0 files
.  status:lite:jupyter-lite.json
[LiteBuildApp]     jupyter-lite.(json|ipynb): []
[LiteBuildApp]     jupyter-lite.(json|ipynb): 0 files
.  status:mimetypes:jupyter-lite.json
    filetypes:         26
.  status:serve:contents
    url: http://127.0.0.1:8000/
    server: tornado
    headers:
.  status:settings:overrides
    overrides.json: 0
.  status:translation:translation
[LiteBuildApp] [lite] [translation] All Translations []
    translation files: 0 files
static:output_dir
.  pre_init:static:output_dir
static:unpack
.  init:static:unpack
C:\Users\User Name\AppData\Roaming\Python\Python312\site-packages\jupyterlite_core\addons\base.py:294: UserWarning: install libarchive-c for better performance when working with archives: argument of type 'NoneType' is not iterable
  return has_optional_dependency(
contents:copy:Jupyter1.ipynb
contents:copy:Jupyter_Beispiel.ipynb
contents:copy:Jupyter1_Beispiel1.png
contents:copy:Jupyter1_Beispiel1a.png
icons:copy
translation:copy
.  build:contents:copy:Jupyter1.ipynb
[LiteBuildApp] creating folder C:\Users\User Name\git\gdp2024\public\lite\files
.  build:contents:copy:Jupyter_Beispiel.ipynb
.  build:contents:copy:Jupyter1_Beispiel1.png
.  build:contents:copy:Jupyter1_Beispiel1a.png
.  build:icons:copy
[LiteBuildApp] creating folder C:\Users\User Name\git\gdp2024\public\lite\static
.  build:translation:copy
contents:contents:.
federated_extensions:patch
federated_extensions:settings
mimetypes:patch
.  post_build:contents:contents:.
.  post_build:federated_extensions:patch
.  post_build:federated_extensions:settings
.  post_build:mimetypes:patch
[LiteBuildApp] Exiting application: jupyter
[jupyterlite-sphinx] JupyterLite build done

I cannot spot a problem there... Is it possible the problem is actually serving the website? I tried only locally by running python -m http.server in the public folder.

[agriyakhetarpal]

Yes, I can't find a problem either. I don't think actually serving the website would make a difference, though. If it doesn't work with python -m http.server, it will be the same as where you would deploy it. Maybe something to try would be to use a fresh virtual environment, install jupyterlite-sphinx and jupyterlite-pyodide-kernel in it along with other dependencies like sphinx, etc. and see if the problem resurfaces? You might have already tried that, but that's the only suggestion I have right now. Perhaps clearing your browser cache and the user data stored on the page might help as well. From your logs, it does list jupyterlite-pyodide-kernel-pyodide:pyodide, so the kernel is installed indeed.

[reox]

seems to be a problem with my local environment indeed. I just build it with gitlab CI and deployed it and there it works... Guess I have to look around which caches I can remove :D

edit: It seems to be harder than I thought. Every time I update now something via gitlab-CI and deploy it, I would have to clean everything... In the browser where I do not clean, I still get a very old version. Is that supposed to work like that?

[tpike3]

Apologies, I was having to fix some other issues and got sidetracked. However, taking a break and readdressing I realized most of my issues was getting the right paths in my conf.py file. However, I am not getting this error

Exception occurred: -- 93 | File "/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/conda/latest/lib/python3.10/site-packages/jupyter_client/kernelspec.py", line 285, in get_kernel_spec 94 | raise NoSuchKernel(kernel_name)

File "/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/conda/latest/lib/python3.10/site-packages/jupyter_client/kernelspec.py", line 285, in get_kernel_spec -- 89 | raise NoSuchKernel(kernel_name) 90 | jupyter_client.kernelspec.NoSuchKernel: No such kernel named mesa_geo

But I cant find where kernelspec is specified. On my machine I am using the xeus-python kernel. I am printing out the kernelspec on the readthedocs build but it only shows a python3 kernel. I am also searching through my local build and can't find where the kernelspec is.

Thoughts? and thank you!

[tpike3]

I have a brute force solution where I replace the kernel with one called mesa-geo (see below) in my conf.py file. I have also narrowed my problem down to what files are available via the various channels.

As this is a bit larger now, would you like me to close the issue?

def create_new_kernel_directory_and_update_json():
    base_path = Path('/home/docs/checkouts/readthedocs.org/user_builds/mesa-geo-test/conda/latest/share/jupyter/kernels/')
    old_kernel_name = "python3"
    new_kernel_name = "mesa_geo"

    old_kernel_path = base_path / old_kernel_name
    new_kernel_path = base_path / new_kernel_name

    try:
        # Check if the old kernel directory exists
        if old_kernel_path.is_dir():
            # Copy the kernel directory to a new directory
            shutil.copytree(old_kernel_path, new_kernel_path)
            print(f"Copied kernel directory from {old_kernel_name} to {new_kernel_name}")

            # Path to the kernel.json file in the new directory
            kernel_json_path = new_kernel_path / 'kernel.json'

            # Update the kernel.json file
            if kernel_json_path.is_file():
                with open(kernel_json_path) as f:
                    kernel_data = json.load(f)
                    print("Original kernel.json data:", kernel_data)

                # Update the display name
                kernel_data['display_name'] = new_kernel_name
                print("Updated kernel.json data:", kernel_data)

                # Write the changes back to the kernel.json file
                with open(kernel_json_path, 'w') as f:
                    json.dump(kernel_data, f, indent=4)

                print(f"Updated display name in kernel.json to '{new_kernel_name}'")
            else:
                print(f"kernel.json not found at {kernel_json_path}")
        else:
            print(f"Old kernel directory '{old_kernel_name}' not found at {old_kernel_path}")
    except Exception as e:
        print(f"Error during kernel directory copy or update: {e}")

# Call the function to create a new kernel directory and update kernel.json
create_new_kernel_directory_and_update_json()