npm-related build issue

[adamjstewart]

trafficstars

Description

I'm encountering issues when building jupyterlab-widgets from source. I'm not sure if this is the right repo to report this to, but https://pypi.org/project/jupyterlab-widgets/ lists this as the homepage.

Reproduce

1. Install all dependencies
2. Download https://files.pythonhosted.org/packages/27/f1/0d3a09c0069467ed9636faf3a9b5b758378216bb891066a0bc24799681e3/jupyterlab_widgets-1.0.2.tar.gz
3. Extract the tarball and enter the directory
4. Run pip install .

It may also be possible to reproduce using pip install --no-binary jupyterlab-widgets but I haven't tried this yet.

During installation, jupyter-packaging seems to be using npm to download random dependencies from the internet (hopefully verifying those checksums) and fails to install one of them:

  running bdist_wheel
  running jsdeps
  Installing build dependencies with npm.  This may take a while...
  > jlpm install
  yarn install v1.21.1
  info No lockfile found.
  [1/4] Resolving packages...
...
  $ jlpm run clean && jlpm run build:prod
  yarn run v1.21.1
  $ jlpm run clean:lib
  $ rimraf lib tsconfig.tsbuildinfo
  Done in 1.34s.
  yarn run v1.21.1
  $ jlpm run build:lib && jlpm run build:labextension
  $ tsc
  tsconfig.json(2,14): error TS6053: File '../tsconfigbase' not found.
  tsconfig.json(10,5): error TS6053: File '/tmp/adam/spack-stage/spack-stage-py-jupyterlab-widgets-1.0.2-lxlhfv4qa2lkn4sn6sn2odsqgarlp5gd/packages/base' not found.
  tsconfig.json(13,5): error TS6053: File '/tmp/adam/spack-stage/spack-stage-py-jupyterlab-widgets-1.0.2-lxlhfv4qa2lkn4sn6sn2odsqgarlp5gd/packages/controls' not found.
  tsconfig.json(16,5): error TS6053: File '/tmp/adam/spack-stage/spack-stage-py-jupyterlab-widgets-1.0.2-lxlhfv4qa2lkn4sn6sn2odsqgarlp5gd/packages/output' not found.
  error Command failed with exit code 2.
  info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
  error Command failed with exit code 2.
  info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
  error Command failed with exit code 2.
  info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

Expected behavior

I would expect to be able to build jupyterlab-widgets without having to install npm, without npm downloading random dependencies from the internet, and without those npm installations failing.

Context

• ipywidgets version: jupyterlab-widgets 1.0.2
• Operating System and version: Ubuntu 18.04
• Browser and version: N/A

[vidartf]

There are multiple points in this issue to address, but it is probably best to start with the expectations: You are saying that you want to do a source build of the package, but also saying that you do not expect the build step to require the build tools needed for building, or download the dependencies needed for building. Could you please elaborate on why you are doing a source build instead of an installation from a wheel ?

[adamjstewart]

@vidartf Sorry, let me clarify. I'm a developer for the Spack package manager. Spack is designed for supercomputers and obscure architectures like BlueGene-Q, ARM, RISC-V, Cray, etc. We try to build everything from source whenever possible to avoid OS compatibility issues and allow the user to specify their preferred compiler.

You are saying that you want to do a source build of the package, but also saying that you do not expect the build step to require the build tools needed for building, or download the dependencies needed for building.

This is more of a hope than an expectation. Many packages in a similar situation will use npm during creation of the sdist and wheel so that the release tarball contains all necessary JavaScript files and only requires Python and Python deps to build and install. In our case, we support a lot of air-gapped clusters with no internet access, so having npm downloading dependencies is problematic. We also have strict security requirements and try to checksum all downloads.

Could you please elaborate on why you are doing a source build instead of an installation from a wheel ?

In the specific case of jupyterlab_widgets, since the wheel is not OS-specific and the library is pure-Python, we could install from the wheel, but we would still like the ability to install from source if possible.

[vidartf]

@adamjstewart Thanks for clarifying. I'll try to bring it up during the discussions for the next release, and see what the mood is for/against. That said, we cannot go back and patch this into previous releases.

[jasongrout]

The build assumes it is building from this repo (for example, the typescript configuration inherits from a base configuration at the root of this repo). So if you want to build from source, I'd suggest downloading this repo, then basically following the steps in the release process documented at https://github.com/jupyter-widgets/ipywidgets/blob/master/docs/source/dev_release.md#jupyterlab_widgets:

yarn install # installs npm packages
yarn run build # builds all javascript packages
cd python/jupyterlab_widgets
python -m build

then copy the tarball out of the dist/ directory.

[jasongrout]

Many packages in a similar situation will use npm during creation of the sdist and wheel so that the release tarball contains all necessary JavaScript files and only requires Python and Python deps to build and install. In our case, we support a lot of air-gapped clusters with no internet access, so having npm downloading dependencies is problematic. We also have strict security requirements and try to checksum all downloads.

We do the same (with yarn instead of npm) - the python package includes the bundled javascript files.

[jasongrout]

It looks like our autodetection of whether the build should happen or not is missing now on master, so it will always try to rebuild the javascript. You see remnants of our crude check here, for example: https://github.com/jupyter-widgets/ipywidgets/blob/045cb0eeb21643373ddbd49a2a12acaad8cbb900/python/jupyterlab_widgets/setup.py#L16 (the build is only supposed to happen if we are in the git repo and we don't have the necessary source files).

[jasongrout]

Oh, it looks like that logic got moved to here: https://github.com/jupyter-widgets/ipywidgets/blob/045cb0eeb21643373ddbd49a2a12acaad8cbb900/python/jupyterlab_widgets/pyproject.toml#L6-L7

[adamjstewart]

@jasongrout thanks, it sounds like we just need to fix the autodetection for whether or not to build the javascript components. If we can figure out where to fix this, I can add patches to previous releases in Spack so that everything builds as expected.

[vidartf]

It might be that adding the option --skip-npm will do what you want in the meanwhile. The option is meant to go to the "build" command, so if you are using pip you can probably pass it with --global-option or --install-option (from memory, you would need to confirm the actual spelling).

[adamjstewart]

@vidartf tried both --global-option and --install-option but the build still tries to run jlpm install.