Problem installing polars_1xx branch, possible issue through pybigtools [BUG]

[ckv2115147]

I want to install the polars_1xx branch to process a large dataset >100k cells, but am having problems installing pycisTopic. It seems the issue may be from installing pybigtools.

I followed these steps to install the polars_1xx branch in a conda env:

$ conda create --name pycistopicpolars python=3.11
$ conda install -n pycistopicpolars -c conda-forge rust=1.82.0
$ git clone https://github.com/aertslab/pycisTopic.git
$ git checkout polars_1xx
$ pip install .

But got this error:

  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting numpy<2,>=1.20.3 (from pycisTopic==2.0a0)
  Using cached numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Collecting pandas==1.5 (from pycisTopic==2.0a0)  
  Using cached pandas-1.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
Collecting polars>=1 (from pycisTopic==2.0a0)
  Using cached polars-1.18.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (14 kB)
Collecting pyarrow>=8.0.0 (from pycisTopic==2.0a0)
  Using cached pyarrow-18.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.3 kB)
Collecting pyranges<0.0.128 (from pycisTopic==2.0a0)
  Using cached pyranges-0.0.127-py3-none-any.whl.metadata (15 kB)
Collecting numba (from pycisTopic==2.0a0)
  Using cached numba-0.60.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.7 kB)
Collecting ray (from pycisTopic==2.0a0)
  Using cached ray-2.40.0-cp311-cp311-manylinux2014_x86_64.whl.metadata (17 kB)
Collecting scatac_fragment_tools<0.2.0 (from pycisTopic==2.0a0)
  Using cached scatac_fragment_tools-0.1.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.8 kB)
Collecting scikit-learn (from pycisTopic==2.0a0) 
  Using cached scikit_learn-1.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)
Collecting lda (from pycisTopic==2.0a0)
  Using cached lda-3.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.6 kB)
Collecting matplotlib<3.7 (from pycisTopic==2.0a0)
  Using cached matplotlib-3.6.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.8 kB)
Collecting tmtoolkit (from pycisTopic==2.0a0)
  Using cached tmtoolkit-0.12.0-py3-none-any.whl.metadata (14 kB)
Collecting gensim (from pycisTopic==2.0a0)
  Using cached gensim-4.3.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.1 kB)
Collecting macs2 (from pycisTopic==2.0a0)
  Using cached MACS2-2.2.9.1-cp311-cp311-linux_x86_64.whl
Collecting pybiomart (from pycisTopic==2.0a0)
  Using cached pybiomart-0.2.0-py3-none-any.whl.metadata (1.1 kB)
Collecting pyscenic (from pycisTopic==2.0a0)
  Using cached pyscenic-0.12.1-py3-none-any.whl.metadata (9.8 kB)
Collecting scrublet (from pycisTopic==2.0a0)
  Using cached scrublet-0.2.3-py3-none-any.whl.metadata (3.1 kB)
Collecting harmonypy (from pycisTopic==2.0a0)
  Using cached harmonypy-0.0.10-py3-none-any.whl.metadata (3.9 kB)
Collecting igraph (from pycisTopic==2.0a0)
  Using cached igraph-0.11.8-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
Collecting leidenalg (from pycisTopic==2.0a0)
  Using cached leidenalg-0.10.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB)
Collecting seaborn (from pycisTopic==2.0a0)
  Using cached seaborn-0.13.2-py3-none-any.whl.metadata (5.4 kB)
Collecting adjustText (from pycisTopic==2.0a0)   
  Using cached adjustText-1.3.0-py3-none-any.whl.metadata (3.1 kB)
Collecting statsmodels (from pycisTopic==2.0a0)  
  Using cached statsmodels-0.14.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.2 kB)
Collecting scanpy (from pycisTopic==2.0a0)
  Using cached scanpy-1.10.4-py3-none-any.whl.metadata (9.3 kB)
Collecting scanorama (from pycisTopic==2.0a0)
  Using cached scanorama-1.7.4-py3-none-any.whl.metadata (525 bytes)
Collecting bbknn (from pycisTopic==2.0a0)
  Using cached bbknn-1.6.0-py3-none-any.whl.metadata (8.2 kB)
Collecting loomxpy (from pycisTopic==2.0a0)
  Using cached loomxpy-0.4.2-py3-none-any.whl.metadata (1.6 kB)
Collecting bs4 (from pycisTopic==2.0a0)
  Using cached bs4-0.0.2-py2.py3-none-any.whl.metadata (411 bytes)
Collecting lxml (from pycisTopic==2.0a0)
  Using cached lxml-5.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
Collecting python-dateutil>=2.8.1 (from pandas==1.5->pycisTopic==2.0a0)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2020.1 (from pandas==1.5->pycisTopic==2.0a0)
  Using cached pytz-2024.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting contourpy>=1.0.1 (from matplotlib<3.7->pycisTopic==2.0a0)
  Using cached contourpy-1.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.4 kB)
Collecting cycler>=0.10 (from matplotlib<3.7->pycisTopic==2.0a0)
  Using cached cycler-0.12.1-py3-none-any.whl.metadata (3.8 kB)
Collecting fonttools>=4.22.0 (from matplotlib<3.7->pycisTopic==2.0a0)
  Using cached fonttools-4.55.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (165 kB)
Collecting kiwisolver>=1.0.1 (from matplotlib<3.7->pycisTopic==2.0a0)
  Using cached kiwisolver-1.4.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB)
Collecting packaging>=20.0 (from matplotlib<3.7->pycisTopic==2.0a0)
  Using cached packaging-24.2-py3-none-any.whl.metadata (3.2 kB)
Collecting pillow>=6.2.0 (from matplotlib<3.7->pycisTopic==2.0a0)
  Using cached pillow-11.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.1 kB)
Collecting pyparsing>=2.2.1 (from matplotlib<3.7->pycisTopic==2.0a0)
  Using cached pyparsing-3.2.0-py3-none-any.whl.metadata (5.0 kB)
Collecting ncls>=0.0.63 (from pyranges<0.0.128->pycisTopic==2.0a0)
  Using cached ncls-0.0.68-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.6 kB)
Collecting tabulate (from pyranges<0.0.128->pycisTopic==2.0a0)
  Using cached tabulate-0.9.0-py3-none-any.whl.metadata (34 kB)
Collecting sorted-nearest>=0.0.33 (from pyranges<0.0.128->pycisTopic==2.0a0)
  Using cached sorted_nearest-0.0.39-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (964 bytes)
Collecting natsort (from pyranges<0.0.128->pycisTopic==2.0a0)
  Using cached natsort-8.4.0-py3-none-any.whl.metadata (21 kB)
Collecting pybigwig (from scatac_fragment_tools<0.2.0->pycisTopic==2.0a0)
  Using cached pyBigWig-0.3.23-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
Collecting pybigtools>=0.2.0 (from scatac_fragment_tools<0.2.0->pycisTopic==2.0a0)
  Using cached pybigtools-0.2.3.tar.gz (9.5 MB)  
  Installing build dependencies ... done
  Getting requirements to build wheel ... done   
  Preparing metadata (pyproject.toml) ... error  
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [22 lines of output]
          Updating crates.io index
           Locking 15 packages to latest compatible versions
            Adding base64 v0.21.7 (latest: v0.22.1)
          Updating bigtools v0.3.1-dev (/work/tmp/pip-install-0jw93rsr/pybigtools_1227f15a78884eeebc41971527c93b01/bigtools) -> v0.5.4
            Adding getrandom v0.2.10 (latest: v0.2.15)
            Adding index_list v0.2.15
          Updating pybigtools v0.1.0 (/work/tmp/pip-install-0jw93rsr/pybigtools_1227f15a78884eeebc41971527c93b01/pybigtools) -> v0.2.3
            Adding ring v0.17.3 (latest: v0.17.8)
            Adding rustls v0.21.12 (latest: v0.23.20)
            Adding rustls-native-certs v0.6.3 (latest: v0.8.1)
            Adding rustls-pemfile v1.0.4 (latest: v2.2.0)
            Adding rustls-webpki v0.101.7 (latest: v0.102.8)
            Adding sct v0.7.1
            Adding spin v0.9.8
            Adding untrusted v0.9.0
            Adding wasi v0.11.0+wasi-snapshot-preview1 (latest: v0.13.3+wasi-0.2.2)
            Adding webpki v0.22.4
      � maturin failed
        Caused by: `project.version` field is required in pyproject.toml unless it is present in the `project.dynamic` list
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/work/tmp/pip-modern-metadata-pnst2ea0', '--interpreter', '/miniconda/envs/pycistopicpolars/bin/python3.11']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /work/tmp/pip-modern-metadata-pnst2ea0 --interpreter /miniconda/envs/pycistopicpolars/bin/python3.11`
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

Version:

• Python: 3.11.10
• gcc 11.3.0

I think the bug is related to installing scatac_fragment_tools, which in turn is trying to install pybigtools, which is where the installation stalls. Would appreciate any help with this!

[ghuls]

Looks like there is a breaking change in maturin downgrade to maturin 1.7.8 for now: https://github.com/PyO3/maturin/issues/2416

[ghuls]

Or you can clone the bigtools repo and apply this patch and install pybigtools from the patched version: https://github.com/jackh726/bigtools/pull/68

Update: Building from bigtools master branch should work now.