rattler-build Building from a local sdist file url broken in 0.21.0

Support for local source file url scheme was added in https://github.com/prefix-dev/rattler-build/pull/177 and working in version 0.5.0.

I hadn't tested that in a while. When trying to build a recipe using a local file as source with rattler-build 0.21.0, it fails.

Issue can be reproduced with:

context:
  version: "13.4.2"

package:
  name: "rich"
  version: ${{ version }}

source:
  - url: file:///tmp/rich/rich-13.4.2.tar.gz
    sha256: d653d6bccede5844304c605d5aac802c7cf9621efd700b46c7ec2b51ea914898

build:
  # Thanks to `noarch: python` this package works on all platforms
  noarch: python
  script:
    - python -m pip install . -vv --no-deps --no-build-isolation

requirements:
  host:
    - pip
    - poetry-core >=1.0.0
    - python 3.10
  run:
    # sync with normalized deps from poetry-generated setup.py
    - markdown-it-py >=2.2.0
    - pygments >=2.13.0,<3.0.0
    - python 3.10
    - typing_extensions >=4.0.0,<5.0.0

tests:
  - python:
      imports:
        - rich
      pip_check: true

about:
  homepage: https://github.com/Textualize/rich
  license: MIT
  license_file: LICENSE
  summary: Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal
  description: |
    Rich is a Python library for rich text and beautiful formatting in the terminal.

    The Rich API makes it easy to add color and style to terminal output. Rich
    can also render pretty tables, progress bars, markdown, syntax highlighted
    source code, tracebacks, and more — out of the box.
  documentation: https://rich.readthedocs.io
  repository: https://github.com/Textualize/rich

$ rattler-build build
...
 ╭─ Running build for recipe: rich-13.4.2-pyh4616a5c_0
 │
 │ ╭─ Fetching source code
 │ │ Validated SHA256 values of the downloaded file!
 │ │ Using local source file.
 │ │ Copying source from url: "/tmp/rich/rich-13.4.2.tar.gz" to "/tmp/rich/output/bld/rattler-build_rich_1725435886/work"
...
 │ ╭─ Running build script
 │ │ + python -m pip install . -vv --no-deps --no-build-isolation
 │ │ Using pip 24.2 from $PREFIX/lib/python3.10/site-packages/pip (python 3.10)
 │ │ Non-user install because user site-packages disabled
 │ │ Ignoring indexes: https://pypi.org/simple
 │ │ Created temporary directory: /tmp/pip-build-tracker-l3rer2po
 │ │ Initialized build tracking at /tmp/pip-build-tracker-l3rer2po
 │ │ Created build tracker: /tmp/pip-build-tracker-l3rer2po
 │ │ Entered build tracker: /tmp/pip-build-tracker-l3rer2po
 │ │ Created temporary directory: /tmp/pip-install-149wgmke
 │ │ ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
...

$ ls /tmp/rich/output/bld/rattler-build_rich_1725435886/work
build_env.sh  conda_build.sh  rich-13.4.2.tar.gz

The local file was copied to the work directory but wasn't unarchived.

Sep 04 '24 07:09 beenje

Thanks! There are a few workarounds, of course (e.g. making pip unarchive the file). I would also be interested if path: /tmp/rich-.tar.gz works differently?

Lastly, I do think you are right and this file should be un-archived to have the same behavior as fetching from a URL.

Sep 04 '24 08:09 wolfv

It's working with path: :-)

 ╭─ Running build for recipe: rich-13.4.2-pyh4616a5c_0
 │
 │ ╭─ Fetching source code
 │ │ Fetching source from path: "/tmp/rich/rich-13.4.2.tar.gz"
 │ │ Extracted to "/tmp/rich/output/bld/rattler-build_rich_1725441621/work"
 │ │
 │ ╰─────────────────── (took 0 seconds)

We can see in the logs that it is extracted.

Using path: instead of url: file:// is fine for me.

Would still be nice to fix the file url behaviour as you said.

Sep 04 '24 09:09 beenje

Same here, but unfortunately path: also fails:

An url: key followed by a file:// URL fails to build:

source:
  url: file:///path/to/matlab-runtime/MATLAB_Runtime_R2019b_Update_9_glnxa64.zip

rattler-build just copies the ZIP file, does not unarchive it

 │ ╭─ Fetching source code
 │ │ Validated SHA256 values of the downloaded file!
 │ │ Using local source file.
 │ │ Copying source from url: "/path/to//matlab-runtime/MATLAB_Runtime_R2019b_Update_9_glnxa64.zip" to "/tmp/channel/bld
 │ │ /rattler-build_matlab-runtime_1728648393/work"
 │ │
 │ ╰─────────────────── (took 69 seconds)

An url: key followed by an https:// URL works just fine:

source:
  url: https://ssd.mathworks.com/supportfiles/downloads/R2019b/Release/9/deployment_files/installer/complete/glnxa64/MATLAB_Runtime_R2019b_Update_9_glnxa64.zip

rattler-build unarchives the ZIP file

 │ ╭─ Fetching source code
 │ │ Validated SHA256 values of the downloaded file!
 │ │ Found valid source cache file.
 │ │ Using extracted directory from cache: "/tmp/channel/src_cache/MATLAB_Runtime_R2019b_Update_9_glnxa64_d213e296"
 │ │ Copying source from url: "/tmp/channel/src_cache/MATLAB_Runtime_R2019b_Update_9_glnxa64_d213e296" to "/tmp/channel/bld/rattler-
 │ │ build_matlab-runtime_1728648935/work"
 │ │
 │ ╰─────────────────── (took 32 seconds)

A path: key initially seemed to work equally fine, but rattler-build keeps unarchiving forever:

  source:
    path: /path/to/matlab-runtime/MATLAB_Runtime_R2019b_Update_9_glnxa64.zip

rattler-build attempts to unarchive the ZIP file, but unzipping lasts forever...

 │ ╭─ Fetching source code
 │ │ Fetching source from path: "/path/to/matlab-runtime/MATLAB_Runtime_R2019b_Update_9_glnxa64.zip"
 │ │ ⠤ Extracting zip       [00:04:53] [━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╾──────] 2.16 GiB @ 7.56 MiB/s

The issue ~~is probably~~ might be that rattler-build is unable to handle ZIP files larger than 2 GB:

$ ls -lh /path/to/matlab-runtime/MATLAB_Runtime_R2019b_Update_9_glnxa64.zip
-rwxrwx---+ 1 username nogroup 2.6G Aug 12  2021 /path/to/matlab-runtime/MATLAB_Runtime_R2019b_Update_9_glnxa64.zip
$

Oct 11 '24 12:10 DimitriPapadopoulos

@DimitriPapadopoulos thank you for the detailed write-up!

~~4:53 doesn't sound like forever to me. Also the indicator is still going at 7.50 MiB/s. I am wondering if it's just slow? Do you have a reference for how long it should take to extract?~~

Ah, I see that in the URL case it takes only 30 seconds so something is wrong. I'll have to take a look.

Oct 11 '24 13:10 wolfv

While /path/to is indeed on a network (NFS) share, our workstations have 1 Gb/s network interfaces and our storage infrastructure is a CephFS cluster with quite decent throughput:

$ rsync --progress /path/to/matlab-runtime/MATLAB_Runtime_R2019b_Update_9_glnxa64.zip /tmp/
MATLAB_Runtime_R2019b_Update_9_glnxa64.zip
  2,786,688,287 100%  448.76MB/s    0:00:05 (xfr#1, to-chk=0/1)
$