rattler-build icon indicating copy to clipboard operation
rattler-build copied to clipboard

Make searching for `prefix_placeholder` string and make async/parallel (and add progress bar?)

Open wolfv opened this issue 1 year ago • 2 comments

We are currently searching for the prefix placeholder string in the files one-by-one. It does take a little while for larger files. Would be nice to

  1. make the function async
  2. parallelize it and run over multiple files at once
  3. Or even parallize it for single large files (chunk the file up and run multiple threads)

wolfv avatar Feb 12 '24 12:02 wolfv

Example of a big file package:

package:
  name: big
  version: "0.1.0"

build:
  # prefix_detection:
  #   ignore_binary_files: true
  script:
    - dd if=/dev/urandom of=$PREFIX/bigfile bs=1M count=1000

wolfv avatar Feb 12 '24 12:02 wolfv

It would also be good to only mmap the files instead of loading them in memory!

wolfv avatar Feb 12 '24 13:02 wolfv

We can close this for now, I made a bunch of performance improvements.

wolfv avatar Oct 10 '24 11:10 wolfv