bacalhau icon indicating copy to clipboard operation
bacalhau copied to clipboard

Published 20 hours ago • verified publisher icon bacalhau-project

sharding example with 10000 Images fails

Open js-ts opened this issue 2 years ago • 3 comments

Example https://docs.bacalhau.org/getting-started/parallel-workloads

CID: https://bafybeiew7kq43tx77tlkwafb6sep7oszq3g444i2qqlu7nxuedc4ngaapq.ipfs.nftstorage.link/

COMMAND: bacalhau docker run -v bafybeiew7kq43tx77tlkwafb6sep7oszq3g444i2qqlu7nxuedc4ngaapq:/input_images -o results:/output_images --sharding-glob-pattern ISIC_*.jpg --sharding-base-path /input_images/ --sharding-batch-size 100 dpokidov/imagemagick -- -resize 100x100 -quality 100 -path /output_images /input_images/*.jpg gives error image

Possible causes of the issue: the IPLD traversing code that explodes the glob pattern

js-ts avatar Aug 10 '22 07:08 js-ts

Cc @binocarlos - can you take a look?

aronchick avatar Aug 12 '22 15:08 aronchick

@js-ts please try to repro with latest code (not the binary).

aronchick avatar Aug 24 '22 01:08 aronchick

(i suspect this is the trailing slash issue, which I've now fixed in main)

aronchick avatar Aug 24 '22 01:08 aronchick

please verify if this is still a problem

aronchick avatar Nov 21 '22 00:11 aronchick

Yes, since sharding is broken it is also broken https://github.com/filecoin-project/bacalhau/issues/1043

js-ts avatar Nov 21 '22 15:11 js-ts

Sharding has been removed.

simonwo avatar Mar 30 '23 08:03 simonwo