tarp
tarp copied to clipboard
webdataset
Fast and simple stream processing of files in tar files, useful for deep learning, big data, and many other applications.
Installation instructions need some updating. On a fresh EC2 machine w/ Ubuntu or Amazon Linux--- ## installation method 1 `go get -v github.com/tmbdev/tarp/tarp` => ``` go: go.mod file not found...
The `go install` tool complains about the replace directive in `go.mod`: ```console # go install github.com/webdataset/tarp/tarp@latest go: downloading github.com/webdataset/tarp v0.0.2 go: downloading github.com/webdataset/tarp/tarp v0.0.0-20221009163818-4aac5677b928 go: github.com/webdataset/tarp/tarp@latest (in github.com/webdataset/tarp/[email protected]): The go.mod...
I tried to split a 2.3TB tar file into many 1GB tar files. With tarp I got this error after 55 shards: ```[progress] # shard /scratch/dejavutxt-000056.tar panic: runtime error: index...
I think Docker support might be helpful - especially since many Deep Learning practitioners using this might not be familiar with the Go tool chain. Imagining an executable image like...
Hello, I encounter the following error when trying to install tarp ``` webdataset/tarp/tarp/sort.go:44:17: undefined: msgpack.Encode webdataset/tarp/tarp/sort.go:83:6: undefined: msgpack.Decode ``` I tried both `go get -v github.com/webdataset/tarp/tarp` and `go get -v...
This repo does not contain any information about licensing. Can you please add the relevant information?
Hello, I have two issues/questions with the `tarp sort` command: ### No output produced When running: ``` tarp sort data/* -f rgb.png -o - | tarp split -c 512 -...
There where 4 problems: - go-tfdata is private - go-tfdata is on gitlab-master.nvidia.com, not github.com/NVIDIA/go-tfdata - go-tfdata is on gitlab-master.nvidia.com, but declares itself as github.com/NVIDIA/go-tfdata - `go mod tidy` doesn't...
Metadata
Owner
Metadata
Fast and simple stream processing of files in tar files, useful for deep learning, big data, and many other applications.