tarp icon indicating copy to clipboard operation
tarp copied to clipboard

Go error for a very large tar file

Open rvencu opened this issue 2 years ago • 1 comments

I tried to split a 2.3TB tar file into many 1GB tar files. With tarp I got this error after 55 shards:

panic: runtime error: index out of range [1] with length 0

goroutine 19 [running]:
github.com/tmbdev/tarp/dpipes.FnameSplit({0xc0002201e0?, 0xc0005dc030?})
        /root/tarp/dpipes/rawtario.go:36 +0xd2
github.com/tmbdev/tarp/dpipes.Aggregate(0x0?, 0xc0003ec060)
        /root/tarp/dpipes/rawtario.go:55 +0x137
created by github.com/tmbdev/tarp/dpipes.TarSource.func1
        /root/tarp/dpipes/tario.go:55 +0x9e

At first I thought the tar is damaged but trying the old tarproc in python (installed with pip install tarproc) I was able to split the big tar. The speed dropped from 32 shards per minute to 18 shards per minute but I am happy my problem is solved.

rvencu avatar Dec 29 '22 15:12 rvencu

Thanks; I'll try to reproduce it. It looks like there may be an unexpected filename in there somewhere. The Python code is a bit more forgiving.

tmbdev avatar Aug 02 '23 08:08 tmbdev