tarp
tarp copied to clipboard
Go error for a very large tar file
I tried to split a 2.3TB tar file into many 1GB tar files. With tarp I got this error after 55 shards:
panic: runtime error: index out of range [1] with length 0
goroutine 19 [running]:
github.com/tmbdev/tarp/dpipes.FnameSplit({0xc0002201e0?, 0xc0005dc030?})
/root/tarp/dpipes/rawtario.go:36 +0xd2
github.com/tmbdev/tarp/dpipes.Aggregate(0x0?, 0xc0003ec060)
/root/tarp/dpipes/rawtario.go:55 +0x137
created by github.com/tmbdev/tarp/dpipes.TarSource.func1
/root/tarp/dpipes/tario.go:55 +0x9e
At first I thought the tar is damaged but trying the old tarproc in python (installed with pip install tarproc
) I was able to split the big tar. The speed dropped from 32 shards per minute to 18 shards per minute but I am happy my problem is solved.
Thanks; I'll try to reproduce it. It looks like there may be an unexpected filename in there somewhere. The Python code is a bit more forgiving.