dbpedia-docs icon indicating copy to clipboard operation
dbpedia-docs copied to clipboard

more efficient re-compression from bzip2 to gzip

Open neradis opened this issue 11 years ago • 0 comments

  • re-compression should be performed by piping the uncompressed data stream from the bzip decompressor directly into gzip, something along the lines of:
for a in *.bz2; do 
    bunzip2 -c "$a" | gzip - > "${a%\.bz2}.gz"; 
done
  • if pbzip2 is available on the machine, using pbzip2 instead of bzip2 can drastically reduce the time necessary for the decompression of bzip2 archives, so it should be used if available:
BZIP2_CMD=`which pbzip2 &> /dev/null && echo 'pbzip2' || echo 'bzip2'`

neradis avatar Oct 05 '14 13:10 neradis