pcompress icon indicating copy to clipboard operation
pcompress copied to clipboard

Published 20 hours ago • verified publisher icon moinakg

Failing tests: FATAL: Decompression failed and LZMA: Memory allocation error

Open szepeviktor opened this issue 10 years ago • 10 comments

make test with gcc version 4.9.1 (Debian 4.9.1-19) ./configoutput: https://gist.github.com/szepeviktor/6b26af8344bff1aa1284

Please help me.

szepeviktor avatar Feb 14 '15 14:02 szepeviktor

The VPS has 2GB of memory. After stopping mysql etc. the LZMA memory error disappeared. On low memory is it OK to throw allocation errors?

szepeviktor avatar Feb 14 '15 14:02 szepeviktor

FATALS

#################################################
# Simple compress and decompress
#################################################
Running ../../pcompress -c adapt -l 1 -s 1m /root/pcompress/test/datafiles/bin.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/bin.dat.pz /root/pcompress/test/datafiles/bin.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 1 -s 1m /root/pcompress/test/datafiles/share.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/share.dat.pz /root/pcompress/test/datafiles/share.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 1 -s 1m /root/pcompress/test/datafiles/inc.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/inc.dat.pz /root/pcompress/test/datafiles/inc.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 1 -s 1m /root/pcompress/test/datafiles/combined.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/combined.dat.pz /root/pcompress/test/datafiles/combined.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 1 -s 1m /root/pcompress/test/datafiles/comb_d.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/comb_d.dat.pz /root/pcompress/test/datafiles/comb_d.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 3 -s 1m /root/pcompress/test/datafiles/bin.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/bin.dat.pz /root/pcompress/test/datafiles/bin.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 3 -s 1m /root/pcompress/test/datafiles/share.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/share.dat.pz /root/pcompress/test/datafiles/share.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 3 -s 1m /root/pcompress/test/datafiles/inc.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/inc.dat.pz /root/pcompress/test/datafiles/inc.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 3 -s 1m /root/pcompress/test/datafiles/combined.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/combined.dat.pz /root/pcompress/test/datafiles/combined.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 3 -s 1m /root/pcompress/test/datafiles/comb_d.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/comb_d.dat.pz /root/pcompress/test/datafiles/comb_d.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 9 -s 1m /root/pcompress/test/datafiles/bin.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/bin.dat.pz /root/pcompress/test/datafiles/bin.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 9 -s 1m /root/pcompress/test/datafiles/share.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/share.dat.pz /root/pcompress/test/datafiles/share.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 9 -s 1m /root/pcompress/test/datafiles/inc.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/inc.dat.pz /root/pcompress/test/datafiles/inc.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 9 -s 1m /root/pcompress/test/datafiles/combined.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/combined.dat.pz /root/pcompress/test/datafiles/combined.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 9 -s 1m /root/pcompress/test/datafiles/comb_d.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/comb_d.dat.pz /root/pcompress/test/datafiles/comb_d.dat.1
Scaling to 2 threads
Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/bin.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/bin.dat.pz /root/pcompress/test/datafiles/bin.dat.1
Scaling to 2 threads
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 100m /root/pcompress/test/datafiles/bin.dat
Scaling to 1 thread
Running ../../pcompress -d /root/pcompress/test/datafiles/bin.dat.pz /root/pcompress/test/datafiles/bin.dat.1
Scaling to 1 thread
Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/share.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/share.dat.pz /root/pcompress/test/datafiles/share.dat.1
Scaling to 2 threads
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 100m /root/pcompress/test/datafiles/share.dat
Scaling to 1 thread
Running ../../pcompress -d /root/pcompress/test/datafiles/share.dat.pz /root/pcompress/test/datafiles/share.dat.1
Scaling to 1 thread
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/inc.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/inc.dat.pz /root/pcompress/test/datafiles/inc.dat.1
Scaling to 2 threads
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 100m /root/pcompress/test/datafiles/inc.dat
Scaling to 1 thread
Running ../../pcompress -d /root/pcompress/test/datafiles/inc.dat.pz /root/pcompress/test/datafiles/inc.dat.1
Scaling to 1 thread
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/combined.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/combined.dat.pz /root/pcompress/test/datafiles/combined.dat.1
Scaling to 2 threads
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 100m /root/pcompress/test/datafiles/combined.dat
Scaling to 1 thread
Running ../../pcompress -d /root/pcompress/test/datafiles/combined.dat.pz /root/pcompress/test/datafiles/combined.dat.1
Scaling to 1 thread
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/comb_d.dat
Scaling to 2 threads
Running ../../pcompress -d /root/pcompress/test/datafiles/comb_d.dat.pz /root/pcompress/test/datafiles/comb_d.dat.1
Scaling to 2 threads
Segmentation fault (core dumped)
FATAL: Decompression failed.
Running ../../pcompress -c adapt -l 14 -s 100m /root/pcompress/test/datafiles/comb_d.dat
Scaling to 1 thread
Running ../../pcompress -d /root/pcompress/test/datafiles/comb_d.dat.pz /root/pcompress/test/datafiles/comb_d.dat.1
Scaling to 1 thread
Segmentation fault (core dumped)
FATAL: Decompression failed.

szepeviktor avatar Feb 14 '15 14:02 szepeviktor

Your precompiled pcompress_static_linux.tar.bz2 does not fail.

szepeviktor avatar Feb 14 '15 14:02 szepeviktor

Here is what I do:

# pbuilder login

cd /root
echo 'deb ftp://debian.mirrors.crysys.hu/debian/ jessie main contrib' > /etc/apt/sources.list
echo 'deb-src ftp://debian.mirrors.crysys.hu/debian/ jessie main contrib' >> /etc/apt/sources.list

# tools
apt-get update && apt-get install -y git ca-certificates mc \
    file libtool autoconf automake automake1.11 yasm

# deps
apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev libarchive-dev

# sources
git clone https://github.com/moinakg/pcompress.git
apt-get source libwavpack-dev

# build
cd pcompress/ && ./config --wavpack-dir=/root/wavpack-4.70.0; echo "config=$?"
make && make test

szepeviktor avatar Feb 14 '15 14:02 szepeviktor

The first fail is always the decompression of Running ../../pcompress -c adapt -l 14 -s 1m /root/pcompress/test/datafiles/bin.dat

szepeviktor avatar Feb 14 '15 15:02 szepeviktor

BTW. This is a very nice project: fast and very good on compression.

szepeviktor avatar Feb 14 '15 15:02 szepeviktor

Your precompiled pcompress_static_linux.tar.bz2 does not fail.

Not really:

# ./pcompress -c adapt -l 14 -s 1m ../pcompress/test/datafiles/bin.dat a.pz
Scaling to 2 threads
Minimum chunk size for Global Dedup must be 2097152 bytes

Error compressing file: /root/src/pcompress/test/datafiles/bin.dat

szepeviktor avatar Feb 14 '15 16:02 szepeviktor

Thanks a lot for reporting. This looks like a regression. I will check this out. I also have to improve detection of low memory systems and handle appropriately.

Regards, Moinak.

On Sat, Feb 14, 2015 at 8:57 AM, Viktor Szépe [email protected] wrote:

Your precompiled pcompress_static_linux.tar.bz2 does not fail.

Not really:

./pcompress -c adapt -l 14 -s 1m ../pcompress/test/datafiles/bin.dat a.pz

Scaling to 2 threads Minimum chunk size for Global Dedup must be 2097152 bytes

Error compressing file: /root/src/pcompress/test/datafiles/bin.dat

— Reply to this email directly or view it on GitHub https://github.com/moinakg/pcompress/issues/28#issuecomment-74382619.

http://moinakg.wordpress.com/

moinakg avatar Feb 15 '15 02:02 moinakg

@moinakg Is it the same issue?

# pcompress -a -j WordPress-4.1/ wp-pcompr
Scanning files.
Sorting ...
Scaling to 2 threads
Segmentation fault

2 GB free RAM. v3.1 Even with -t 1.

szepeviktor avatar Apr 18 '15 11:04 szepeviktor

Not sure. I have to reproduce this on a 2GB VM.

moinakg avatar Apr 29 '15 08:04 moinakg