Out of memory exception.

[fiasko131]

When I try to create a 7z archive with:

outArchive7z = SevenZip.openOutArchive7z();
outArchive7z.setLevel(level);
outArchive7z.createArchive(new RandomAccessFileOutStream(raf),//
             items.length/**/, new CallBack7z());

I get an Exception when the compression level is greater than 3:

2021-03-01 10:01:19.134 26545-26834/com.techniwork.fastsharing W/System.err: 7z-Error occurs: 2021-03-01 10:01:19.134 26545-26834/com.techniwork.fastsharing W/System.err: HRESULT: 0x8007000E (Out of memory). Error creating '7z' archive with 12 items 2021-03-01 10:01:19.134 26545-26834/com.techniwork.fastsharing W/System.err: at net.sf.sevenzipjbinding.impl.OutArchiveImpl.nativeUpdateItems(Native Method) 2021-03-01 10:01:19.134 26545-26834/com.techniwork.fastsharing W/System.err: at net.sf.sevenzipjbinding.impl.OutArchiveImpl.doUpdateItems(OutArchiveImpl.java:141) 2021-03-01 10:01:19.134 26545-26834/com.techniwork.fastsharing W/System.err: at net.sf.sevenzipjbinding.impl.OutArchiveImpl.createArchive(OutArchiveImpl.java:150)

I don't have this exception for .zip archives.

[omicronapps]

Check how you're implementing ISequentialInStream.read(). You may need to provide the contents of each file in smaller sections. For example, even if read() requests "16777216" bytes you may not want to read out the whole 16 MB at one time.

Also see sevenzipjbinding Issue #15 for additional suggestions.

If this still fails, then please provide a complete source code example for testing locally.

[fiasko131]

Thank you for your reply,

Here is the code I am using taken directly from the snippets:

private final class CallBack7z implements IOutCreateCallback<IOutItem7z>,ICryptoGetTextPassword {
        IOutItem7z item;
        int index;
        byte[] content;
        @Override
        public void setOperationResult(boolean operationResultOk)//
                throws SevenZipException {
            // Track each operation result here
                    if (!items[index].isDir){
                        Arrays.fill(content, (byte)0);
                        fileDone++;
                        currentFileName = new File(items[index].getPath()).getName();
                        publishProgress();
                    }


        }

        @Override
        public IOutItem7z getItemInformation(int index, OutItemFactory<IOutItem7z> outItemFactory) throws SevenZipException {
            item = outItemFactory.createOutItem();

            if (items[index].isDir) {
                item.setPropertyIsDir(true);
            } else {
                item.setDataSize(items[index].getFile().length());
                nbFiles++;
            }

            item.setPropertyPath(items[index].getPath());
            return item;
        }


        @Override
        public void setTotal(long total) throws SevenZipException {
            // Track operation progress here
            totalBytes = total;
        }
        @Override
        public void setCompleted(long complete) throws SevenZipException {
            // Track operation progress here
            totalCompressed = complete;
        }

        @Override
        public ISequentialInStream getStream(int i) throws SevenZipException {
            if (items[i].isDir) {
                index = i;
                return null;
            }else {
                index = i;
                content = convertFileToByteArray(items[i].getFile());
                return new ByteArrayStream(content, true);
            }

        }

        @Override
        public String cryptoGetTextPassword() throws SevenZipException {
            return passWord;
        }
    }

I don't know yet how to split the byte array of the file into several pieces.

[fiasko131]

So I tried with ISequentialInStream:

private class SequentialInputStream implements  ISequentialInStream{
        InputStream inputStream;
        int result;
        public SequentialInputStream(InputStream inputStream){
            this.inputStream = inputStream;
        }

        @Override
        public int read(byte[] data) throws SevenZipException {
            try {
                result = inputStream.read(data);
                if (result <= 0) {

                    return 0;
                }
                Log.i("readresult", String.valueOf(result));
                totalCompressedFile +=result;
                Log.i("readresult", String.valueOf(totalCompressedFile));
                return result;
            } catch (IOException e) {
                e.printStackTrace();
                return 0;
            }
        }
        @Override
        public void close() throws IOException {
            inputStream.close();
        }
    }

and in the Callback:

 @Override
        public ISequentialInStream getStream(int i) throws SevenZipException {
            if (items[i].isDir) {
                index = i;
                return null;
            }else {
                index = i;
                totalBytesFile = items[index].getFile().length();
                InputStream inputStream = null;
                try {
                    currentFileName = new File(items[index].getPath()).getName();
                    publishProgress();
                    inputStream = new FileInputStream(items[i].getFile());
                } catch (FileNotFoundException e) {
                    e.printStackTrace();
                }
                return new SequentialInputStream(inputStream);
            }

        }

It does not change for:

 outArchive7z = SevenZip.openOutArchive7z();
 outArchive7z.setLevel(level);
 outArchive7z.createArchive(new RandomAccessFileOutStream(raf),//
              items.length/**/, new CallBack7z());

out of memory exception when the level is higher than 3. But this method seems more reliable since we do not actually copy the content of the file in memory.

On the other hand I tried to apply it to:

outArchiveZip = SevenZip.openOutArchiveZip();
outArchiveZip.setLevel(level);
outArchiveZip.createArchive(new RandomAccessFileOutStream(raf),//
            items.length, new CallBackZip());

with the same getStream(int i) methode in callback

and there I get this exception:

2021-03-02 15:51:21.772 21050-21334/com.techniwork.fastsharing W/System.err: +-- The SevenZipException itself with first thrown 'cause by' exception (first cause): 2021-03-02 15:51:21.772 21050-21334/com.techniwork.fastsharing W/System.err: | HRESULT: 0x80004001 (Not implemented). Error creating 'zip' archive with 3 items 2021-03-02 15:51:21.773 21050-21334/com.techniwork.fastsharing W/System.err: | at net.sf.sevenzipjbinding.impl.OutArchiveImpl.nativeUpdateItems(Native Method) 2021-03-02 15:51:21.773 21050-21334/com.techniwork.fastsharing W/System.err: F at net.sf.sevenzipjbinding.impl.OutArchiveImpl.doUpdateItems(OutArchiveImpl.java:141) 2021-03-02 15:51:21.773 21050-21334/com.techniwork.fastsharing W/System.err: I at net.sf.sevenzipjbinding.impl.OutArchiveImpl.createArchive(OutArchiveImpl.java:150) 2021-03-02 15:51:21.774 21050-21334/com.techniwork.fastsharing W/System.err: R at net.sf.sevenzipjbinding.impl.OutArchiveZipImpl.createArchive(OutArchiveZipImpl

This library is still a bit confusing for me!

[omicronapps]

I get an Exception when the compression level is greater than 3:

I created the following project to try and debug the "out of memory" error. (This uses the source code you provided, but just adds random data to a 7z archive.) https://github.com/omicronapps/Outofmemoryexception

But I can't reproduce the "Out of memory" error that you're seeing. Can you modify this project to reproduce this issue?

This library is still a bit confusing for me!

I'm sorry about that. It's a fairly large library, implementing most of the 7zip functionality. There's always the documentation available here: javadoc. And for more complex API questions, there's also the main Java library project: sevenzipjbinding.

[fiasko131]

thank you, I modified the code and did some tests here: https://github.com/omicronapps/Outofmemoryexception/issues/1

This library is still a bit confusing for me!

Obviously this is not a criticism, far from me this idea. It's just that I have a very limited level of java in general :)) Thank you again for all this work and all this availability!

[omicronapps]

When archiving large files with a high level of compression, the sevenzipjbinding library can use large amounts of Java heap memory.

To be able to handle such scenarios on Android, you may need to set android:largeHeap to true in the application Manifest file. This will ensure that the application process is created with a large ART Java heap.

[fiasko131]

Yes selecting largeHeap to true solves the problem for the test application, but not for my app for which I had already selected largeHeap.

In my case the fact that I load a lot of bitmaps in memory managed by Glide, and it seems that ARP does not allow me to go further. With a 7z compression and 3 files of 100 Mb I cannot go further than level 2.

Regarding the level, whatever the level and the type of archive selected, the obtained archive weighs 300Mb for 3 files of 100Mb, I conclude that the level of compression is not effective?

I implemented in Outofmemoryexception a handler that displays the memory used by the app and the ram usage of the device. Indeed it is huge> 2Gb!

[omicronapps]

Please see sevenzipjbinding Issue #7 "OutOfMemoryError while constructing ByteArrayStream":

But it is the case of "It's not a bug, it's a feture" :) Sorry.

You see, the ByteArrayStream was explicitly designed for the small files, that should completely fit into the memory.

So using method1 with ByteArrayStream may not be feasible for large files.

Instead using RandomAccessFileInStream is recommended, as described in the documentation here: http://sevenzipjbind.sourceforge.net/first_steps.html

Regarding the poor compression ratio, the sevenzipjbinding library is based on 7-Zip, so it should be able to achieve the same level of compression as the stand-alone 7-Zip command line tool. Would using the command line tool with the same options result in the same archive file size? Also, to confirm, these are actual files and not purely random data generated with Java Random?

[asthagarg2428]

@fiasko131 Can you please share your complete code which you used for compressing big files?

[lancewoo]

Is it because this library of SDK version can not limit the dictionary size when doing work? I did not find any API to set a dictionary size. While I encountered a similar problem of Out Of Memory(OOM) on Android, I changed to use the command line 7zzs to do my compression work. Then I limited the dictionary size on the command line parameter by -md=10m. After that crashes due to OOM seldomly happened.

[omicronapps]

There is a property kDictionarySize defined in the Java interface: https://github.com/borisbrodski/sevenzipjbinding/blob/master/jbinding-java/src/net/sf/sevenzipjbinding/NCoderPropID.java#L13

However, it doesn't look like this is actually implemented and used the in Java library.

Can you hear in sevenzipjbinding if and how the dictionary size can be controlled through the Java API?

[lancewoo]

I just found some usage tips on the memory issue.

But I failed to find an API to control the dictionary size in Java. Just posted an inquiry.