ZipFile.jl icon indicating copy to clipboard operation
ZipFile.jl copied to clipboard

Allow compressing arrays larger than 4 GB

Open nhz2 opened this issue 2 years ago • 1 comments

Work toward fixing #95

Before this PR Zlib.Writer and Zlib.crc32 could not handle more than 4 GB at a time because of a limitation of the zlib C library.

This PR runs the compressor on chunks of the input data in a loop to avoid the issue.

To test this requires at least 8 GB of memory, so I put the tests in a separate file, "bigtests.jl".

To run these tests run:

julia> import Pkg

julia> Pkg.test("ZipFile"; test_args=["bigtests"])

Or just run "bigtests.jl" as a julia script in a project with ZipFile.

nhz2 avatar May 16 '23 01:05 nhz2

@ThummeTo @ssingh13-rms @fhs This is ready for review. For reference, The zlib C documentation is here: https://github.com/madler/zlib/blob/master/zlib.h And https://github.com/madler/zipflow/blob/2bef2123ebe519c17b18d2d0c3c71065088de952/zipflow.c#L774C1-L827 contains an example of using zlib to write zip64 files in C.

nhz2 avatar May 16 '23 12:05 nhz2