funzip and large files

Open lazureykis opened this issue 3 years ago • 2 comments

Hi there!

I've faced an issue with funzip. It crashes when uncompressed data reaches about 2Gb.

Here is an example:

> curl -s https://data.binance.vision/data/spot/monthly/trades/BTCUSDT/BTCUSDT-trades-2020-07.zip | funzip > ok.csv


> curl -s https://data.binance.vision/data/spot/monthly/trades/BTCUSDT/BTCUSDT-trades-2020-08.zip | funzip > fail.csv                                                                 

funzip error: invalid compressed data--length error

Is there something I can do to make it work? I'm not an expert, but maybe there is a simple solution to my problem.

Aug 26 '22 16:08 lazureykis

It looks like funzip hasn't been touched in 20 years. It does not support the Zip64 format being used by your larger file. I will look into updating funzip.

Oct 10 '22 20:10 madler

I have added commit https://github.com/madler/unzip/commit/0b82c20ac7375b522215b567174f370be89a4b12 to workaround this issue for now.

Oct 10 '22 21:10 madler