StreamSaver.js
StreamSaver.js copied to clipboard
jimmywarting
Zip64
Quote from Zip saver: " // NOTE: My zip library can't generate zip's over 4gb and has no compresseion // it was built solo for the purpus of saving multiple files in browser "
Are you talking about not having zip64 support or about some other limitations(Like blob size?) A Zip64 generator for unlimited size would be extremely useful.
What needs to be done to support that? is it doable?
Are you talking about not having zip64 support
Yes
or about some other limitations(Like blob size?)
Nope, everything you write is written to the hard drive in a streaming mannar. so there is no blob size limit
A Zip64 generator for unlimited size would be extremely useful.
What needs to be done to support that? is it doable?
Yes it would, it's also doable. I didn't know how first but i have recently learn how to parse a zip64 file. The next thing would be to generate one in the same way... By forcing it to be a zip64 format from the beginning
(some zip application can pre determine if the zip entry should be in good old 32 bit or 64 bit format by looking at the file size, but StreamSaver is all about saving a stream with an unknown size so it must always use 64 bit to represent the file size of files etc)
The one issue I can anticipate is Zip64 implementations are a mess. I have implemented a Zip64 generator in PHP in the past, had issues making it work with some file managers.
Most of the work on this will be testing.
As for other implementations, is https://github.com/Stuk/jszip compatible with StreamSaver? That would be a good match for both if not a lot of work is required.
https://github.com/Stuk/jszip/blob/9fb481ac2a294f9c894226ea2992919d9d6a70aa/lib/zipEntries.js
It seems Zip64 support is clearly available there
As for other implementations, is https://github.com/Stuk/jszip compatible with StreamSaver? That would be a good match for both if not a lot of work is required.
Somewhat yes. but i wrote my streaming zip version for a reason. jszip where too bloated and to slow. uses to much memory and the development have been put on halt. To get some kind of streaming with streamsaver you need to dig into this issue: https://github.com/Stuk/jszip/issues/343
the only advantage you get by using jszip is compression, but then you can't stream everything as good
https://github.com/Stuk/jszip/blob/9fb481ac2a294f9c894226ea2992919d9d6a70aa/lib/zipEntries.js
It seems Zip64 support is clearly available there
I have digged into jszip a lot and what you linked to seems only to be about parsing zip64 files i don't believe jszip ever when to writing support for generating zip64 files since having 4gb+ data in memory was a limitation in the first place
Right. Just read the topic, seems like an ill-maintained project. How much work do you expect for Zip64 support? did you look over the spec?
Tried to. Was hard so i did it reverce to try and read a zip64 File first to understand the format. Shouldn't require so many changes Urs more about getting it right
Imo the spec is incomplete. The actual work time required is around 40 hours on this.
Has there been any progress on the zip64 generator? Does any library support it?
Would be really nice to use this to zip multiple azure blob files into a big zip stream without having to download the files first
Don't worry about the Mac one. They use an old implemention of unzip on the OS level. Any mac user that downloads large archives should know to use Unarchiver, no way around that.
I dug through some old stuff, found this: https://github.com/maennchen/ZipStream-PHP/blob/master/src/ZipStream.php
It is pretty short and maps out perfectly what goes into Zip64 It's PHP, but the shortest example I found of this implemented.
Think we're getting close here https://github.com/transcend-io/conflux/compare/jimmywarting/zip64. Happy to work together on this Jimmy
Think we're getting close here https://github.com/transcend-io/conflux/compare/jimmywarting/zip64. Happy to work together on this Jimmy
Is it ready for deployment now?
I am still not yet fully mastering the streams thing, but for me it looks like https://github.com/archiverjs/node-archiver/issues could solve this problem as the archiver has seemingly zip64 support and streams, would that be something worth to try out?
