camlzip
camlzip copied to clipboard
xavierleroy
Add bigstring interfaces for deflate/inflate
Add {deflate,inflate}_bigstring similar to {delfate,inflate} that read from / write to bigstrings instead of bytes.
bigstring is defined as: type bigstring = (char, Bigarray.int8_unsigned_elt, Bigarray.c_layout) Bigarray.Array1.t See https://github.com/ocaml/ocaml/blob/trunk/bytecomp/lambda.ml#L138
The Z_BUF_ERROR fix (GPR #6) is included in the new C functions camlzip_deflate_bigstring and camlzip_inflate_bigstring.
I know some projets use "bigstrings" as out-of-heap buffers, and others to work around the length limitation on strings on 32-bit platforms. So, this is a reasonable proposal. On the other hand I note that only the low-level primitives are provided for bigstrings but not the high-level APIs in modules Zip and Gzip. Just to convince myself that this is really what we want, could you tell me about the intended usage scenario for these new primitives?
In the intended use case we compress/uncompress an infinite stream of data that uses bigstrings as buffers.
I would also like this (in the high level API's) so I can unzip things that aren't files (HTTP responses and other things in memory).
@brendanlong I wrote this somewhat hacked together library on top of cryptokit's zlib support to handle in-memory g(un)zip handling of strings. It may be useful to you until something better is available.
