pako
pako copied to clipboard
"incorrect header check" when there's extra input data
Background:
I'm trying to read a record from a git packfile using pako. This format embeds zlib data inside each record, and relies on zlib to indicate when the compressed data is done. In pako 1 this is easy enough - after the decompression is done I can use inflate.strm.avail_in
to check how much input was consumed so I can read the next packfile record from the right spot. In pako 2, however, I no longer get a clean indication of the end of the compressed data, and instead get a -3 "incorrect header check" error.
Issue: When inflating a zlib stream with extra input data at the end (that's not another zlib stream), pako 2 gives me an "incorrect header check". This works fine in pako 1.
It looks like the inflate wrapper in pako 2 assumes the extra input data is another zlib stream and tries to start reading it again, which fails in this case because the extra data is unrelated. https://github.com/nodeca/pako/blob/0398fad238edc29df44f78e338cbcfd5ee2657d3/lib/inflate.js#L237-L245
Removing this from the wrapper allows it to work. Maybe this could be a configuration setting? Or changed to only happen for gzip streams? The gzip spec has repeated records, but zlib does not.
Small example:
const pako = require("pako");
const dataHex =
"789C0B492D2E5170492C49E4020013630345" // deflate("Test Data")
+ "14303893"; // 4 bytes of extra data
const data = new Uint8Array(dataHex.match(/.{1,2}/g).map(b => parseInt(b, 16))); // hex to Uint8Array
const inflate = new pako.Inflate({ windowBits: 15 });
inflate.push(data);
console.log("avail_in", inflate.strm.avail_in);
console.log("msg", inflate.msg);
console.log("result", new TextDecoder().decode(inflate.result));
Pako 1.0.11
avail_in: 4
msg:
result: Test Data
Pako 2.*
avail_in: 2
msg: incorrect header check
result:
It's a problem to make wrapper universal. v2 closed many other dirty cases.
If you have time, i'd suggest to inspect node.js wrapper for zlib
- https://github.com/nodejs/node/blob/master/lib/zlib.js
- https://github.com/nodejs/node/blob/master/src/node_zlib.cc
May be that will give you idea what can be improved. It was used as base for v2 wrapper, but may be i missed (or could not understand) something important.
i have the same question