lz4_flex icon indicating copy to clipboard operation
lz4_flex copied to clipboard

provide size of uncompressed data

Open jtmoon79 opened this issue 9 months ago • 5 comments

Can lz4_flex provide an API for getting the uncompressed size of data before decompressing that data? I assume the uncompressed size is in some kind of block header or something like that? (I don't know the LZ4 binary format; just guessing)

This would be particularly helpful for lz4 compressed files.

jtmoon79 avatar May 06 '24 03:05 jtmoon79

There's the content_size flag in the frame format header, but it's existence is optional. Also there may be multiple frames. So it would be the sum of the content_size.

This would be particularly helpful for lz4 compressed files.

Why would that be helpful?

PSeitz avatar May 10 '24 03:05 PSeitz

There's the content_size flag in the frame format header, but it's existence is optional.

How do I access the content_size for some file? Can you complete this code example:

use std::fs::File;
use std::fs::OpenOptions;
use std::io::prelude::Read;
use std::io::BufReader;
use std::path::Path;
use ::lz4_flex;

fn main() {
    let mut open_options = OpenOptions::new();
    let path = String::from("/tmp/file.lz4");
    let path_std = Path::new(&path);
    let file_lz: File = match open_options.read(true).open(path_std) {
        Ok(val) => val,
        Err(err) => panic!("{}", err),
    };
    let bufreader = BufReader::<File>::new(file_lz);
    let mut lz4_decoder = lz4_flex::frame::FrameDecoder::new(bufreader);
    // ... how to access content_size ?
}

I created /tmp/file.lz4 with

$ lz4c --version
*** LZ4 command line interface 64-bits v1.9.3, by Yann Collet ***
$ lz4c -9kv /var/log/syslog -c > /tmp/file.lz4

jtmoon79 avatar May 27 '24 21:05 jtmoon79

current_frame_info in FrameDecoder is private and depends on the current frame

PSeitz avatar May 28 '24 03:05 PSeitz

There's the content_size flag in the frame format header, but it's existence is optional... So it would be the sum of the content_size. current_frame_info in FrameDecoder is private and depends on the current frame

I'm confused. Could you please provide a code example?

jtmoon79 avatar May 28 '24 05:05 jtmoon79

It's not accessible currently. It's existence is optional, so not sure what it can be used for externally

PSeitz avatar May 28 '24 06:05 PSeitz