legacy writer produces invalid output for large input

[anatol]

Moving discussion from https://github.com/anatol/booster/issues/117

If I feed a large input into legacy writer it produces output that neither Linux kernel nor lz4 tool likes.

To reproduce the problem generate a large input e.g. using

 dd if=/dev/urandom of=testdata/vmlinux_LZ4_19377
^C177126+0 records in
177125+0 records out
90688000 bytes (91 MB, 86 MiB) copied, 2.29452 s, 39.5 MB/s

and then apply patch from https://github.com/pierrec/lz4/issues/151#issuecomment-985185751 and you'll see output like

Stream followed by undecodable data at position 8 
/tmp/1383460459      : decoded 0 bytes 

for smaller files the output looks fine.

[anatol]

@pierrec is there any chance you can look at this issue?

[pierrec]

@anatol sorry for my slow follow up on this issue. I will try to look into this next week.

[anatol]

Hi @pierrec, friendly ping on this issue. Is there any way the community could help with moving it forward?

[pierrec]

Hi @anatol, I am sorry that I still havent had time to look into this. In the meanwhile, I encourage anyone who can to do so :)

[pierrec]

@anatol sorry about this long delay. I am now looking into this but I cannot reproduce the issue in legacy mode. I have tried with multiple random dumps without any luck in doing so. I know the inputs have to be quite large, but any chance you could share one that fails on your side please?

[anatol]

@pierrec Yes I still see this issue at the head of v4 branch. Here is how I reproduce it

1. generate a large input file e.g. dd if=/dev/urandom of=testdata/bzImage_lz4_isolated bs=64M count=1
2. apply patch e6f5e6f2e982f2c688155f35ea789e64dc26eea4 that uses lz4 tool to validate the output file
3. run TestWriterLegacy test, it says Stream followed by undecodable data at position 8 that indicates format error

hope it helps

[pierrec]

@anatol thank you for your patience! The issue should be fixed as of commit bc1239ba9073d34e1c5cc25110deb212d3e72526. Please give it a try.

[pierrec]

There is still an issue, will look into it tomorrow.

[anatol]

My booster test still fails with

➜  ~ lz4 --test booster.img.lz4 
Stream followed by undecodable data at position 67471982 
booster.img.lz4      : decoded 134217728 bytes