Appending to an existing file doesn't work

[kestrel-one]

I'd like to append to an existing LZ4 file in a single thread. I only ever have one writer at a time.

Writing to an existing file (O_APPEND mode) appears to be possible, but I'm unable to read anything after the "last block" that was created by the first writer. Simple example below:

package main

import (
	"fmt"
	"io"
	"os"

	"github.com/pierrec/lz4/v4"
)

func main() {
	// Open a file, truncate it, and write to it
	write1File, _ := os.OpenFile("test", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0666)
	write1 := lz4.NewWriter(write1File)
	write1.Write([]byte{1})
	write1.Close()
	write1File.Close()

	// Open the same file and append to it
	write2File, _ := os.OpenFile("test", os.O_APPEND|os.O_WRONLY, 0666)
	write2 := lz4.NewWriter(write2File)
	write2.Write([]byte{2})
	write2.Close()
	write2File.Close()

	// Now try to read from the file
	file, _ := os.Open("test")
	reader := lz4.NewReader(file)
	contents, _ := io.ReadAll(reader)

	fmt.Println(contents) // []byte{1}
}

[pierrec]

This is expected behaviour: you are basically appending multiple lz4 files and a reader will only see the first one. There is a PR that supports this use case by disabling the header being written for the second file (when opened in append mode), but it need some work.

[pierrec]

Reopening this as a good night sleep makes you reconsider things :) The specs do mention frame concatenation and the fact that the reference implementation does support it, so I will make sure we do as well. Thanks for pointing that out.