benbjohnson
Litestream crashes with index out of bounds error
Litestream crashes with the following error:
litestream 0.3.8
initialized db: ******/data.sqlite
replicating to: name="s3" type="s3" bucket="******" path="*****" region="" endpoint="" sync-interval=1s
panic: runtime error: index out of range [2055336] with length 2055336
goroutine 45 [running]:
github.com/pierrec/lz4/v4/internal/lz4block.(*Compressor).CompressBlock(0x14002166000, {0x14001800000, 0x1f5ca8, 0x400000}, {0x14000f00000, 0x1f5ca8, 0x400000})
github.com/pierrec/lz4/[email protected]/internal/lz4block/block.go:207 +0x8bc
github.com/pierrec/lz4/v4/internal/lz4block.CompressBlock({0x14001800000, 0x1f5ca8, 0x400000}, {0x14000f00000, 0x1f5ca8, 0x400000})
github.com/pierrec/lz4/[email protected]/internal/lz4block/block.go:91 +0x74
github.com/pierrec/lz4/v4/internal/lz4stream.(*FrameDataBlock).Compress(0x140004e4000, 0x140004aa630, {0x14001800000, 0x1025a0130?, 0x400000}, 0x1018?)
github.com/pierrec/lz4/[email protected]/internal/lz4stream/block.go:221 +0x90
github.com/pierrec/lz4/v4.(*Writer).write(0x140004aa6c0, {0x14001800000?, 0x140000d9a58?, 0x10295bebc?}, 0x58?)
github.com/pierrec/lz4/[email protected]/writer.go:128 +0x184
github.com/pierrec/lz4/v4.(*Writer).Close(0x140004aa6c0)
github.com/pierrec/lz4/[email protected]/writer.go:162 +0xa0
github.com/benbjohnson/litestream.(*Replica).syncWAL(0x14000238000, {0x10332f6a0?, 0x14000230540})
github.com/benbjohnson/litestream/replica.go:274 +0x894
github.com/benbjohnson/litestream.(*Replica).Sync(0x14000238000, {0x10332f6a0, 0x14000230540})
github.com/benbjohnson/litestream/replica.go:190 +0x724
github.com/benbjohnson/litestream.(*Replica).monitor(0x14000238000, {0x10332f6a0, 0x14000230540})
github.com/benbjohnson/litestream/replica.go:629 +0x14c
github.com/benbjohnson/litestream.(*Replica).Start.func1()
github.com/benbjohnson/litestream/replica.go:113 +0x60
created by github.com/benbjohnson/litestream.(*Replica).Start
github.com/benbjohnson/litestream/replica.go:113 +0x114
The database looks like this:
-rw-r--r-- 1 **** 1837986975 622305280 Jul 23 13:06 data.sqlite
-rw-r--r-- 1 **** 1837986975 32768 Jul 23 14:24 data.sqlite-shm
-rw-r--r-- 1 **** 1837986975 26812992 Jul 23 13:06 data.sqlite-wal
Litestream looks like this: litestream: Mach-O 64-bit arm64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE>
I don't use a config file, but provide the paths on the command line and the S3 key using environment variables.
Litestream wal list the following contents:
litestream wal -config ./litestream.yml data.sqlite
replica generation index offset size created
s3 6219b86a1b582be2 0 0 1856 2022-07-23T11:01:32Z
s3 6219b86a1b582be2 0 57712 7046627 2022-07-23T11:01:53Z
s3 6219b86a1b582be2 1 0 7258107 2022-07-23T11:02:33Z
s3 6219b86a1b582be2 2 0 8746649 2022-07-23T11:02:56Z
s3 6219b86a1b582be2 3 0 7376692 2022-07-23T11:03:41Z
s3 6219b86a1b582be2 4 0 7706093 2022-07-23T11:04:04Z
s3 6219b86a1b582be2 5 0 11785493 2022-07-23T11:04:46Z
s3 6219b86a1b582be2 6 0 16618152 2022-07-23T11:05:10Z
s3 6219b86a1b582be2 7 0 10724377 2022-07-23T11:05:37Z
s3 6219b86a1b582be2 8 0 20171587 2022-07-23T11:06:00Z
s3 6219b86a1b582be2 9 0 20300614 2022-07-23T11:06:26Z
It looks like this is a bug in the LZ4 compressor: https://github.com/pierrec/lz4/issues/116
It was fixed in v4.1.4 of the LZ4 library, but litestream 0.3.8 (latest available) is still on v4.1.3.
The Litestream development branch updated to 4.1.14, which will fix the issue, so this should go away in the next Litestream release.
Just for reference: I wrote some bigger BLOBs in my database when the error occurred. Probably, writing BLOBs bigger than 64kB might trigger the bug in the compressor.
Good catch, @mtlynch. I pushed a PR to update some dependencies: https://github.com/benbjohnson/litestream/pull/404
I'll cut a v0.3.9 release once that passes CI. 👍
I kicked off the release build for v0.3.9: https://github.com/benbjohnson/litestream/releases/tag/v0.3.9
Sounds like this is now fixed, as of last summer's release. @geggo98, can you confirm?
