leveldb/table: Writer: keys are not in increasing order X, Y

[karalabe]

One of our users reported a system crash (unrelated to leveldb, some memory management error) and on startup they failed to open the leveldb database again (https://github.com/ethereum/go-ethereum/issues/15879) with the following error:

leveldb/table: Writer: keys are not in increasing order: "LastFast\x01\xa7ݞ\x05\x00\x00\x00", "LastFast\x01\xa1۞\x05\x00\x00\x00"

Seems to me that some compaction round got interrupted halfway through. The reason I'm opening this ticket because I'd expect this to be journalled and being impossible to achieve (i.e. until compaction is done, the file being generated shouldn't be marked as done/usable).

Unsure if this report helps you in any way (feel free to close if not), just figured it's a curiosity that may be useful to submit upstream.

[ghost]

I had the same problem with syncthing.

[JhetoX]

Same here: CRIT [08-28|00:18:25.480] Failed to write headers error="leveldb/table: Writer: keys are not in increasing order: "\x15\x9c\x9ba\xae+\xf7\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x03\x00\x00\x00\x00\x00\x01\x00\xd9u\xff\xdaR\xb8\xe4\x8f\xc9\xcf", "\x0e\xbc\xb4/y4\x91\x9c圩\x99\x98&\xba\xb9D\xeb\xd0i\r\x14\xa4K\x9azN\x8c\x10\xd2\xc6b\x01~g#\x06\x00\x00\x00""

[jwitko]

Just had this same problem as well.

CRIT [01-02|18:09:49.044] Failed to write skeleton headers and progress err="leveldb/table: Writer: keys are not in increasing order: \"S\\x00\\x00\\x00\\x00\\x01\\xae@g\\x01\\x9dKK\\x00\\x00\\x00\\x00\", \"S\\x00\\x00\\x00\\x00\\x00\\xae@h\\x01\\x9cKK\\x00\\x00\\x00\\x00\""

Happened after syncing beacon headers using go-ethereum

[BrandonWeng]

Ran into the same issue on one of our validators for a Cosmos SDK based chain, only affecting one validator in the chain.

The validator panics with this error and its able to commit a couple new blocks upon restart but panics repeatedly

[FUINY7]

Same here, it went from few line from time to time to this growing up till it fill whole screen

tested ram/ssd nothing, i did not manage to sync Ethereum with geth but also besu/nethermind (ryzen 5 5600g 32Go ddr4, WD black 2T, B550I MB)

[FUINY7]

For info i got also problem with others Ethereum EC, here trying to sync with besu/prysm, whatever i choose/do i ended getting bricked DB at around 50% sync, like a clock.

so in the end i have the feeling it is more hardware that anything else maybe, some incompatibility ?

[jwitko]

so in the end i have the feeling it is more hardware that anything else maybe, some incompatibility ?

You and I have nearly identical hardware specs and as I noted in this thread a few months ago I had same exact problem. I ended up testing my TeamGroup 2x32G RAM sticks and across my 3 home servers I had 6 bad sticks of brand new RAM from memtest86+ results. I RMA'd them all and just recently got them back. Haven't tried to go at it again yet but the fact that our hardware is nearly identical doesn't give me a lot of hope for compatibility at some level. Hoping it was just my bad RAM.

[FUINY7]

I already run a ram test with 3 pass, will do more then, see you in few hours.

Edit: nothing, 0 errors

[FUINY7]

The reason was apparently the processor overheating 85°C+ on heavy load (sync for hours) that would make the ram too hot and making errors, while during memtest86 with not much CPU load it wasn't making any. Even if a CPU can go fine till 95°C ram start producing errors after 50°C So the problem was invisible while testing.

I had the feeling that the ram could be sensitive to that, so i added to the build a water cooling solution AIO, and now it's all good.

So don't take into account my issue.