bbolt panic with invalid freelist page

panic: invalid page type: 0: 4 panic: invalid freelist page: 0, page type is meta [recovered] panic: invalid freelist page: 0, page type is meta goroutine 228 [running]: github.com/PPIO/go-ppio/util/sys.Go.func1.1() /home/workspace/go/src/github.com/PPIO/go-ppio/util/sys/panic.go:55 +0x76 panic(0xd27bc0, 0xc42093a9c0) /usr/lib/go/src/runtime/panic.go:502 +0x229 go.etcd.io/bbolt.(*freelist).read(0xc4202ef230, 0x7fa4f8655000) /home/workspace/go/src/go.etcd.io/bbolt/freelist.go:237 +0x2fd go.etcd.io/bbolt.(*freelist).reload(0xc4202ef230, 0x7fa4f8655000) /home/workspace/go/src/go.etcd.io/bbolt/freelist.go:297 +0x50 go.etcd.io/bbolt.(*Tx).rollback(0xc4201880e0) /home/workspace/go/src/go.etcd.io/bbolt/tx.go:267 +0xd0 go.etcd.io/bbolt.(*DB).Update.func1(0xc4201880e0) /home/workspace/go/src/go.etcd.io/bbolt/db.go:662 +0x3e panic(0xd27bc0, 0xc42093a9a0) /usr/lib/go/src/runtime/panic.go:502 +0x229 go.etcd.io/bbolt.(*Cursor).search(0xc4201195f8, 0xc42091dee0, 0xb, 0x10, 0x4) /home/workspace/go/src/go.etcd.io/bbolt/cursor.go:250 +0x388 go.etcd.io/bbolt.(*Cursor).seek(0xc4201195f8, 0xc42091dee0, 0xb, 0x10, 0x0, 0x0, 0xe27560, 0xc4209122e0, 0xc420119520, 0xc42095afc0, ...) /home/workspace/go/src/go.etcd.io/bbolt/cursor.go:159 +0xa5 go.etcd.io/bbolt.(*Bucket).CreateBucket(0xc4201880f8, 0xc42091dee0, 0xb, 0x10, 0xc42091deeb, 0x5, 0xcb6400) /home/workspace/go/src/go.etcd.io/bbolt/bucket.go:165 +0xfa go.etcd.io/bbolt.(*Bucket).CreateBucketIfNotExists(0xc4201880f8, 0xc42091dee0, 0xb, 0x10, 0xb, 0x10, 0xc42091dee0) /home/workspace/go/src/go.etcd.io/bbolt/bucket.go:199 +0x4d go.etcd.io/bbolt.(*Tx).CreateBucketIfNotExists(0xc4201880e0, 0xc42091dee0, 0xb, 0x10, 0xb, 0x10, 0x2) /home/workspace/go/src/go.etcd.io/bbolt/tx.go:115 +0x4f github.com/timshannon/bolthold.(*Store).TxInsert(0xc4200c4630, 0xc4201880e0, 0xd27bc0, 0xc42093a990, 0xe27560, 0xc4201baa00, 0x0, 0xc4201198f0) /home/workspace/go/src/github.com/timshannon/bolthold/put.go:50 +0xc3 github.com/timshannon/bolthold.(*Store).Insert.func1(0xc4201880e0, 0x1009490, 0xc4201880e0) /home/workspace/go/src/github.com/timshannon/bolthold/put.go:38 +0x58 go.etcd.io/bbolt.(*DB).Update(0xc4203023c0, 0xc420119958, 0x0, 0x0) /home/workspace/go/src/go.etcd.io/bbolt/db.go:670 +0x90 github.com/timshannon/bolthold.(*Store).Insert(0xc4200c4630, 0xd27bc0, 0xc42093a990, 0xe27560, 0xc4201baa00, 0x0, 0x0) /home/workspace/go/src/github.com/timshannon/bolthold/put.go:37 +0x8a

Jan 09 '19 03:01 Cardinalinux

the freelist is kinda corrupt maybe I think

Jan 23 '19 02:01 WIZARD-CXY

How can one recover from this state? My containerd won't start after a power-outage due to:

panic: invalid freelist page: 1113, page type is leaf
goroutine 1 [running]:
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*freelist).read(0xc000018480, 0x7fbacd454000)
       /build/microk8s/parts/containerd/go/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/freelist.go:266 +0x265
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*DB).loadFreelist.func1()

Dec 13 '20 20:12 jar349

Please try to run bboltctl check on your DB file:

https://github.com/etcd-io/bbolt/blob/f6be82302843a215152f5a1daf652c1ee5503f85/cmd/bbolt/main.go#L111

Dec 14 '20 09:12 ptabor

Having the same issue. Can't start the docker daemon anymore after a server failure due to

panic: invalid freelist page: 1490838, page type is leaf

goroutine 744 [running]:
github.com/docker/docker/vendor/go.etcd.io/bbolt.(*freelist).read(0xc001868300, 0x7f4a6ef34000)

Where do I need to run bboltctl check? It seems I don't even have bboltctl installed, seems to be handled somehow via Docker.

How can I get my docker environment back up and running without losing any data from my docker containers?!

Jan 10 '21 17:01 smn-snkl

You can install the bolt command line tool following https://github.com/etcd-io/bbolt/tree/master#installing. The executable is bbolt and should end up in your go path.

Feb 26 '21 19:02 lukevalenta

Somewhat related issue: https://github.com/etcd-io/bbolt/issues/103

Mar 13 '21 08:03 alakesh

You can install the bolt command line tool following https://github.com/etcd-io/bbolt/tree/master#installing. The executable is bbolt and should end up in your go path.

Which is great. It checks the bolt meta.db and confirms it's broken:

page 35: reachable freed
page 4: reachable freed
page 54: reachable freed
page 2: unreachable unfreed
...
page 41: unreachable unfreed
38 errors found
invalid value

But that doesn't actually help anyone fix anything. There appears to be no extra switches to check to fix the databases.

Vaguely hoping that a compact would fix things - one tries bbolt compact -o fixed.db meta.db just to get:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4f0fa9]

leading to no-where...

So - how does one fix a corrupt bbolt database, it's all well and good knowning it's broken, but that's no help if you can't fix it. You may as well just tell people to delete the database and start again...

Apr 24 '21 13:04 Xalior