celestiaorg
crash on node launch: Arena too small, toWrite:16678 newTotal:21810897 limit:21810342
Celestia Node version
v0.12.2
OS
macOS 14.1.0
Install tools
No response
Others
celestia-da e1ad7cf4
Steps to reproduce it
it crashed after launch and keeps crashing, probably due to using the same node store
Expected result
node launches without crash
Actual result
node crashes on launch with the following log:
Relevant log output
2024-01-11T09:27:46.968-0800 INFO cmd celestia-da/server.go:48 serving celestia-da over gRPC on: 127.0.0.1:57213
2024-01-11T09:27:46.971-0800 INFO node nodebuilder/module.go:25 Accessing keyring...
2024-01-11T09:27:46.974-0800 INFO module/state state/keyring.go:40 constructed keyring signer {"backend": "test", "path": "/Users/tuxcanfly/.celestia-light/keys", "key name": "my_celes_key", "chain-id": "celestia"}
2024/01/11 09:27:47 Arena too small, toWrite:16678 newTotal:21810653 limit:21810342
github.com/dgraph-io/badger/v4/y.AssertTruef
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/error.go:62
github.com/dgraph-io/badger/v4/skl.(*Arena).putVal
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/[email protected]/skl/arena.go:81
github.com/dgraph-io/badger/v4/skl.newNode
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/[email protected]/skl/skl.go:114
github.com/dgraph-io/badger/v4/skl.(*Skiplist).Put
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/[email protected]/skl/skl.go:304
github.com/dgraph-io/badger/v4.(*memTable).UpdateSkipList.(*memTable).replayFunction.func1
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/[email protected]/memtable.go:257
github.com/dgraph-io/badger/v4.(*logFile).iterate
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/[email protected]/memtable.go:516
github.com/dgraph-io/badger/v4.(*memTable).UpdateSkipList
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/[email protected]/memtable.go:218
github.com/dgraph-io/badger/v4.(*DB).openMemTable
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/[email protected]/memtable.go:146
github.com/dgraph-io/badger/v4.(*DB).openMemTables
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/[email protected]/memtable.go:88
github.com/dgraph-io/badger/v4.Open
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:335
github.com/celestiaorg/go-ds-badger4.NewDatastore
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/celestiaorg/[email protected]/datastore.go:186
github.com/celestiaorg/celestia-node/nodebuilder.(*fsStore).Datastore
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/github.com/celestiaorg/[email protected]/nodebuilder/store.go:125
reflect.Value.call
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/go/src/reflect/value.go:596
reflect.Value.Call
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/go/src/reflect/value.go:380
go.uber.org/dig.defaultInvoker
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/container.go:238
go.uber.org/dig.(*constructorNode).Call
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/constructor.go:185
go.uber.org/dig.paramSingle.Build
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:287
go.uber.org/dig.paramList.BuildList
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:150
go.uber.org/dig.(*constructorNode).Call
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/constructor.go:155
go.uber.org/dig.paramSingle.Build
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:287
go.uber.org/dig.paramList.BuildList
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:150
go.uber.org/dig.(*constructorNode).Call
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/constructor.go:155
go.uber.org/dig.paramSingle.Build
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:287
go.uber.org/dig.paramObjectField.Build
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:484
go.uber.org/dig.paramObject.Build
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:412
go.uber.org/dig.paramList.BuildList
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:150
go.uber.org/dig.(*constructorNode).Call
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/constructor.go:155
go.uber.org/dig.paramSingle.Build
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:287
go.uber.org/dig.paramList.BuildList
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:150
go.uber.org/dig.(*constructorNode).Call
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/constructor.go:155
go.uber.org/dig.paramSingle.Build
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:287
go.uber.org/dig.paramList.BuildList
/Users/tuxcanfly/.asdf/installs/golang/1.21.1/packages/pkg/mod/go.uber.org/[email protected]/param.go:150
Notes
No response
~~Does it constantly crash or it was a single event?~~
Nvm, found the answer
The problem arises from the badger-db internal allocator, which failed to acquire memory. One possible reason for this issue could be that the store was initialized with a larger Memtable size than it actually ran on. This discrepancy might be due to the MemTable size being downscaled in the v0.12.1 release.
Is there a possibility that you upgraded the node application from a version pre-dating v0.12.1 without reinitializing the store?
Hi @walldiss I didn't remember which version I upgraded from, so I assume it was from v0.12.0. I tried running v0.12.0 with the same store, then upgraded to v0.12.2 and tried again. It didn't crash and I'm no longer able to reproduce this.
Also, is it generally recommended to reinit the store on minor version upgrades or major version upgrades or both? I'm asking because this could be useful information for service providers assuming not reiniting causes issues like this.
It also happens without upgrading the version.
- Sync light node v0.12.2
- Stop light node v0.12.2
- Start light node v0.12.2 again -> Arena too small
But the last time I upgraded to v0.12.3 it worked. Possibly this behavior is random.
We don't use Badger that extensively as before. Closing as not relevant anymore. Thanks!
