dgraph-io
Badger shouldn't just panic when open a bad database
When BadgerDB tries to open a corrupted dababase, sometime it would just panic(seems that this panic only occurs on Windows):
panic: runtime error: index out of range [3] with length 0 [recovered]
panic:
== Recovering from initIndex crash ==
File Info: [ID: 38, Size: 408, Zeros: 408]
isEnrypted: false checksumLen: 0 checksum: indexLen: 0 index: []
== Recovered ==
goroutine 80 [running]:
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1.1(0xc0001c1aa0)
C:/gocode/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:351 +0x165
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1(0xc0001d6180)
C:/gocode/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:397 +0x6e
panic(0x1d77ca0, 0xc0003bcd68)
C:/go/src/runtime/panic.go:971 +0x49a
github.com/google/flatbuffers/go.GetInt32(...)
C:/gocode/pkg/mod/github.com/google/[email protected]/go/encode.go:85
github.com/google/flatbuffers/go.GetUOffsetT(...)
C:/gocode/pkg/mod/github.com/google/[email protected]/go/encode.go:121
github.com/dgraph-io/badger/v3/fb.GetRootAsTableIndex(...)
C:/gocode/pkg/mod/github.com/dgraph-io/badger/[email protected]/fb/TableIndex.go:14
github.com/dgraph-io/badger/v3/table.(*Table).readTableIndex(0xc0001d6180, 0x0, 0x8, 0xc0001c1a40)
C:/gocode/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:704 +0x491
github.com/dgraph-io/badger/v3/table.(*Table).initIndex(0xc0001d6180, 0x0, 0x0, 0xc0002a3260)
C:/gocode/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:462 +0x545
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest(0xc0001d6180, 0x0, 0x0)
C:/gocode/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:401 +0xab
github.com/dgraph-io/badger/v3/table.OpenTable(0xc000434280, 0x0, 0x200000, 0x0, 0x0, 0x3f847ae147ae147b, 0x1000, 0x0, 0x1, 0xc00053a080, ...)
C:/gocode/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:308 +0x36e
github.com/dgraph-io/badger/v3.newLevelsController.func1(0xc00043e180, 0xc0002b57b0, 0xc000589000, 0xc0002b5798, 0xc000178d10, 0x7, 0x7, 0xc00050a780, 0x21, 0x6, ...)
C:/gocode/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:150 +0x2d8
created by github.com/dgraph-io/badger/v3.newLevelsController
As a widely used dependency, BadgerDB should not just panic. Those panics cannot be captured and then breaks the whole application.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@HaoboGu will do our best. For the time being, I'm tagging @pandeyshubham25 (he may have come across this) ...
@HaoboGu would you be able to provide us with sample steps to reproduce this issue?
some other qs:
- details about your badger version # or tag or commit-sha?
- are you seeing this issue with latest release of badger?
- system specs would be helpful
- detailed steps to reproduce the issue, it would be good to get sample data for this issue (possibly)
- detail expected behavior (you have described it partially in the issue, a little more Information may help)
- badger version: github.com/dgraph-io/badger/v3 v3.2103.2
- Yes, I'm using latest release version of badger
- Windows 10 1903 (OS build 18362.1082) / go1.16.2 windows/amd64
- reproduce:
- download the corrupted db db.zip
- open it on Windows using
badger.Open(badger.DefaultOptions(dbPath).WithVerifyValueChecksum(true)) - badger panics then
- badger should return an error instead of just panic
i'll try this later in the day, and circle back. pls bear with me as I figure this out (will be discussing with the internal team on this ...)
ok this may not be very helpful, tested this on Mac-OS 12.4 (won't have access to windows machine till tomorrow)
Looks like I get a different error while trying to do this:
$ badger --dir ~/Downloads/db/ info show-keys
Listening for /debug HTTP requests at port: 8080
jemalloc enabled: false
Using Go memory
[ 2021-05-05T15:08:08Z] MANIFEST 716 B MA
[ 1 month earlier] 000002.sst 247 B L6
[ 1 month earlier] 000004.sst 246 B L6
[ 1 month earlier] 000006.sst 247 B L6
[ 1 month earlier] 000008.sst 247 B L6
[ 1 month earlier] 000010.sst 247 B L6
[ 1 month earlier] 000012.sst 247 B L6
[ 1 month earlier] 000014.sst 247 B L6
[ 1 month earlier] 000016.sst 247 B L6
[ 1 month earlier] 000018.sst 247 B L6
[ 1 month earlier] 000023.sst 337 B L6
[ 3 weeks earlier] 000025.sst 247 B L6
[ 21 hours earlier] 000028.sst 408 B L6
[ 21 hours earlier] 000030.sst 247 B L6
[ 21 hours earlier] 000032.sst 247 B L6
[ 19 hours earlier] 000035.sst 410 B L6
[ 30 minutes earlier] 000038.sst 408 B L6
[ 29 minutes earlier] 000041.vlog 20 B VL
[ 29 minutes earlier] 000042.vlog 2.0 GiB VL
[EXTRA]
[2021-05-05T15:08:08Z] 00086.mem 128 MiB
[2021-03-08T17:42:34Z] DISCARD 1.0 MiB
[2021-03-08T17:42:34Z] KEYREGISTRY 28 B
[Summary]
Level 0 size: 0 B
Level 1 size: 0 B
Level 2 size: 0 B
Level 3 size: 0 B
Level 4 size: 0 B
Level 5 size: 0 B
Level 6 size: 4.4 KiB
Total SST size: 4.4 KiB
Value log size: 2.0 GiB
Abnormalities:
3 extra files.
0 missing files.
0 empty files.
0 truncated manifests.
Error: failed to open database err: while opening memtables error: while opening fid: 86 error: while updating skiplist error: end offset: 20 < size: 134217728 error: Log truncate required to run DB. This might result in data loss
github.com/dgraph-io/badger/v3.init
/Users/sudhishkr/scratch/github/dgraph-io/badger/errors.go:101
runtime.doInit
/usr/local/opt/go/libexec/src/runtime/proc.go:6222
runtime.doInit
/usr/local/opt/go/libexec/src/runtime/proc.go:6199
runtime.doInit
/usr/local/opt/go/libexec/src/runtime/proc.go:6199
runtime.main
/usr/local/opt/go/libexec/src/runtime/proc.go:233
runtime.goexit
/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1571
Are you trying to open the DB like this? badger --dir ~/Downloads/db/ info show-keys <<
I got this:
PS C:\Users\ADMIN\Downloads> badger --dir db info show-keys
Listening for /debug HTTP requests at port: 8080
jemalloc enabled: false
Using Go memory
[2021-05-05T15:08:07+08:00] MANIFEST 716 B MA
[ 1 month earlier] 000002.sst 247 B L6
[ 1 month earlier] 000004.sst 246 B L6
[ 1 month earlier] 000006.sst 247 B L6
[ 1 month earlier] 000008.sst 247 B L6
[ 1 month earlier] 000010.sst 247 B L6
[ 1 month earlier] 000012.sst 247 B L6
[ 1 month earlier] 000014.sst 247 B L6
[ 1 month earlier] 000016.sst 247 B L6
[ 1 month earlier] 000018.sst 247 B L6
[ 1 month earlier] 000023.sst 337 B L6
[ 3 weeks earlier] 000025.sst 247 B L6
[ 21 hours earlier] 000028.sst 408 B L6
[ 21 hours earlier] 000030.sst 247 B L6
[ 21 hours earlier] 000032.sst 247 B L6
[ 19 hours earlier] 000035.sst 410 B L6
[ 30 minutes earlier] 000038.sst 408 B L6
[ 29 minutes earlier] 000041.vlog 20 B VL
[ 29 minutes earlier] 000042.vlog 2.0 GiB VL
[EXTRA]
[2021-05-05T15:08:07+08:00] 00086.mem 128 MiB
[2021-03-08T17:42:33+08:00] DISCARD 1.0 MiB
[2021-03-08T17:42:32+08:00] KEYREGISTRY 28 B
[Summary]
Level 0 size: 0 B
Level 1 size: 0 B
Level 2 size: 0 B
Level 3 size: 0 B
Level 4 size: 0 B
Level 5 size: 0 B
Level 6 size: 4.4 KiB
Total SST size: 4.4 KiB
Value log size: 2.0 GiB
Abnormalities:
3 extra files.
0 missing files.
0 empty files.
0 truncated manifests.
Error: failed to open database err: Read-only mode is not supported on Windows
github.com/dgraph-io/badger/v3.init
C:/Users/ADMIN/Downloads/badger-3.2103.2/errors.go:94
runtime.doInit
C:/Program Files/Go/src/runtime/proc.go:6222
runtime.doInit
C:/Program Files/Go/src/runtime/proc.go:6199
runtime.doInit
C:/Program Files/Go/src/runtime/proc.go:6199
runtime.main
C:/Program Files/Go/src/runtime/proc.go:233
runtime.goexit
C:/Program Files/Go/src/runtime/asm_amd64.s:1571
Usage:
badger info [flags]
Flags:
--cv-mode string [none, table, block, tableAndBlock] Specifies when the db should verify checksum for SST. (default "none")
--discard Parse and print DISCARD file from value logs.
--enc-key string Use the provided encryption key
-h, --help help for info
--histogram Show a histogram of the key and value sizes.
--history Show all versions of a key
-l, --lookup string Hex of the key to lookup
--read-only If set to true, DB will be opened in read only mode. If DB has not been closed properly, this option can be set to false to open DB. (default true)
--show-internal Show internal keys along with other keys. This option should be used along with --show-key option
--show-keys Show keys stored in Badger
--show-meta Output item meta data as well (default true)
-s, --show-tables If set to true, show tables as well.
--truncate If set to true, it allows truncation of value log files if they have corrupt data.
--with-prefix string Consider only the keys with specified prefix
Global Flags:
--dir string Directory where the LSM tree files are located. (required)
--vlog-dir string Directory where the value log files are located, if different from --dir
failed to open database err: Read-only mode is not supported on Windows
github.com/dgraph-io/badger/v3.init
C:/Users/ADMIN/Downloads/badger-3.2103.2/errors.go:94
runtime.doInit
C:/Program Files/Go/src/runtime/proc.go:6222
runtime.doInit
C:/Program Files/Go/src/runtime/proc.go:6199
runtime.doInit
C:/Program Files/Go/src/runtime/proc.go:6199
runtime.main
C:/Program Files/Go/src/runtime/proc.go:233
runtime.goexit
C:/Program Files/Go/src/runtime/asm_amd64.s:1571
your error is probably related to folder permissions -> failed to open database err: Read-only mode is not supported on Windows.
my error is something I need to look further into. @MichelDiz have you see this before?
@HaoboGu - how did you exactly simulate the error you have indicated in your original post https://github.com/dgraph-io/badger/issues/1797#issue-1320470136? You did respond saying
open it on Windows using badger.Open(badger.DefaultOptions(dbPath).WithVerifyValueChecksum(true))
if you could elaborate on the steps it will be helpful.
This is the minimal repro that I can do:
Here is the main.go:
package main
import (
"fmt"
"github.com/dgraph-io/badger/v3"
)
func main() {
opts := badger.DefaultOptions("C://Users/ADMIN/Downloads/db")
_, err := badger.Open(opts)
if err != nil {
fmt.Print(err)
}
}
Then go run ./main.go
I would suggest in function newLevelsController of levels.go, add a defer function to recover and panic again in the go routine (line 129), so that the panic in this go routine can be caught by the client code. in this way, the application can handle the panic from badgerDB when opening a corrupted db, then delete the corrupted db and create a new db to serve.
