nanos icon indicating copy to clipboard operation
nanos copied to clipboard
verified publisher icon nanovms

dgraph tracking tkt

Open eyberg opened this issue 6 years ago • 5 comments

I've seen this a few times w/various daemons including influxdb && now dgraph

i see the missing symlink calls but i'm not sure if that's actually why it's failing

1 open: "/tmp/dgraph.uniboot.unknownuser.log.INFO.20190318-194703.2", flags 80242, mode 1b6
 1 do_mkent: creat (mode 438) pathname /tmp/dgraph.uniboot.unknownuser.log.INFO.20190318-194703.2 => /tmp/dgraph.uniboot.unknownuser.log.INFO.20190318-194703.2
 1    fd 3, length 0, offset 0
 1 direct return: 3
 1 epoll_ctl
 1 direct return: 0
 1 fcntl
 1 direct return: 2
 1 fcntl
 1 fcntl: fd 3, F_SETFL, 802
 1 direct return: 0
 1 unlinkat
 1 nosyscall unlinkat
 1 unlinkat
 1 nosyscall unlinkat
 1 symlinkat
 1 nosyscall symlinkat
 1 write
 1 file_write: f 0000000010002e1b0, dest 0000000c0005a4000, offset 0 (file), length 184, file length 0
 1 file_write: b_ref: 00000000100c2d100
 1 file_op_complete: len 0, status (result:no such file) (NOTOK)
 1 wakeup 1->1 000000000008f8950

 wget https://github.com/dgraph-io/dgraph/releases/download/v1.0.13/dgraph-linux-amd64.tar.gz
....

eyberg@s1:~/d$ ops run -c config.json -d -p 5080 dgraph
eyberg@s1:~/d$ cat config.json
{
  "Args": ["dgraph", "zero", "--my=zero:5080"],
  "Dirs": ["tmp"]
}

eyberg avatar Mar 18 '19 19:03 eyberg

now this is getting to flock

1 flock
1 nosyscall flock
1 epoll_ctl
1 direct return: 0, rsp 0xc000235318
1 close
1 close: fd 7
1 direct return: 0, rsp 0xc000235348
1 gettimeofday
1 direct return: 0, rsp 0x763ffdd0
1 gettimeofday
1 direct return: 0, rsp 0x763ffdd0
1 write
2019/05/23 21:55:17 Error while opening WAL store error: function not implemented
Cannot acquire directory lock on "zw".  Another process is using this Badger database.
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.acquireDirectoryLock
        /ext-go/1/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/dir_unix.go:64
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.Open
        /ext-go/1/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:212
github.com/dgraph-io/dgraph/dgraph/cmd/zero.run
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:225
github.com/dgraph-io/dgraph/dgraph/cmd/zero.init.0.func1
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/zero/run.go:73
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).execute
        /ext-go/1/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:702
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        /ext-go/1/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:783
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).Execute
        /ext-go/1/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:736
github.com/dgraph-io/dgraph/dgraph/cmd.Execute
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:60
main.main
        /ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/main.go:33
runtime.main
        /usr/local/go/src/runtime/proc.go:201
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1333
1 direct return: 1536, rsp 0xc000235678
1 exit_group
exit status 3

eyberg avatar May 23 '19 22:05 eyberg

https://github.com/nanovms/nanos/issues/829

eyberg avatar May 24 '19 15:05 eyberg

https://github.com/nanovms/nanos/pull/1027

eyberg avatar Nov 23 '19 18:11 eyberg

first config works, second blows up in the same spot in mmap

eyberg@box:~/d$ ops run -c config2.json -p 7080 dgraph
[dgraph alpha --my=server:7080 --lru_mb=2048 --zero=zero:5080]
booting /home/eyberg/.ops/images/dgraph.img ...
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
assigned: 10.0.2.15
I1126 01:03:31.228599       2 init.go:88]

Dgraph version   : v1.0.13
Commit SHA-1     : 691b3b35
Commit timestamp : 2019-03-09 19:33:59 -0800
Branch           : HEAD
Go version       : go1.11.5

For Dgraph official documentation, visit https://docs.dgraph.io.
For discussions about Dgraph     , visit https://discuss.dgraph.io.
To say hi to the community       , visit https://dgraph.slack.com.

Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2018 Dgraph Labs, Inc.


I1126 01:03:31.276872       2 server.go:117] Setting Badger table load option: mmap
I1126 01:03:31.280647       2 server.go:129] Setting Badger value log load option: mmap
I1126 01:03:31.282401       2 server.go:157] Opening write-ahead log BadgerDB with options: {Dir:w ValueDir:w SyncWrites:true TableLoadingMode:1 ValueLogLoadingMode:2 NumVersionsToKeep:1 MaxTableSize:67108864 LevelSizeMultiplier:10 MaxLev
els:7 ValueThreshold:65500 NumMemtables:5 NumLevelZeroTables:5 NumLevelZeroTablesStall:10 LevelOneSize:268435456 ValueLogFileSize:1073741823 ValueLogMaxEntries:10000 NumCompactors:2 CompactL0OnClose:true managedTxns:false maxBatchCount:0
maxBatchSize:0 ReadOnly:false Truncate:true Logger:0x1fc13d0}
I1126 01:03:31.340886       2 node.go:83] All 0 tables opened in 0s

Kernel mode: Page fault
interrupt: 000000000000000e
frame: 0000000100a03200
error code: 0000000000000000
address: ffffffffffffffff
  rax: 8000000000000204
  rbx: 000000007f069b48 (bootstrap_region + 0000000000000f48/0000000000200000)
  rcx: 0000000000000204
  rdx: 0000000101204000
  rsi: 0000000080000000
  rdi: 000000007f069d40 (bootstrap_region + 0000000000001140/0000000000200000)
  rbp: 000000007701ff80
  rsp: 000000007701fed0
   r8: 000000007f069d40 (bootstrap_region + 0000000000001140/0000000000200000)
   r9: 0000000100200300
  r10: 0000000000000008
  r11: 0000000000000000
  r12: 0000000101800318
  r13: 0000007180000000
  r14: ffffffffffffffff
  r15: 0000000101a03000
  rip: 000000007f01ee7f (mmap + 000000000000029f/0000000000000546)
flags: 0000000000000013
   ss: 0000000000000010
   cs: 0000000000000008
   ds: 0000000000000000
   es: 0000000000000000
   fs: 0000000000000000
   gs: 0000000000000000

frame trace:

eyberg@box:~/d$ cat config.json
{
  "Args": ["dgraph", "zero", "--my=zero:5080"],
  "Dirs": ["tmp"]
}
eyberg@box:~/d$ cat config2.json
{
  "Args": ["dgraph", "alpha", "--my=server:7080", "--lru_mb=2048", "--zero=zero:5080"],
  "Dirs": ["tmp"]
}

eyberg avatar Nov 26 '19 01:11 eyberg

this is now working, although you need to disable sentry as it tries to fork?

~/d/dgraph/dgraph$ cat config.json
{
  "Args": ["dgraph", "zero", "--enable_sentry=false"],
  "Dirs": ["tmp"]
}

not sure how we'd want to package this as this is one of those programs that wants to run several different daemons depending on flags passed to it

eyberg avatar Apr 30 '20 17:04 eyberg

wanted to trow my 0.02 here. there are a number of apps that use badgerDB (go-ethereum is a fairly popular one). getting around this is will be a fairly big deal unfortunately

zeroecco avatar Feb 04 '23 00:02 zeroecco

taking the example above with badger v3 the stack trace is:

en1: assigned 10.0.2.15
badger 2023/02/04 01:16:37 INFO: All 0 tables opened in 0s
badger 2023/02/04 01:16:37 INFO: Discard stats nextEmptySlot: 0
badger 2023/02/04 01:16:37 INFO: Set nextTxnTs to 0
The answer is: 42
badger 2023/02/04 01:16:37 INFO: Lifetime L0 stalled for: 0s
panic: runtime error: slice bounds out of range [4092:178] [recovered]
	panic: runtime error: slice bounds out of range [4092:178]
	panic:
== Recovering from initIndex crash ==
File Info: [ID: 1, Size: 4096, Zeros: 0]
isEnrypted: false
== Recovered ==


goroutine 20 [running]:
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1.1()
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:351 +0xa8
panic({0x8f19c0, 0xc0000281f8})
	/snap/go/10030/src/runtime/panic.go:884 +0x212
github.com/dgraph-io/ristretto/z.(*MmapFile).Bytes(...)
	/home/rich/go/pkg/mod/github.com/dgraph-io/[email protected]/z/file.go:116
github.com/dgraph-io/badger/v3/table.(*Table).read(...)
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:418
github.com/dgraph-io/badger/v3/table.(*Table).readNoFail(0x9e9cc0?, 0xc0001c6cc0?, 0x929022?)
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:422 +0xd0
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1()
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:374 +0x265
panic({0x8f19c0, 0xc0000281e0})
	/snap/go/10030/src/runtime/panic.go:884 +0x212
github.com/dgraph-io/ristretto/z.(*MmapFile).Bytes(...)
	/home/rich/go/pkg/mod/github.com/dgraph-io/[email protected]/z/file.go:116
github.com/dgraph-io/badger/v3/table.(*Table).read(...)
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:418
github.com/dgraph-io/badger/v3/table.(*Table).readNoFail(0x6?, 0x0?, 0x9?)
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:422 +0xd0
github.com/dgraph-io/badger/v3/table.(*Table).initIndex(0xc0000e8180)
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:434 +0x48
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest(0xc0000e8180)
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:401 +0x7f
github.com/dgraph-io/badger/v3/table.OpenTable(0xc00005c7e0, {0x0, 0x1, 0x200000, 0x1e6666, 0x0, 0x3f847ae147ae147b, 0x1000, 0x0, 0x1, ...})
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:308 +0x272
github.com/dgraph-io/badger/v3/table.CreateTable({0xc0001d83b0, 0xb}, 0xc0000dc090)
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:273 +0x305
github.com/dgraph-io/badger/v3.(*DB).handleFlushTask(0xc000097b00, {0xc0000ecc00?, {0x0?, 0x0?, 0x0?}})
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1062 +0x232
github.com/dgraph-io/badger/v3.(*DB).flushMemtable(0xc000097b00, 0x0?)
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1084 +0x214
github.com/dgraph-io/badger/v3.Open.func5()
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:357 +0x25
created by github.com/dgraph-io/badger/v3.Open
	/home/rich/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:356 +0x108e

code:

package main

import (
  "log"
    "fmt"

  badger "github.com/dgraph-io/badger/v3"
)

func main() {
  db, err := badger.Open(badger.DefaultOptions("/"))
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

err = db.Update(func(txn *badger.Txn) error {
  err := txn.Set([]byte("answer"), []byte("42"))
    if err != nil {
        fmt.Println(err)
    }



  item, err := txn.Get([]byte("answer"))
    if err != nil {
        fmt.Println(err)
    }

  err = item.Value(func(val []byte) error {

    fmt.Printf("The answer is: %s\n", val)

    return nil
  })

return nil
})

}

zeroecco avatar Feb 04 '23 01:02 zeroecco

update; I tested the nightly and unfortunately the nightly build also does not work

zeroecco avatar Feb 04 '23 17:02 zeroecco

We identified a bug in the kernel that is causing the above runtime error in badger v3. Will be fixed by https://github.com/nanovms/nanos/pull/1817

francescolavra avatar Feb 07 '23 12:02 francescolavra

thank you so much!

zeroecco avatar Feb 08 '23 18:02 zeroecco

closing for now, feel free to re-open if there any new issues

eyberg avatar Feb 08 '23 19:02 eyberg