panic: pgid (6348659008741242810) above high water mark (236003)

Hi,

We are in the process of replacing a MySQL database with a bolt database, and after inserting a lot of records (we are estimating in the range of 10 million) into a single boltdb file we ran into the following panic:

panic: pgid (6348659008741242810) above high water mark (236003)

goroutine 6170 [running]:
project/vendor/github.com/boltdb/bolt.(*node).spill(0xc42021bd50, 0xc420309168, 0x10)
	/opt/go/src/project/vendor/github.com/boltdb/bolt/node.go:375 +0x631
project/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc4204c9a00, 0xc4203d6000, 0xc420309538)
	/opt/go/src/project/vendor/github.com/boltdb/bolt/bucket.go:570 +0x17b
project/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc420200718, 0x145ddfdb, 0x100a9a0)
	/opt/go/src/project/vendor/github.com/boltdb/bolt/bucket.go:537 +0x8f8
project/vendor/github.com/boltdb/bolt.(*Tx).Commit(0xc420200700, 0x0, 0x0)
	/opt/go/src/project/vendor/github.com/boltdb/bolt/tx.go:163 +0x125
project/vendor/github.com/boltdb/bolt.(*DB).Update(0xc4201c83c0, 0xc420309770, 0x0, 0x0)
	/opt/go/src/project/vendor/github.com/boltdb/bolt/db.go:605 +0xf7
project/vendor/github.com/asdine/storm.(*node).readWriteTx(0xc4204c9780, 0xc4205a7360, 0xc42017da50, 0xc420832a70)
	/opt/go/src/project/vendor/github.com/asdine/storm/node.go:113 +0x107
project/vendor/github.com/asdine/storm.(*node).Set(0xc4204c9780, 0xad93e8, 0xa, 0x9f5b00, 0xc42017da40, 0x9f5b00, 0xc42017da50, 0xc4203098e8, 0xc42017da50)
	/opt/go/src/project/vendor/github.com/asdine/storm/kv.go:72 +0x19d
project/vendor/github.com/asdine/storm.(*DB).Set(0xc42021bb20, 0xad93e8, 0xa, 0x9f5b00, 0xc42017da40, 0x9f5b00, 0xc42017da50, 0xfd9440, 0xc42012e360)
	/opt/go/src/project/vendor/github.com/asdine/storm/storm.go:147 +0x75
project/vendor/github.com/asdine/storm.(*DB).checkVersion(0xc42021bb20, 0x3e, 0x180)
	/opt/go/src/project/vendor/github.com/asdine/storm/storm.go:276 +0x236
project/vendor/github.com/asdine/storm.Open(0xc4204c9740, 0x3e, 0xc420309a10, 0x1, 0x1, 0x0, 0xc420309a18, 0x58b6b3)
	/opt/go/src/project/vendor/github.com/asdine/storm/storm.go:56 +0x261
project/database/boltdb.ConstructDB(0xc4204c9740, 0x3e, 0xc4205a7310, 0x9f5b00, 0xc42017d930, 0xc4205a7310)
	/opt/go/src/project/database/boltdb/db.go:25 +0x8d

We make use of the Storm ORM (https://github.com/asdine/storm) and this panic happened when attempting to reopen the database we just migrated and wrote the above number of records into. Storm then tried to set its version into a bucket, triggering this panic.

Do you have any idea what could cause this and how we can prevent it in the future? (I am posting this here because after tracing the error it seems to occur deep in boltdb and it seems unrelated to Storm)

Thanks

maybe system is flushing data to disk?

Any more insight into this issue? This just bit us as well.

I'm afraid I cannot offer you more insight, I hit this error once, from what looked to be a corrupt bolt database file and I have not seen the issue since. I sincerely hope I don't see it again..