dgraph
dgraph copied to clipboard
fix(bulk load): panic: z.Buffer max size exceeded
(stack trace from release v21.12.0)
panic: z.Buffer max size exceeded: 68719476736 offset: 68719476724 grow: 50
goroutine 12167 [running]:
github.com/dgraph-io/ristretto/z.(*Buffer).Grow(0xc02c4a4000, 0x32)
/go/pkg/mod/github.com/dgraph-io/[email protected]/z/buffer.go:180 +0x56e
github.com/dgraph-io/ristretto/z.(*Buffer).SliceAllocate(0xc02c4a4000, 0x2e)
/go/pkg/mod/github.com/dgraph-io/[email protected]/z/buffer.go:266 +0x2e
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*mapIterator).Next(0xc027cfaf60, 0xc02c4a4000, {0xc00378fd00, 0x1f, 0x20})
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:206 +0x127
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).reduce.func2()
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:488 +0x5cd
created by github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).reduce
/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:478 +0x465
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 10 committers have signed the CLA.
:white_check_mark: matthewmcneely
:x: ahsanbarkati
:x: manishrjain
:x: danielmai
:x: iluminae
:x: NamanJain8
:x: aman-bansal
:x: Phill240
:x: Joel W. Reed
:x: dpeek
Joel W. Reed seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.
@JoelWesleyReed if you are still interested in merging this change, could you rebase on latest main please and sign the CLA? Thanks
I encountered the same problem trying to bulk import a huge dataset and this PR fixed the max size panic. Is it possible to merge it? Otherwise @mangalaman93 I can open a new PR.
Yeah, you'll have to cherry pick your changes on main instead of master. You should still be able to force push to the same branch. You are welcome to create another PR too.
This PR has been stale for 60 days and will be closed automatically in 7 days. Comment to keep it open.