dgraph icon indicating copy to clipboard operation
dgraph copied to clipboard

fix(bulk load): panic: z.Buffer max size exceeded

Open JoelWesleyReed opened this issue 3 years ago • 4 comments

(stack trace from release v21.12.0)

panic: z.Buffer max size exceeded: 68719476736 offset: 68719476724 grow: 50

goroutine 12167 [running]:
github.com/dgraph-io/ristretto/z.(*Buffer).Grow(0xc02c4a4000, 0x32)
	/go/pkg/mod/github.com/dgraph-io/[email protected]/z/buffer.go:180 +0x56e
github.com/dgraph-io/ristretto/z.(*Buffer).SliceAllocate(0xc02c4a4000, 0x2e)
	/go/pkg/mod/github.com/dgraph-io/[email protected]/z/buffer.go:266 +0x2e
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*mapIterator).Next(0xc027cfaf60, 0xc02c4a4000, {0xc00378fd00, 0x1f, 0x20})
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:206 +0x127
github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).reduce.func2()
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:488 +0x5cd
created by github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*reducer).reduce
	/ext-go/1/src/github.com/dgraph-io/dgraph/dgraph/cmd/bulk/reduce.go:478 +0x465

This change is Reviewable

JoelWesleyReed avatar Dec 22 '21 12:12 JoelWesleyReed

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 10 committers have signed the CLA.

:white_check_mark: matthewmcneely
:x: ahsanbarkati
:x: manishrjain
:x: danielmai
:x: iluminae
:x: NamanJain8
:x: aman-bansal
:x: Phill240
:x: Joel W. Reed
:x: dpeek


Joel W. Reed seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

CLAassistant avatar Dec 22 '21 12:12 CLAassistant

@JoelWesleyReed if you are still interested in merging this change, could you rebase on latest main please and sign the CLA? Thanks

mangalaman93 avatar Mar 21 '23 05:03 mangalaman93

I encountered the same problem trying to bulk import a huge dataset and this PR fixed the max size panic. Is it possible to merge it? Otherwise @mangalaman93 I can open a new PR.

davideaimar avatar May 26 '23 10:05 davideaimar

Yeah, you'll have to cherry pick your changes on main instead of master. You should still be able to force push to the same branch. You are welcome to create another PR too.

mangalaman93 avatar May 26 '23 14:05 mangalaman93

This PR has been stale for 60 days and will be closed automatically in 7 days. Comment to keep it open.

github-actions[bot] avatar Aug 01 '24 14:08 github-actions[bot]