parca-agent icon indicating copy to clipboard operation
parca-agent copied to clipboard

Published 20 hours ago • verified publisher icon parca-dev

client: RPC message too big

Open javierhonduco opened this issue 3 years ago • 2 comments

Just so I don't forget to look into this. I've occasionally stumbled upon this error

level=debug ts=2022-05-25T09:42:00.90387003Z caller=write_client.go:92 msg="batch write client failed to send profiles" retry=2.604991381s count=1 err="rpc error: code = Internal desc = failed to normalize pprof: Key with size 96032 exceeded 65000 limit. Key:\n00000000[...]"

javierhonduco avatar May 26 '22 10:05 javierhonduco

Possible duplicate https://github.com/parca-dev/parca-agent/issues/203

kakkoyun avatar May 30 '22 09:05 kakkoyun

This seems different to me from #203, the error we see here is actually happening on the server-side. During normalization on the server-side, a unique key is built to identify/deduplicate a stack trace and used within a map[string], and it appears that we are exceeding the size of string that a map allows. I think this should be an issue on the Parca server.

brancz avatar May 31 '22 08:05 brancz

I did some troubleshooting on this issue and I think the error message is come from the badger. This line of code(https://github.com/dgraph-io/badger/blob/master/txn.go#L357) match the error message above. Let me study how to fix this.

heylongdacoder avatar Oct 13 '22 12:10 heylongdacoder

I think this issue is fixed by https://github.com/parca-dev/parca/pull/1178. Since this PR introduce some hashing to reduce the length of key.

heylongdacoder avatar Oct 14 '22 17:10 heylongdacoder

We haven't seen this has actually surfaced for a while now. I'm happy to close it. @javierhonduco WDYT?

kakkoyun avatar Oct 26 '22 13:10 kakkoyun

@kakkoyun sounds good to me! Closing it, let's keep an eye just in case it happens again

javierhonduco avatar Oct 27 '22 07:10 javierhonduco