Failed to batch upload data

[vchuravy]

On a server with 64 cores / 128threads under heavy load the agent is unable to succeed in uploading data.

level=info name=parca-agent ts=2023-07-13T11:36:37.947986544Z caller=maps.go:1038 component=cpu_profiler component=bpf_maps msg="creating a new shard to avoid splitting the unwind ta
ble for a function"                                                                                                                                                                   
level=warn name=parca-agent ts=2023-07-13T11:41:08.474484902Z caller=batch_remote_write_client.go:139 msg="batch write client failed to send profiles" count=0 err="rpc error: code = 
DeadlineExceeded desc = context deadline exceeded"                                                                                                                                    
level=warn name=parca-agent ts=2023-07-13T11:46:08.475662242Z caller=batch_remote_write_client.go:139 msg="batch write client failed to send profiles" count=1 err="rpc error: code = 
DeadlineExceeded desc = context deadline exceeded"                                                                                                                                    
^B"level=warn name=parca-agent ts=2023-07-13T11:51:08.476589687Z caller=batch_remote_write_client.go:139 msg="batch write client failed to send profiles" count=0 err="rpc error: code
 = DeadlineExceeded desc = context deadline exceeded"                                                                                                                                 
^Clevel=warn name=parca-agent ts=2023-07-13T11:52:07.797922251Z caller=info.go:336 

[vchuravy]

level=warn name=parca-agent ts=2023-07-13T15:52:32.55032116Z caller=batch_remote_write_client.go:139 msg="batch write client failed to send profiles" count=0 err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
level=warn name=parca-agent ts=2023-07-13T15:57:32.551395853Z caller=batch_remote_write_client.go:139 msg="batch write client failed to send profiles" count=2 err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
level=warn name=parca-agent ts=2023-07-13T16:02:24.648231125Z caller=pprof.go:278 component=converter_manager pid=2358790 msg="failed to normalize address" address=7ffc75fe87f0 err="failed to compute base: failed to open mapped object file: error opening /proc/2358790/root/[stack]: open /proc/2358790/root/[stack]: no such file or directory"
level=warn name=parca-agent ts=2023-07-13T16:02:32.551769414Z caller=batch_remote_write_client.go:139 msg="batch write client failed to send profiles" count=1 err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
level=warn name=parca-agent ts=2023-07-13T16:04:03.444822617Z caller=pprof.go:278 component=converter_manager pid=2358961 msg="failed to normalize address" address=7ffc8d67a2a8 err="failed to compute base: failed to open mapped object file: error opening /proc/2358961/root/[stack]: open /proc/2358961/root/[stack]: no such file or directory"
level=warn name=parca-agent ts=2023-07-13T16:07:32.552355854Z caller=batch_remote_write_client.go:139 msg="batch write client failed to send profiles" count=6 err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"

Having left the process run for a while. Nothing seems to be uploaded to polarsignals.

[brancz]

For what it's worth we should never attempt to do a request with 0 count.

[vchuravy]

I think we determined that this was a case of the endpoint not accepting the data (due to the account not being fully active)

Having a better error message here would be great, so that one could differentiate between "local problem" and "remote problem" e.g. auth worked but data is not accepted.