normalize profile: normalize locations: Transaction Conflict. Please retry

[wbollock]

Hi all, running Parca Server 0.20.0 (this also occurs on 0.18.0) and using the Prometheus Linode Service Discovery to scrape the pprof endpoints of many Prometheus instances

Looking at the debug logs I see:

level=debug name=parca ts=2023-10-20T15:57:17.453241741Z caller=scrape.go:486 scrape_pool=prometheus_pprof_<redacted> target="https://<redacted>:9999/prometheus/prometheus/debug/pprof/profile?seconds=10" msg="appending new sample" labels="{component=\"prometheus\", environment=\"production\", instance=\"<redacted>\", job=\"pprof\", __name__=\"process_cpu\"}"

level=error name=parca ts=2023-10-20T15:57:17.69897217Z caller=scrape.go:565 scrape_pool=prometheus_pprof_<redacted> target="https://<redacted>:9999/prometheus/prometheus/debug/pprof/profile?seconds=10" msg="WriteRaw failed for scraped profile" err="normalize profile: normalize locations: Transaction Conflict. Please retry"

level=debug name=parca ts=2023-10-20T15:57:17.699114143Z caller=scrape.go:330 scrape_pool=prometheus_pprof_<redacted> msg="scraping profile" url="https://<redacted>:9999/prometheus/prometheus/debug/pprof/profile?seconds=10"

Is this normal or does it indicate an issue? The most relevant related issue I found was https://github.com/timshannon/badgerhold/issues/46

[brancz]

Are you seeing this all the time? Does the data still look right? We do already retry, so if the data is still right we might be too eager to log it as an error.

[wbollock]

Yes I am seeing it all the time. The data looks right and Parca seems to work great. I had a hunch it might've not qualified as an error either

[Sylfrena]

I am coming across the same error as well. It is easily reproducible for me on a lima x86 vm with parca v0.20.0 while just scraping itself. I get the following error consistently:

level=error name=parca ts=2024-02-07T16:41:05.590435665Z caller=scrape.go:565 scrape_pool=default target="http://127.0.0.1:7073/debug/pprof/profile?seconds=3" msg="WriteRaw failed for scraped profile" err="normalize profile: normalize functions: get or create functions: Transaction Conflict. Please retry"