parca-dev
"too many executable mappings" error
Describe the bug I am running locally parca and parca-agent and I see a stack trace in parca and lots of error logs in parca-agent. I can't find traces for the python processes I'm trying to analyze (I searched by pid). I googled the error messages but couldn't find anything useful. Can you help me figure out what's going on?
To Reproduce run parca, parca-agent.
# parca.yaml config:
object_storage:
bucket:
type: "FILESYSTEM"
config:
directory: "./parca_data"
scrape_configs:
- job_name: "parca"
scrape_interval: "10s"
static_configs:
- targets: [ '127.0.0.1:7070' ]
run any python process (tried: ipython 8.5.0, pytest (local project, python 3.11.8)).
Expected behavior I'd expect to see profiles for the python programs running on the host.
Screenshots or Profiles (using https://pprof.me) n/a
Logs From parca: (repeated multiple times)
recovered stack trace:
goroutine 232 [running]:
runtime/debug.Stack()
runtime/debug/stack.go:24 +0x5e
github.com/parca-dev/parca/pkg/symbol/addr2line.(*DwarfLiner).PCToLines.func1()
github.com/parca-dev/parca/pkg/symbol/addr2line/dwarf.go:107 +0x45
panic({0x4001d60?, 0xc056084270?})
runtime/panic.go:914 +0x21f
github.com/parca-dev/parca/pkg/symbol/elfutils.(*debugInfoFile).SourceLines(0xc0284ab170, 0x243bc)
github.com/parca-dev/parca/pkg/symbol/elfutils/debuginfofile.go:120 +0x7a8
github.com/parca-dev/parca/pkg/symbol/addr2line.(*DwarfLiner).PCToLines(0xc051b00500?, 0xc056027680?)
github.com/parca-dev/parca/pkg/symbol/addr2line/dwarf.go:112 +0x6c
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).pcToLines(0xc0016dfad0, {0x60b65a0, 0xc02a826d40}, {0xc0551d3050, 0x28}, 0x0?)
github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:574 +0x53
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).symbolizeLocationsForMapping(0xc0016dfad0, {0x60b5bf0, 0xc00185f890}, 0xc051ee3880, {0xc051ee6de0, 0x3, 0xc051ee3680?})
github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:517 +0x1afb
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Symbolize(0xc0016dfad0, {0x60b5bf0, 0xc00185f890}, {0xc000efe600, 0x32f, 0x7?})
github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:297 +0x971
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).runSymbolizationCycle(0xc0016dfad0, {0x60b5bf0, 0xc00185f890})
github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:221 +0x152
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run.func1()
github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:190 +0xf1
github.com/parca-dev/parca/pkg/runutil.Repeat(0xc001222040?, 0xc0001822a0, 0xc007e15dd8)
github.com/parca-dev/parca/pkg/runutil/runutil.go:38 +0x82
github.com/parca-dev/parca/pkg/symbolizer.(*Symbolizer).Run(0xc0016dfad0, {0x60b5bf0, 0xc00185f890}, 0x3f87f00?)
github.com/parca-dev/parca/pkg/symbolizer/symbolizer.go:188 +0x73
github.com/parca-dev/parca/pkg/parca.Run.func5.1({0x60b5bf0?, 0xc00185f890?})
github.com/parca-dev/parca/pkg/parca/parca.go:531 +0x3d
runtime/pprof.Do({0x60b5c28?, 0xc0012c4e60?}, {{0xc001222040?, 0x0?, 0x0?}}, 0xc00187af40)
runtime/pprof/runtime.go:51 +0x9d
github.com/parca-dev/parca/pkg/parca.Run.func5()
github.com/parca-dev/parca/pkg/parca/parca.go:530 +0x1ef
github.com/oklog/run.(*Group).Run.func1({0xc0013374c0?, 0xc0013374e0?})
github.com/oklog/[email protected]/group.go:38 +0x29
created by github.com/oklog/run.(*Group).Run in goroutine 1
github.com/oklog/[email protected]/group.go:37 +0x67
From parca-agent: (repeated many times)
level=warn name=parca-agent ts=2024-04-04T12:48:18.16188908Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"
level=warn name=parca-agent ts=2024-04-04T12:48:19.081906622Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"
level=warn name=parca-agent ts=2024-04-04T12:48:21.172132294Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"
level=warn name=parca-agent ts=2024-04-04T12:48:21.173922614Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"
level=warn name=parca-agent ts=2024-04-04T12:48:22.5785825Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"
level=warn name=parca-agent ts=2024-04-04T12:48:22.879245819Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"
level=warn name=parca-agent ts=2024-04-04T12:48:24.995136311Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"
level=error name=parca-agent ts=2024-04-04T12:48:25.245026903Z caller=maps.go:1279 component=cpu_profiler component=bpf_maps msg="addUnwindTableForProcess failed" err="setUnwindTableForMapping for executable /home/me/lib/python3.10/site-packages/pydantic_core/_pydantic_core.cpython-310-x86_64-linux-gnu.so starting at 0x7a940524b000 failed: not enough profiling rounds with this unwind info"
level=warn name=parca-agent ts=2024-04-04T12:48:25.991695868Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"
level=warn name=parca-agent ts=2024-04-04T12:48:25.995384796Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"
level=warn name=parca-agent ts=2024-04-04T12:48:26.493269298Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"
level=warn name=parca-agent ts=2024-04-04T12:48:28.751652154Z caller=cpu.go:594 component=cpu_profiler msg="failed to add unwind table due to having too many executable mappings" pid=10119 err="too many executable mappings"
Software (please complete the following information): parca, version 0.21.0 (commit: 089e78068a1c4f8773b1d5a8bac6645c6414fdfd) parca-agent, version 0.30.0 (commit: da59a7c04e2e9ee29fe5871a0fd2d2964a7701cc, date: 2024-03-04T20:55:59Z), arch: amd64
Workload (please complete the following information):
pytest ./some-local-project
Environment (please complete the following information):
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"
- 6.5.0-26-generic
- x86_64
I opened https://github.com/parca-dev/parca-agent/pull/2675 to improve the error message so we understand better how many mappings would be enough.
I think the best resolution for this would be if we could make the max number of mappings configurable.
Regarding the symbolization, we actually just started revamping the symbolizer.
