Failure to symbolize binary

[danipozo]

Parca can't symbolize addresses for ClickHouse binaries.

How to reproduce

First, download and decompress ClickHouse binary:

curl -LO https://s3.amazonaws.com/clickhouse-builds/24.10/2e054d555719f509b96e0cfea4d5f5819c69a61f/package_release/clickhouse
chmod +x clickhouse
./clickhouse --version

Normal flow reproducer

Then, execute ClickHouse server:

mkdir ch-data && cd ch-data
../clickhouse server

Try to profile it with Parca (you will need to compile parca-agent with the latest version of the upstream OTel eBPF profiler to get the fix for this). Also use --debuginfo-upload-max-size=2244458216 to be able to upload the binary debuginfo. You will observe that stacks are missing symbols in the UI.

Smaller reproducer

Compile and run this as a binary in this repo (pardon my poor Go):

package main

import (
	"context"
	"debug/elf"
	"os"

	"github.com/go-kit/log"
	"github.com/go-kit/log/level"
	"github.com/parca-dev/parca/pkg/symbol/addr2line"
	"github.com/parca-dev/parca/pkg/symbol/demangle"
)

func main() {
	ctx := context.Background()

  path := "clickhouse"
  f, err := elf.Open(path)
  if err != nil {
    print(err)
  }

  demangler := demangle.NewDemangler("simple", false)
  writer := log.NewSyncWriter(os.Stdout)
  logger := log.NewLogfmtLogger(writer)
  liner, err := addr2line.DWARF(logger, path, f, demangler)
  if err != nil {
    print(err)
  }

  lines, err := liner.PCToLines(ctx, 0x1192c860)
  if err != nil {
    level.Error(logger).Log("err", err)
  }
}

Error logs

These come ultimately from debug/dwarf/entry.go, method dwarf5Ranges.

Parca:

level=debug name=parca ts=2024-11-15T12:39:31.37757694Z caller=symbolizer.go:172 component=symbolizer msg="failed to get lines" err="liner pctolines: failed to extract dwarf tree: invalid rnglist offset 409180350 (max 636512)"

Reproducer bin:

level=error err="failed to extract dwarf tree: invalid rnglist offset 2858131463 (max 636512)"

Cross-checking with other tools

Using gimli's addr2line (0x1192c860 is one of the addresses it fails to symbolize):

$ ./addr2line-gimli -e clickhouse -fC 0x1192c860
DB::AggregateUtils::isAggregateFunction(DB::ASTFunction const&)
./build_docker/./src/AggregateFunctions/AggregateFunctionFactory.h:119

Also to note, I've checked some of the generated stacks with this addr2line implementation and they seem to make sense, so stack walking seems to be producing valid addresses.

Other comments

Any tips to debug or cross-check this with dwarfdump or other tools are welcome. For example, how could I check if the offsets that the entry offsets the code is parsing at each step are valid?

[danipozo]

This should get partially fixed when using newer versions of Go: https://go-review.googlesource.com/c/go/+/655976

[danishubin]

Hi, wanted to ask if you have some eta for when Parca will be updated with the newer bersion of Go. I seem to be having a similar problem with Parca failing to symbolize a large binary similar to Clickhouse and was hoping to deploy profiling in the next week or two. Thanks for your great work!

[danipozo]

Hey @danishubin you can use the flag introduced in https://github.com/parca-dev/parca/pull/5299 along with an external addr2line implementation (I recommend gimli-rs/addr2line for performance and completeness) for symbolization.

[danishubin]

Hi,

Thank you for your reply. Though it successfully calls addr2 (based on log messages), it still does not seem to be symbolizing successfully (see below image). Specifically, the error which I think is most relevant is I am getting on the parca-server pod is component=symbolizer msg="failed to symbolize: BuildID d1aa79a916f6d082" err="get debuginfo: fetching metadata: debuginfo metadata not found" where the buildid corresponds to my binary I am trying to symbolize. Initially I thought this was because the agent was failing to upload the binary because it was so large 3GB+ but I'm not sure sure if there are any other flags I can use to solve this problem. This is my current setup below. I appreciate any help you can give.

Parca Server yaml args - /parca - --http-address=:7070 - --config-path=/etc/parca/parca.yaml - --log-level=debug - --cors-allowed-origins=* - --debuginfo-upload-max-size=5244458216 - --symbolizer-external-addr-2-line-path=/usr/bin/addr2line - --debuginfo-upload-max-duration=15m

Parca Agent yaml args - args: - --http-address=:7071 - --node=$(NODE_NAME) - --config-path=/etc/parca-agent/parca-agent.yaml - --remote-store-address=parca.parca.svc.cluster.local:7070 - --remote-store-insecure - --log-level=debug - --debuginfo-upload-timeout-duration=2m - --remote-store-grpc-connection-timeout=300s - --remote-store-grpc-max-call-recv-msg-size=335544320000 - --remote-store-grpc-max-call-send-msg-size=335544320000 - --debuginfo-compress - --debuginfo-strip=true

[brancz]

This sounds like the binary simply doesn’t have symbols.

[danishubin]

Checking my binary, it is compiled with debug symbols and not stripped. Output of file <binary_name>: ... with debug_info, not stripped