pyroscope icon indicating copy to clipboard operation
pyroscope copied to clipboard

golang_pull :The agent cannot be started

Open ly0511 opened this issue 2 years ago • 7 comments

Describe the bug

The agent cannot be started. An error occurs I used the examples/golang-pull/static/docker-compose.yml file and started with docker-compose up -d, but the agent container failed to start and exited

To Reproduce

Steps to reproduce the behavior: In examples/golang-pull/static/, run docker-compose up -d

Expected behavior

Application performance data can be successfully captured on pyroscope server, resulting in only pyroscope's own

Additional Context

The log information about the agent startup failure is as follows: image

ly0511 avatar Dec 14 '23 10:12 ly0511

can you provide more info? docker version, os version & architecture, more agent logs, git revision where you try to run examples

korniltsev avatar Dec 14 '23 10:12 korniltsev

provide

docker version:18.06.3-ce docker-compose version: 1.29.2 os: linux git version :1.8.3.1 I am running on a linux operating system, directly clone our gitlab code, using the docker-compose.yml file on gitlab to boot The file path is:https://github.com/grafana/pyroscope/blob/main/examples/golang-pull/static/docker-compose.yml

ly0511 avatar Dec 14 '23 10:12 ly0511

I dont understand. Do you run example from pyroscope github repo or something from your repo?

Share full agent logs please

korniltsev avatar Dec 14 '23 13:12 korniltsev

My steps are as follows: On a linux server 1、git clone https://github.com/grafana/pyroscope.git 2、cd pyroscope/examples/golang-pull/static 3、run docker-compose up -d 4、run docker-compose ps Displays the startup status and finds that the agent container fails to be started The following log indicates that the agent container fails to start: agent_1 | runtime/cgo: pthread_create failed: Operation not permitted agent_1 | SIGABRT: abort agent_1 | PC=0x7fe44d7e552b m=0 sigcode=18446744073709551610 agent_1 | agent_1 | goroutine 0 [idle]: agent_1 | runtime: g 0: unknown pc 0x7fe44d7e552b agent_1 | stack: frame={sp:0x7fff5497de80, fp:0x0} stack=[0x7fff5417f300,0x7fff5497e310) agent_1 | 0x00007fff5497dd80: 0x0000000000000000 0x0000000000000000 agent_1 | 0x00007fff5497de70: 0x0000000000000000 0x00007fe44d7e551e agent_1 | 0x00007fff5497de80: <0x0000000000000000 0xbb817044c78c5700 agent_1 | 0x00007fff5497de90: 0x0000000000000006 0x00007fe44d751740 agent_1 | 0x00007fff5497dea0: 0x0000000000000001 0x00007fff5497e150 agent_1 | 0x00007fff5497deb0: 0x000000000cf3e0e0 0x00007fe44d7903b6 agent_1 | 0x00007fff5497dec0: 0x00007fe44d94be90 0x00007fe44d77687c agent_1 | 0x00007fff5497ded0: 0x0000000000000020 0x0000000000000000 agent_1 | runtime: g 0: unknown pc 0x7fe44d7e552b agent_1 | stack: frame={sp:0x7fff5497de80, fp:0x0} stack=[0x7fff5417f300,0x7fff5497e310) agent_1 | 0x00007fff5497dd80: 0x0000000000000000 0x0000000000000000 agent_1 | 0x00007fff5497dd90: 0x0000000000000000 0x0000000000000000 agent_1 | 0x00007fff5497de80: <0x0000000000000000 0xbb817044c78c5700 agent_1 | 0x00007fff5497de90: 0x0000000000000006 0x00007fe44d751740 agent_1 | 0x00007fff5497dea0: 0x0000000000000001 0x00007fff5497e150 agent_1 | 0x00007fff5497deb0: 0x000000000cf3e0e0 0x00007fe44d7903b6 agent_1 | 0x00007fff5497dec0: 0x00007fe44d94be90 0x00007fe44d77687c agent_1 | agent_1 | goroutine 1 [running]: agent_1 | runtime.systemstack_switch() agent_1 | /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc0000bc740 sp=0xc0000bc730 pc=0x4729e8 agent_1 | runtime.main() agent_1 | /usr/local/go/src/runtime/proc.go:169 +0x6d fp=0xc0000bc7e0 sp=0xc0000bc740 pc=0x4409ad agent_1 | runtime.goexit() agent_1 | /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000bc7e8 sp=0xc0000bc7e0 pc=0x4749c1 agent_1 | agent_1 | rax 0x0 agent_1 | rbx 0x1 agent_1 | rcx 0xffffffffffffffff agent_1 | rdx 0x6 agent_1 | rdi 0x1 agent_1 | rsi 0x1 agent_1 | rbp 0x6 agent_1 | rsp 0x7fff5497de80 agent_1 | r8 0x0 agent_1 | r9 0x0 agent_1 | r10 0x8 agent_1 | r11 0x246 agent_1 | r12 0x1 agent_1 | r13 0x16 agent_1 | r14 0xcf3e0e0 agent_1 | r15 0x6 agent_1 | rip 0x7fe44d7e552b agent_1 | rflags 0x246 agent_1 | cs 0x33 agent_1 | fs 0x0 agent_1 | gs 0x0

ly0511 avatar Dec 15 '23 05:12 ly0511

ok, my bet is that you have toooooo old docker version which has seccomp policy blocking some syscalls unknown to it and new golang version is using these syscalls.

you can verify this by running agent container with --security-opt seccomp=unconfined

If it does help, I suggest you to update docker and libseccomp. If you have to run old docker, then try to run with seccomp policy from new docker.

korniltsev avatar Dec 15 '23 06:12 korniltsev

@ly0511 https://github.com/docker-library/golang/issues/467

korniltsev avatar Dec 15 '23 06:12 korniltsev

@ly0511 docker-library/golang#467

ok, I got it

ly0511 avatar Dec 15 '23 07:12 ly0511