coroot-node-agent icon indicating copy to clipboard operation
coroot-node-agent copied to clipboard
verified publisher icon coroot

Bug: coroot-node-agent crashes the service rtpengine-daemon with "segmentation fault"

Open nlegyes opened this issue 1 year ago • 4 comments

Hi coroot-node-agent team,

I have currently the problem, that coroot-node-agent crashes the service rtpengine-daemon with "segmentation fault" rtpengine-daemon.service: Main process exited, code=killed, status=11/SEGV. Originally I have opened this issue at the sipwise/rtpengine issue: 1888 project (if additional context is needed).

Operating System

  • OS: Debian 12 Bookworm
  • Kernel: 6.11.5+bpo-amd64
  • CPU architecture: x86_64

The OS is running as a VM on Proxmox 8.3. I am using Debian 12 Bookworm Cloud Image.

Application Versions

  • RTPEngine: Version: 13.0.1.6-1~bpo12+1
  • coroot-node-agent: v1.22.10

Steps to reproduce the problem

  1. Install rtpengine-daemon.
  2. Install coroot-node-agent.
  3. Loaded Kernel Module modprobe -v nf_conntrack
  4. RTPEngine crashes after 10 seconds.

Workaround

  1. Uninstall coroot-node-agent.
  2. Loaded Kernel Module modprobe -v nf_conntrack
  3. Start RTPEngine systemctl start rtpengine-daemon
  4. RTPEngine runs now without issues.

Logs Starting coroot-node-agent + rtpengine-daemon

Starting coroot-node-agent and rtpengine-daemon with systemctl start rtpengine-daemon coroot-node-agent && journalctl -u rtpengine-daemon -u coroot-node-agent -f.

Dec 17 17:13:56 scb02proxy02 systemd[1]: Starting rtpengine-daemon.service - NGCP RTP/media Proxy Daemon...
Dec 17 17:13:56 scb02proxy02 systemd[1]: Starting coroot-node-agent.service - Coroot node agent...
Dec 17 17:13:56 scb02proxy02 systemd[1]: Started coroot-node-agent.service - Coroot node agent.
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.718544    6032 net.go:24] whitelisted public IPs: [0.0.0.0/0]
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.718617    6032 net.go:36] ephemeral-port-range: 32768-60999
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724081    6032 cilium.go:30] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_ct4_global: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724130    6032 cilium.go:36] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_ct6_global: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724140    6032 cilium.go:43] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_lb4_backends_v2: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724149    6032 cilium.go:43] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_lb4_backends_v3: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724159    6032 cilium.go:52] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_lb6_backends_v2: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724167    6032 cilium.go:52] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_lb6_backends_v3: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.727962    6032 main.go:108] agent version: 1.22.10
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728043    6032 main.go:114] hostname: scb02proxy02
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728056    6032 main.go:115] kernel version: 6.11.5+bpo-amd64
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728437    6032 main.go:72] machine-id:  b7076c686366482fa38a1b7837685421
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728506    6032 tracing.go:40] OpenTelemetry traces collector endpoint: https://coroot.int.s24.de.tkelevator.com/v1/traces
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728571    6032 otel.go:30] OpenTelemetry logs collector endpoint: https://coroot.int.s24.de.tkelevator.com/v1/logs
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728717    6032 metadata.go:67] cloud provider:
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728732    6032 collector.go:157] instance metadata: <nil>
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728853    6032 profiling.go:55] profiles endpoint: https://coroot.int.s24.de.tkelevator.com/v1/profiles
Dec 17 17:13:56 scb02proxy02 rtpengine[6031]: INFO: [core] compile-time OpenSSL library: OpenSSL 3.0.15 3 Sep 2024
Dec 17 17:13:56 scb02proxy02 rtpengine[6031]: INFO: [core] run-time OpenSSL library: OpenSSL 3.0.15 3 Sep 2024
Dec 17 17:13:56 scb02proxy02 rtpengine[6031]: INFO: [crypto] Generating new DTLS certificate
Dec 17 17:13:56 scb02proxy02 rtpengine[6031]: INFO: [core] Startup complete, version 13.0.1.6-1~bpo12+1
Dec 17 17:13:56 scb02proxy02 rtpengine[6031]: INFO: [http] Websocket listener thread running
Dec 17 17:13:56 scb02proxy02 systemd[1]: Started rtpengine-daemon.service - NGCP RTP/media Proxy Daemon.
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: W1217 17:13:56.979120    6032 registry.go:86] Cannot connect to the Docker daemon at unix:///proc/1/root/run/docker.sock. Is the docker daemon running?
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: W1217 17:13:56.979120    6032 registry.go:86] Cannot connect to the Docker daemon at unix:///proc/1/root/run/docker.sock. Is the docker daemon running?
Dec 17 17:14:00 scb02proxy02 coroot-node-agent[6032]: W1217 17:14:00.980919    6032 registry.go:89] couldn't connect to containerd through the following UNIX sockets [/var/snap/microk8s/common/run/containerd.sock,/run/k0s/containerd.sock,/run/k3s/containerd/containerd.sock,/run/containerd/containerd.sock]: failed to dial "/proc/1/root/run/containerd/containerd.sock": context deadline exceeded
Dec 17 17:14:00 scb02proxy02 coroot-node-agent[6032]: W1217 17:14:00.980919    6032 registry.go:89] couldn't connect to containerd through the following UNIX sockets [/var/snap/microk8s/common/run/containerd.sock,/run/k0s/containerd.sock,/run/k3s/containerd/containerd.sock,/run/containerd/containerd.sock]: failed to dial "/proc/1/root/run/containerd/containerd.sock": context deadline exceeded
Dec 17 17:14:00 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:00.981009    6032 crio.go:58] cri-o socket:
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541666    6032 registry.go:323] calculated container id 1 -> /init.scope ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541745    6032 registry.go:328] "ignoring" cg="/init.scope" pid=1
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541789    6032 registry.go:323] calculated container id 2 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541800    6032 registry.go:328] "ignoring" cg="" pid=2
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541830    6032 registry.go:323] calculated container id 3 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541844    6032 registry.go:328] "ignoring" cg="" pid=3
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541870    6032 registry.go:323] calculated container id 4 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541877    6032 registry.go:328] "ignoring" cg="" pid=4
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541898    6032 registry.go:323] calculated container id 5 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541905    6032 registry.go:328] "ignoring" cg="" pid=5
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541925    6032 registry.go:323] calculated container id 6 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541931    6032 registry.go:328] "ignoring" cg="" pid=6
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541950    6032 registry.go:323] calculated container id 7 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541956    6032 registry.go:328] "ignoring" cg="" pid=7
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541976    6032 registry.go:323] calculated container id 10 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541982    6032 registry.go:328] "ignoring" cg="" pid=10
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542000    6032 registry.go:323] calculated container id 12 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542005    6032 registry.go:328] "ignoring" cg="" pid=12
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542027    6032 registry.go:323] calculated container id 13 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542032    6032 registry.go:328] "ignoring" cg="" pid=13
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542051    6032 registry.go:323] calculated container id 14 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542057    6032 registry.go:328] "ignoring" cg="" pid=14
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542074    6032 registry.go:323] calculated container id 15 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542081    6032 registry.go:328] "ignoring" cg="" pid=15
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542110    6032 registry.go:323] calculated container id 16 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542116    6032 registry.go:328] "ignoring" cg="" pid=16
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542135    6032 registry.go:323] calculated container id 17 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542141    6032 registry.go:328] "ignoring" cg="" pid=17
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542160    6032 registry.go:323] calculated container id 18 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542166    6032 registry.go:328] "ignoring" cg="" pid=18
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542184    6032 registry.go:323] calculated container id 19 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542189    6032 registry.go:328] "ignoring" cg="" pid=19
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542211    6032 registry.go:323] calculated container id 20 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542218    6032 registry.go:328] "ignoring" cg="" pid=20
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542240    6032 registry.go:323] calculated container id 21 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542246    6032 registry.go:328] "ignoring" cg="" pid=21
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542264    6032 registry.go:323] calculated container id 22 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542270    6032 registry.go:328] "ignoring" cg="" pid=22
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542296    6032 registry.go:323] calculated container id 23 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542302    6032 registry.go:328] "ignoring" cg="" pid=23
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542321    6032 registry.go:323] calculated container id 24 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542327    6032 registry.go:328] "ignoring" cg="" pid=24
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542345    6032 registry.go:323] calculated container id 25 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542350    6032 registry.go:328] "ignoring" cg="" pid=25
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542368    6032 registry.go:323] calculated container id 26 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542374    6032 registry.go:328] "ignoring" cg="" pid=26
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542392    6032 registry.go:323] calculated container id 28 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542398    6032 registry.go:328] "ignoring" cg="" pid=28
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542417    6032 registry.go:323] calculated container id 29 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542422    6032 registry.go:328] "ignoring" cg="" pid=29
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542443    6032 registry.go:323] calculated container id 30 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542449    6032 registry.go:328] "ignoring" cg="" pid=30
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542467    6032 registry.go:323] calculated container id 31 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542473    6032 registry.go:328] "ignoring" cg="" pid=31
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542493    6032 registry.go:323] calculated container id 32 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542499    6032 registry.go:328] "ignoring" cg="" pid=32
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542520    6032 registry.go:323] calculated container id 34 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542526    6032 registry.go:328] "ignoring" cg="" pid=34
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542545    6032 registry.go:323] calculated container id 35 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542551    6032 registry.go:328] "ignoring" cg="" pid=35
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542570    6032 registry.go:323] calculated container id 36 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542575    6032 registry.go:328] "ignoring" cg="" pid=36
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542593    6032 registry.go:323] calculated container id 37 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542598    6032 registry.go:328] "ignoring" cg="" pid=37
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542617    6032 registry.go:323] calculated container id 38 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542623    6032 registry.go:328] "ignoring" cg="" pid=38
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542641    6032 registry.go:323] calculated container id 40 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542648    6032 registry.go:328] "ignoring" cg="" pid=40
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542665    6032 registry.go:323] calculated container id 41 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542670    6032 registry.go:328] "ignoring" cg="" pid=41
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542693    6032 registry.go:323] calculated container id 42 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542698    6032 registry.go:328] "ignoring" cg="" pid=42
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542716    6032 registry.go:323] calculated container id 43 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542721    6032 registry.go:328] "ignoring" cg="" pid=43
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542740    6032 registry.go:323] calculated container id 44 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542745    6032 registry.go:328] "ignoring" cg="" pid=44
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542763    6032 registry.go:323] calculated container id 46 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542769    6032 registry.go:328] "ignoring" cg="" pid=46
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542786    6032 registry.go:323] calculated container id 47 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542792    6032 registry.go:328] "ignoring" cg="" pid=47
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542809    6032 registry.go:323] calculated container id 48 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542815    6032 registry.go:328] "ignoring" cg="" pid=48
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542835    6032 registry.go:323] calculated container id 49 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542841    6032 registry.go:328] "ignoring" cg="" pid=49
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542858    6032 registry.go:323] calculated container id 50 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542864    6032 registry.go:328] "ignoring" cg="" pid=50
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542891    6032 registry.go:323] calculated container id 52 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542897    6032 registry.go:328] "ignoring" cg="" pid=52
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542915    6032 registry.go:323] calculated container id 53 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542921    6032 registry.go:328] "ignoring" cg="" pid=53
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542938    6032 registry.go:323] calculated container id 54 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542944    6032 registry.go:328] "ignoring" cg="" pid=54
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542962    6032 registry.go:323] calculated container id 55 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542967    6032 registry.go:328] "ignoring" cg="" pid=55
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542985    6032 registry.go:323] calculated container id 56 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542990    6032 registry.go:328] "ignoring" cg="" pid=56
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.543010    6032 registry.go:323] calculated container id 58 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.543016    6032 registry.go:328] "ignoring" cg="" pid=58
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.543033    6032 registry.go:323] calculated container id 59 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.543039    6032 registry.go:328] "ignoring" cg="" pid=59
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.643595    6032 tls.go:115] pid=758 libssl_version=v3.0.15: libssl uprobes attached
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.845139    6032 tls.go:259] pid=6032 golang_app=/usr/bin/coroot-node-agent golang_version=go1.23.4: crypto/tls uprobes attached
Dec 17 17:14:05 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:05.041006    6032 tls.go:115] pid=602 libssl_version=v3.0.15: libssl uprobes attached
Dec 17 17:14:05 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:05.041178    6032 tls.go:174] pid=602 golang_app=/usr/sbin/zabbix_agent2 golang_version=go1.23.1: no symbol section
Dec 17 17:14:10 scb02proxy02 rtpengine[6031]: INFO: [http] HTTP GET from 10.78.6.26:41630: '/metrics'
Dec 17 17:14:16 scb02proxy02 systemd[1]: rtpengine-daemon.service: Main process exited, code=killed, status=11/SEGV
Dec 17 17:14:16 scb02proxy02 systemd[1]: rtpengine-daemon.service: Failed with result 'signal'.
Dec 17 17:14:16 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:16.978827    6032 registry.go:323] calculated container id 6110 ->  ->
Dec 17 17:14:16 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:16.978861    6032 registry.go:328] "ignoring" cg="" pid=6110

Logs Starting just rtpengine-daemon

Starting rtpengine-daemon with systemctl start rtpengine-daemon && journalctl -u rtpengine-daemon -f.

Dec 17 17:19:05 scb02proxy02 systemd[1]: Starting rtpengine-daemon.service - NGCP RTP/media Proxy Daemon...
Dec 17 17:19:05 scb02proxy02 rtpengine[6167]: INFO: [core] compile-time OpenSSL library: OpenSSL 3.0.15 3 Sep 2024
Dec 17 17:19:05 scb02proxy02 rtpengine[6167]: INFO: [core] run-time OpenSSL library: OpenSSL 3.0.15 3 Sep 2024
Dec 17 17:19:05 scb02proxy02 rtpengine[6167]: INFO: [crypto] Generating new DTLS certificate
Dec 17 17:19:05 scb02proxy02 rtpengine[6167]: INFO: [core] Startup complete, version 13.0.1.6-1~bpo12+1
Dec 17 17:19:05 scb02proxy02 rtpengine[6167]: INFO: [http] Websocket listener thread running
Dec 17 17:19:05 scb02proxy02 systemd[1]: Started rtpengine-daemon.service - NGCP RTP/media Proxy Daemon.
Dec 17 17:19:10 scb02proxy02 rtpengine[6167]: INFO: [http] HTTP GET from 10.78.6.26:41972: '/metrics'
Dec 17 17:19:25 scb02proxy02 rtpengine[6167]: INFO: [http] HTTP GET from 10.78.6.26:34238: '/metrics'
Dec 17 17:19:40 scb02proxy02 rtpengine[6167]: INFO: [http] HTTP GET from 10.78.6.26:35786: '/metrics'
Dec 17 17:19:55 scb02proxy02 rtpengine[6167]: INFO: [http] HTTP GET from 10.78.6.26:41378: '/metrics'

nlegyes avatar Dec 17 '24 16:12 nlegyes

I can't reproduce this issue on Debian 12 with kernel version 6.11.5+bpo-amd64

def avatar Dec 18 '24 13:12 def

Hi @def ,

this is weird. I was able to reproduce this issue in my HomeLab as well. In my HomeLab I use Proxmox 8.3.X too.

Steps to reproduce

Here are the exact steps which I did on a fresh Debian 12 Install:

  1. Import Debian 12 Bookworm Cloud Image into Proxmox 8.3.X .

  2. Install Backports Kernel and reboot

sudo apt update && sudo apt full-upgrade -y
sudo apt install -t bookworm-backports linux-image-amd64 linux-headers-amd64 -y
systemctl reboot
  1. Installing RTPEngine:
wget https://dfx.at/rtpengine/latest/pool/main/r/rtpengine-dfx-repo-keyring/rtpengine-dfx-repo-keyring_1.0_all.deb
sudo dpkg -i rtpengine-dfx-repo-keyring_1.0_all.deb
echo "deb [signed-by=/usr/share/keyrings/dfx.at-rtpengine-archive-keyring.gpg] https://dfx.at/rtpengine/latest bookworm main" | sudo tee /etc/apt/sources.list.d/dfx.at-rtpengine.list
sudo apt update && sudo apt install libavcodec-extra rtpengine -y
  1. Installing coroot-node-agent:
curl -sfL https://raw.githubusercontent.com/coroot/coroot-node-agent/main/install.sh | \
 COLLECTOR_ENDPOINT=http://localhost:8080 \
 SCRAPE_INTERVAL=15s \
 sh -
  1. Checking the logs:
journalctl -u rtpengine-daemon -u coroot-node-agent -e

root@test02:/home/nlegyes# journalctl -u rtpengine-daemon -u coroot-node-agent -e
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681789    5264 registry.go:323] calculated container id 20 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681798    5264 registry.go:328] "ignoring" cg="" pid=20
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681830    5264 registry.go:323] calculated container id 21 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681840    5264 registry.go:328] "ignoring" cg="" pid=21
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681872    5264 registry.go:323] calculated container id 22 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681884    5264 registry.go:328] "ignoring" cg="" pid=22
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681912    5264 registry.go:323] calculated container id 23 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681921    5264 registry.go:328] "ignoring" cg="" pid=23
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681971    5264 registry.go:323] calculated container id 24 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681982    5264 registry.go:328] "ignoring" cg="" pid=24
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682014    5264 registry.go:323] calculated container id 25 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682023    5264 registry.go:328] "ignoring" cg="" pid=25
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682053    5264 registry.go:323] calculated container id 26 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682063    5264 registry.go:328] "ignoring" cg="" pid=26
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682095    5264 registry.go:323] calculated container id 27 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682103    5264 registry.go:328] "ignoring" cg="" pid=27
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682134    5264 registry.go:323] calculated container id 28 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682144    5264 registry.go:328] "ignoring" cg="" pid=28
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682180    5264 registry.go:323] calculated container id 29 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682190    5264 registry.go:328] "ignoring" cg="" pid=29
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682221    5264 registry.go:323] calculated container id 30 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682230    5264 registry.go:328] "ignoring" cg="" pid=30
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682260    5264 registry.go:323] calculated container id 31 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682269    5264 registry.go:328] "ignoring" cg="" pid=31
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682299    5264 registry.go:323] calculated container id 32 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682308    5264 registry.go:328] "ignoring" cg="" pid=32
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682339    5264 registry.go:323] calculated container id 33 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682349    5264 registry.go:328] "ignoring" cg="" pid=33
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682379    5264 registry.go:323] calculated container id 34 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682390    5264 registry.go:328] "ignoring" cg="" pid=34
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682424    5264 registry.go:323] calculated container id 35 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682432    5264 registry.go:328] "ignoring" cg="" pid=35
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682461    5264 registry.go:323] calculated container id 36 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682471    5264 registry.go:328] "ignoring" cg="" pid=36
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682507    5264 registry.go:323] calculated container id 37 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682518    5264 registry.go:328] "ignoring" cg="" pid=37
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682546    5264 registry.go:323] calculated container id 38 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682555    5264 registry.go:328] "ignoring" cg="" pid=38
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682584    5264 registry.go:323] calculated container id 39 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682594    5264 registry.go:328] "ignoring" cg="" pid=39
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682653    5264 registry.go:323] calculated container id 40 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682664    5264 registry.go:328] "ignoring" cg="" pid=40
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682695    5264 registry.go:323] calculated container id 42 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682705    5264 registry.go:328] "ignoring" cg="" pid=42
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682740    5264 registry.go:323] calculated container id 43 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682750    5264 registry.go:328] "ignoring" cg="" pid=43
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682777    5264 registry.go:323] calculated container id 44 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682786    5264 registry.go:328] "ignoring" cg="" pid=44
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682815    5264 registry.go:323] calculated container id 45 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682825    5264 registry.go:328] "ignoring" cg="" pid=45
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682859    5264 registry.go:323] calculated container id 46 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682869    5264 registry.go:328] "ignoring" cg="" pid=46
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682898    5264 registry.go:323] calculated container id 47 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682908    5264 registry.go:328] "ignoring" cg="" pid=47
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682936    5264 registry.go:323] calculated container id 48 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682947    5264 registry.go:328] "ignoring" cg="" pid=48
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682999    5264 registry.go:323] calculated container id 49 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.683010    5264 registry.go:328] "ignoring" cg="" pid=49
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.683041    5264 registry.go:323] calculated container id 50 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.683052    5264 registry.go:328] "ignoring" cg="" pid=50
Dec 25 18:43:16 test02 coroot-node-agent[5264]: 2024/12/25 18:43:16 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:21 test02 coroot-node-agent[5264]: 2024/12/25 18:43:21 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:26 test02 coroot-node-agent[5264]: 2024/12/25 18:43:26 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:28 test02 systemd[1]: rtpengine-daemon.service: Main process exited, code=killed, status=11/SEGV
Dec 25 18:43:28 test02 systemd[1]: rtpengine-daemon.service: Failed with result 'signal'.
Dec 25 18:43:28 test02 coroot-node-agent[5264]: I1225 18:43:28.948619    5264 wrappers.go:218] level info remote_name 06d942 url http://localhost:8080/v1/metrics msg Done replaying WAL duration 12.216946158s
Dec 25 18:43:29 test02 coroot-node-agent[5264]: I1225 18:43:29.054109    5264 tls.go:259] pid=5264 golang_app=/usr/bin/coroot-node-agent golang_version=go1.23.4: crypto/tls uprobes attached
Dec 25 18:43:31 test02 coroot-node-agent[5264]: 2024/12/25 18:43:31 traces export: Post "http://localhost:8080/v1/traces": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:31 test02 coroot-node-agent[5264]: 2024/12/25 18:43:31 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:31 test02 coroot-node-agent[5264]: I1225 18:43:31.764107    5264 wrappers.go:218] level warn remote_name 06d942 url http://localhost:8080/v1/metrics msg Failed to send batch, retrying err Post "http://localhost:8080/v1/metr>
Dec 25 18:43:36 test02 coroot-node-agent[5264]: 2024/12/25 18:43:36 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:41 test02 coroot-node-agent[5264]: 2024/12/25 18:43:41 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:46 test02 coroot-node-agent[5264]: 2024/12/25 18:43:46 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:46 test02 coroot-node-agent[5264]: 2024/12/25 18:43:46 traces export: Post "http://localhost:8080/v1/traces": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:51 test02 coroot-node-agent[5264]: 2024/12/25 18:43:51 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:56 test02 coroot-node-agent[5264]: 2024/12/25 18:43:56 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:01 test02 coroot-node-agent[5264]: 2024/12/25 18:44:01 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:01 test02 coroot-node-agent[5264]: 2024/12/25 18:44:01 traces export: Post "http://localhost:8080/v1/traces": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:06 test02 coroot-node-agent[5264]: 2024/12/25 18:44:06 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:06 test02 coroot-node-agent[5264]: I1225 18:44:06.764793    5264 profiling.go:137] collected 0 profiles in 1ms
Dec 25 18:44:06 test02 coroot-node-agent[5264]: I1225 18:44:06.764832    5264 profiling.go:147] uploaded 0 profiles in 0s
Dec 25 18:44:11 test02 coroot-node-agent[5264]: 2024/12/25 18:44:11 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:16 test02 coroot-node-agent[5264]: 2024/12/25 18:44:16 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:16 test02 coroot-node-agent[5264]: 2024/12/25 18:44:16 traces export: Post "http://localhost:8080/v1/traces": dial tcp [::1]:8080: connect: connection refused

nlegyes avatar Dec 25 '24 18:12 nlegyes

I have similar problem https://github.com/EnterpriseDB/barman/issues/1058

maxpain avatar Jan 20 '25 15:01 maxpain

Also, MongoDB was crashing as well

maxpain avatar Jan 20 '25 15:01 maxpain