otel-profiling-agent icon indicating copy to clipboard operation
otel-profiling-agent copied to clipboard
verified publisher icon elastic

AppImage run error

Open JimLeeee opened this issue 1 year ago • 12 comments

./devfiler-appimage-$(uname -m).AppImage /nix/store/f004k9qdr6ms8xq8gq6qswwg076d1bfr-devfiler-stripped/bin/devfiler: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.26' not found (required by /nix/store/f004k9qdr6ms8xq8gq6qswwg076d1bfr-devfiler-stripped/bin/devfiler) /nix/store/f004k9qdr6ms8xq8gq6qswwg076d1bfr-devfiler-stripped/bin/devfiler: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.29' not found (required by /nix/store/f004k9qdr6ms8xq8gq6qswwg076d1bfr-devfiler-stripped/bin/devfiler) /nix/store/f004k9qdr6ms8xq8gq6qswwg076d1bfr-devfiler-stripped/bin/devfiler: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.30' not found (required by /nix/store/f004k9qdr6ms8xq8gq6qswwg076d1bfr-devfiler-stripped/bin/devfiler) /nix/store/f004k9qdr6ms8xq8gq6qswwg076d1bfr-devfiler-stripped/bin/devfiler: /usr/lib64/libstdc++.so.6: version CXXABI_1.3.13' not found (required by /nix/store/f004k9qdr6ms8xq8gq6qswwg076d1bfr-devfiler-stripped/bin/devfiler)

also agent run error

./otel-profiling-agent -disable-tls -collection-agent=0.0.0.0:11000 -bpf-log-level 2 -verbose time="2024-04-18T08:34:32.057604323Z" level=debug msg="Config:" time="2024-04-18T08:34:32.057761503Z" level=debug msg="bpf-log-level: 2" time="2024-04-18T08:34:32.057786270Z" level=debug msg="bpf-log-size: 65536" time="2024-04-18T08:34:32.057804997Z" level=debug msg="cache-directory: /var/cache/otel/profiling-agent" time="2024-04-18T08:34:32.057824927Z" level=debug msg="collection-agent: 0.0.0.0:11000" time="2024-04-18T08:34:32.057840633Z" level=debug msg="config: /etc/otel/profiling-agent/agent.conf" time="2024-04-18T08:34:32.057868231Z" level=debug msg="copyright: false" time="2024-04-18T08:34:32.057878160Z" level=debug msg="disable-tls: true" time="2024-04-18T08:34:32.057888442Z" level=debug msg="map-scale-factor: 0" time="2024-04-18T08:34:32.057898090Z" level=debug msg="no-kernel-version-check: false" time="2024-04-18T08:34:32.057910669Z" level=debug msg="probabilistic-interval: 1m0s" time="2024-04-18T08:34:32.057920608Z" level=debug msg="probabilistic-threshold: 100" time="2024-04-18T08:34:32.057930438Z" level=debug msg="project-id: 1" time="2024-04-18T08:34:32.057939619Z" level=debug msg="secret-token: abc123" time="2024-04-18T08:34:32.057949204Z" level=debug msg="t: all" time="2024-04-18T08:34:32.057959168Z" level=debug msg="tags: " time="2024-04-18T08:34:32.057968497Z" level=debug msg="tracers: all" time="2024-04-18T08:34:32.057978131Z" level=debug msg="v: true" time="2024-04-18T08:34:32.057987110Z" level=debug msg="verbose: true" time="2024-04-18T08:34:32.057996204Z" level=debug msg="version: false" time="2024-04-18T08:34:32.058007268Z" level=info msg="Starting OTEL profiling agent 1.0.0 (revision OTEL-review, build timestamp N/A)" time="2024-04-18T08:34:32.066764829Z" level=debug msg="Validated tags: " time="2024-04-18T08:34:32.084189854Z" level=debug msg="Traffic to 0.0.0.0 is routed from 127.0.0.1" time="2024-04-18T08:34:32.084326097Z" level=debug msg="Reading the configuration" time="2024-04-18T08:34:32.084385632Z" level=debug msg="Done setting configuration" time="2024-04-18T08:34:32.084399834Z" level=debug msg="Determining tracers to include" time="2024-04-18T08:34:32.084414601Z" level=debug msg="Tracer string: all" time="2024-04-18T08:34:32.084425643Z" level=info msg="Interpreter tracers: perl,php,python,hotspot,ruby,v8" time="2024-04-18T08:34:32.084436565Z" level=info msg="Automatically determining environment and machine ID ..." time="2024-04-18T08:34:35.085739723Z" level=debug msg="Environment tester (azure) failed: failed to get azure metadata: Get "http://169.254.169.254/metadata/instance/compute?api-version=2020-09-01&format=json": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

time="2024-04-18T08:34:44.633481537Z" level=debug msg="Environment tester (aws) failed: failed to get aws metadata: EC2MetadataRequestError: failed to get EC2 instance identity document\ncaused by: RequestError: send request failed\ncaused by: Get "http://169.254.169.254/latest/dynamic/instance-identity/document": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" time="2024-04-18T08:34:46.549829839Z" level=debug msg="Environment tester (gcp) failed: failed to get GCP metadata: Get "http://169.254.169.254/computeMetadata/v1/instance/id": dial tcp 169.254.169.254:80: i/o timeout" time="2024-04-18T08:34:46.553658401Z" level=debug msg="Using MAC: 0x120F95D4E9F4" time="2024-04-18T08:34:46.553711072Z" level=info msg="Environment: hardware, machine ID: 0x13947a896189adc7" time="2024-04-18T08:34:46.553725672Z" level=info msg="Assigned ProjectID: 1 HostID: 1410887313739328967" time="2024-04-18T08:34:46.570770645Z" level=debug msg="Traffic to 0.0.0.0 is routed from 127.0.0.1" time="2024-04-18T08:34:49.586417179Z" level=warning msg="Failed to setup gRPC connection (try 1 of 5): context deadline exceeded: connection error: desc = "transport: Error while dialing: dial tcp 0.0.0.0:11000: connect: connection refused""

JimLeeee avatar Apr 18 '24 08:04 JimLeeee

What distribution are you running? Please paste output of

lsb_release -a

or, if that doesn't work

cat /etc/os-release

The error suggests that your system's installed glibc is older than what we currently support.

athre0z avatar Apr 18 '24 12:04 athre0z

NAME="CentOS Linux" VERSION="8" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="8" PLATFORM_ID="platform:el8" PRETTY_NAME="CentOS Linux 8" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:8" HOME_URL="https://centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-8" CENTOS_MANTISBT_PROJECT_VERSION="8"

@athre0z one more question: -collection-agent=: this param is the server addr which receive profile logs ? or only agent

JimLeeee avatar Apr 19 '24 03:04 JimLeeee

-collection-agent=: this param is the server addr which receive profile logs ? or only agent

It is the address (hostname/ip:port) of the receiving service (devfiler).

rockdaboot avatar Apr 19 '24 09:04 rockdaboot

I can't run it either. It's core dumped.

$ RUST_BACKTRACE=1 RUST_BACKTRACE=full ./devfiler-appimage-x86_64.AppImage
2024-04-20T17:55:25.566483Z  INFO devfiler::collector: Collector listening on 0.0.0.0:11000
thread 'main' panicked at src/main.rs:29:35:
called `Result::unwrap()` on an `Err` value: WinitEventLoop(NotSupported(NotSupportedError))
stack backtrace:
   0:     0x562d86fd7dab - <unknown>
   1:     0x562d8701345c - <unknown>
   2:     0x562d86ffb7ee - <unknown>
   3:     0x562d86fd7b85 - <unknown>
   4:     0x562d86fe9753 - <unknown>
   5:     0x562d86fe93e6 - <unknown>
   6:     0x562d86fe9bf9 - <unknown>
   7:     0x562d86fd81c7 - <unknown>
   8:     0x562d86fd7fb6 - <unknown>
   9:     0x562d86fe9922 - <unknown>
  10:     0x562d87019913 - <unknown>
  11:     0x562d870117b3 - <unknown>
  12:     0x562d863eefb5 - <unknown>
  13:     0x562d8632f153 - <unknown>
  14:     0x562d86397b7d - <unknown>
  15:     0x562d86fd69b7 - <unknown>
  16:     0x562d8640bfac - <unknown>
  17:     0x7fcdc7f81fce - __libc_start_call_main
  18:     0x7fcdc7f82089 - __libc_start_main_impl
  19:     0x562d86274ca5 - <unknown>
  20:                0x0 - <unknown>
/nix/store/n2769lvhb1g27lgg9bh857df884168xl-devfiler-appimage/bin/devfiler-appimage: line 3: 33883 Aborted                 (core dumped) /nix/store/f004k9qdr6ms8xq8gq6qswwg076d1bfr-devfiler-stripped/bin/devfiler "$@"

The distro I use is:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy
$
$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

ms2008 avatar Apr 20 '24 18:04 ms2008

@ms2008 is this a pure Wayland setup without XWayland, by any chance? It looks like winit is failing to create a Window. Also, how are you running this? From a terminal within a desktop session?

athre0z avatar Apr 20 '24 18:04 athre0z

@athre0z Sorry, I do not have much knowledge of Wayland.

I'm not executing from a terminal in a desktop session, but from a terminal on a remote connection via ssh. Also, I have configured the DISPLAY environment variable and tried to forward X11 over SSH to run the program remotely and it still doesn't work.

ms2008 avatar Apr 21 '24 14:04 ms2008

Hmm, I probably shouldn't have fully stripped the build -- it's hard to make sense of the stacktrace now. Does the remote machine have a GPU and OpenGL installation? It's required for rendering on the machine running devfiler. Why are you running devfiler in this setup? Are you trying to profile a remote server? In that case I'd recommend to go with running it locally and forwarding the connection via a reverse SSH tunnel: there's an explainer on how to do that in devfiler's "Add data" dialog.

athre0z avatar Apr 22 '24 13:04 athre0z

I encountered the same problem

$ sudo RUST_BACKTRACE=full ./devfiler-appimage-x86_64.AppImage 
2024-04-23T06:19:22.254142Z  INFO devfiler::collector: Collector listening on 0.0.0.0:11000
thread 'main' panicked at src/main.rs:29:35:
called `Result::unwrap()` on an `Err` value: WinitEventLoop(Os(OsError { line: 786, file: "/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/winit-0.29.15/src/platform_impl/linux/mod.rs", error: Misc("neither WAYLAND_DISPLAY nor WAYLAND_SOCKET nor DISPLAY is set.") }))
stack backtrace:
   0:     0x55ba23dbcdab - <unknown>
   1:     0x55ba23df845c - <unknown>
   2:     0x55ba23de07ee - <unknown>
   3:     0x55ba23dbcb85 - <unknown>
   4:     0x55ba23dce753 - <unknown>
   5:     0x55ba23dce3e6 - <unknown>
   6:     0x55ba23dcebf9 - <unknown>
   7:     0x55ba23dbd1c7 - <unknown>
   8:     0x55ba23dbcfb6 - <unknown>
   9:     0x55ba23dce922 - <unknown>
  10:     0x55ba23dfe913 - <unknown>
  11:     0x55ba23df67b3 - <unknown>
  12:     0x55ba231d3fb5 - <unknown>
  13:     0x55ba23114153 - <unknown>
  14:     0x55ba2317cb7d - <unknown>
  15:     0x55ba23dbb9b7 - <unknown>
  16:     0x55ba231f0fac - <unknown>
  17:     0x7f75aac9cfce - __libc_start_call_main
  18:     0x7f75aac9d089 - __libc_start_main_impl
  19:     0x55ba23059ca5 - <unknown>
  20:                0x0 - <unknown>
/nix/store/n2769lvhb1g27lgg9bh857df884168xl-devfiler-appimage/bin/devfiler-appimage: line 3: 65098 Aborted                 (core dumped) /nix/store/f004k9qdr6ms8xq8gq6qswwg076d1bfr-devfiler-stripped/bin/devfiler "$@"

The distro as follow:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy
$
$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

My machine is a new environment, the system is an ubuntu-server distribution,and there is no GPU

wenhuwang avatar Apr 25 '24 02:04 wenhuwang

I'm getting error when performing

sudo ./otel-profiling-agent -collection-agent=0.0.0.0:11000 -disable-tls -verbose

time="2024-04-30T13:29:55.226985443Z" level=debug msg="Config:"
time="2024-04-30T13:29:55.227066771Z" level=debug msg="bpf-log-level: 0"
time="2024-04-30T13:29:55.227087463Z" level=debug msg="bpf-log-size: 65536"
time="2024-04-30T13:29:55.227110464Z" level=debug msg="cache-directory: /var/cache/otel/profiling-agent"
time="2024-04-30T13:29:55.227129539Z" level=debug msg="collection-agent: 0.0.0.0:11000"
time="2024-04-30T13:29:55.227149099Z" level=debug msg="config: /etc/otel/profiling-agent/agent.conf"
time="2024-04-30T13:29:55.227190824Z" level=debug msg="copyright: false"
time="2024-04-30T13:29:55.227206736Z" level=debug msg="disable-tls: true"
time="2024-04-30T13:29:55.227220390Z" level=debug msg="map-scale-factor: 0"
time="2024-04-30T13:29:55.227235985Z" level=debug msg="no-kernel-version-check: false"
time="2024-04-30T13:29:55.227251849Z" level=debug msg="probabilistic-interval: 1m0s"
time="2024-04-30T13:29:55.227291647Z" level=debug msg="probabilistic-threshold: 100"
time="2024-04-30T13:29:55.227329827Z" level=debug msg="project-id: 1"
time="2024-04-30T13:29:55.227341815Z" level=debug msg="secret-token: abc123"
time="2024-04-30T13:29:55.227356896Z" level=debug msg="t: all"
time="2024-04-30T13:29:55.227371941Z" level=debug msg="tags: "
time="2024-04-30T13:29:55.227388667Z" level=debug msg="tracers: all"
time="2024-04-30T13:29:55.227402181Z" level=debug msg="v: true"
time="2024-04-30T13:29:55.227419503Z" level=debug msg="verbose: true"
time="2024-04-30T13:29:55.227427673Z" level=debug msg="version: false"
time="2024-04-30T13:29:55.227440607Z" level=info msg="Starting OTEL profiling agent 1.0.0 (revision OTEL-review, build timestamp N/A)"
time="2024-04-30T13:29:55.259785290Z" level=debug msg="Validated tags: "
time="2024-04-30T13:29:55.261338146Z" level=debug msg="Traffic to 0.0.0.0 is routed from 127.0.0.1"
time="2024-04-30T13:29:55.261411347Z" level=debug msg="Reading the configuration"
time="2024-04-30T13:29:55.261484324Z" level=debug msg="Done setting configuration"
time="2024-04-30T13:29:55.261504227Z" level=debug msg="Determining tracers to include"
time="2024-04-30T13:29:55.261580421Z" level=debug msg="Tracer string: all"
time="2024-04-30T13:29:55.261594283Z" level=info msg="Interpreter tracers: perl,php,python,hotspot,ruby,v8"
time="2024-04-30T13:29:55.261609722Z" level=info msg="Automatically determining environment and machine ID ..."
time="2024-04-30T13:29:55.263214512Z" level=debug msg="Environment tester (gcp) failed: failed to get GCP metadata: metadata: GCE metadata \"instance/id\" not defined"
time="2024-04-30T13:29:55.263621709Z" level=debug msg="Environment tester (azure) failed: failed to unmarshal JSON response: invalid character '<' looking for beginning of value"
time="2024-04-30T13:29:55.264630824Z" level=info msg="Environment: aws, machine ID: 0x171355c887b026fa"
time="2024-04-30T13:29:55.264651620Z" level=info msg="Assigned ProjectID: 1 HostID: 8580296034829215482"
time="2024-04-30T13:29:55.266135145Z" level=debug msg="Traffic to 0.0.0.0 is routed from 127.0.0.1"
time="2024-04-30T13:29:55.270333637Z" level=debug msg="Unable to get metadata key: ancestor-ami-ids: not found"
time="2024-04-30T13:29:55.275866588Z" level=debug msg="Unable to get metadata key: kernel-id: not found"
time="2024-04-30T13:29:55.280757933Z" level=debug msg="Unable to get metadata key: product-codes: not found"
time="2024-04-30T13:29:58.411917344Z" level=warning msg="Failed to setup gRPC connection (try 1 of 5): context deadline exceeded: connection error: desc = \"transport: Error while dialing: dial tcp 0.0.0.0:11000: connect: connection refused\""

Nothing seems to be starting and listening on port 11000.

Machine is an Ubuntu 20.04 hosted on AWS:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"

nikolai-kirov avatar Apr 30 '24 13:04 nikolai-kirov

I'm getting error when performing

sudo ./otel-profiling-agent -collection-agent=0.0.0.0:11000 -disable-tls -verbose

@nikolai-kirov Please start devfiler first - it listens on 0.0.0.0:11000 by default. Then start the agent with a concrete IP address, e.g.

sudo ./otel-profiling-agent -collection-agent=127.0.0.1:11000 -disable-tls -verbose

rockdaboot avatar Apr 30 '24 15:04 rockdaboot

I'm getting error when performing sudo ./otel-profiling-agent -collection-agent=0.0.0.0:11000 -disable-tls -verbose

@nikolai-kirov Please start devfiler first - it listens on 0.0.0.0:11000 by default. Then start the agent with a concrete IP address, e.g.

sudo ./otel-profiling-agent -collection-agent=127.0.0.1:11000 -disable-tls -verbose

I think I'm failing to understand this. I assumed after I install the agent I'll be able to see the data in Kibana and not some local GUI tool. Is this not the case or I'm misunderstanding?

When I try to start devfiler I get the following error:

./devfiler-appimage-x86_64.AppImage --help
2024-05-01T13:27:33.153009Z  INFO devfiler::collector: Collector listening on 0.0.0.0:11000
thread 'main' panicked at src/main.rs:29:35:
called `Result::unwrap()` on an `Err` value: WinitEventLoop(Os(OsError { line: 786, file: "/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/winit-0.29.15/src/platform_impl/linux/mod.rs", error: Misc("neither WAYLAND_DISPLAY nor WAYLAND_SOCKET nor DISPLAY is set.") }))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
/nix/store/n2769lvhb1g27lgg9bh857df884168xl-devfiler-appimage/bin/devfiler-appimage: line 3: 16224 Aborted       (core dumped) /nix/store/f004k9qdr6ms8xq8gq6qswwg076d1bfr-devfiler-stripped/bin/devfiler "$@"

I suppose its attempting to start the GUI, but I'm on a headless Ubuntu machine on AWS. Is it possible to start it without a gui and have that displayed in Kibana?

nikolai-kirov avatar May 01 '24 10:05 nikolai-kirov

The collection agent that is part of the Elastic stack currently does not yet support the OTel protocol implemented in the open-source variant of the profiling agent. You have two options here:

  • Use our closed-source agent with an Elastic deployment and visualize data in Kibana. The closed-source agent is functionally equivalent to the open-source agent, except that it speaks our old proprietary protocol understood by the Elastic profiling collector. We are working on also teaching the Elastic collector to speak the new protocol, but it's not done yet.
  • Run devfiler locally and tunnel port 11000 on your remote machine to your local machine with an ssh reverse tunnel.

For the latter variant, let me quote the documentation that is also displayed within the devfiler UI:

Profiling on remote hosts

A common use-case is to ssh into and run the profiling agent on a remote machine. The easiest way to set up the connection in this case is with a ssh reverse tunnel. Simply run devfiler locally and then connect to your remote machine like this:

ssh -R11000:localhost:11000 someuser@somehost

This will cause sshd to listen on port 11000 on the remote machine, forwarding all connections to port 11000 on the local machine. When you then run the profiling agent on the remote and point it to 127.0.0.1:11000, the connection will be forwarded to your local devfiler.

athre0z avatar May 01 '24 15:05 athre0z