otel-profiling-agent symbolization issues with python applications

I ran into an issue when trying to profile python applications instrumented with python auto instrumentation

after starting the applications, im not seeing python symbols, only symbols of the python binary itself

after restarting the agent pod, the application is correctly symbolized

iv'e tested this both using parca agent and otel-ebpf-profiler

code to reproduce:

src/app.py :

import os
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
import time

import requests


# A CPU-intensive task: calculating Fibonacci numbers recursively
def cpu_intensive_task(n):
    if n <= 1:
        return n
    else:
        return cpu_intensive_task(n - 1) + cpu_intensive_task(n - 2)


class RequestHandler(BaseHTTPRequestHandler):

    def do_GET(self):
        # Start timing
        start_time = time.time()

        # Perform CPU intensive work
        result = cpu_intensive_task(35)

        # End timing
        end_time = time.time()

        # Send HTTP headers
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()

        # Send the response body
        response = f"<html><body><h1>Result: {result}</h1><p>Processed in {end_time - start_time:.2f} seconds</p></body></html>"
        self.wfile.write(response.encode('utf-8'))


# Define server address and port
HOST = os.environ.get('HTTP_HOST') or 'localhost'
PORT = os.environ.get('HTTP_PORT') or 8080


def periodic_request():
    while True:
        try:
            requests.get(f"http://localhost:{PORT}")
        except Exception as e:
            print(f"Request failed: {e}")
        time.sleep(5)

def run():
    server_address = (HOST, PORT)
    httpd = HTTPServer(server_address, RequestHandler)
    print(f"Starting server on {HOST}:{PORT}")
    # Start periodic request thread
    threading.Thread(target=periodic_request, daemon=True).start()

    httpd.serve_forever()

    httpd.serve_forever()


if __name__ == '__main__':
    run()

requirements.txt :

requests
opentelemetry-distro
opentelemetry-exporter-otlp

Dockerfile :

FROM python:3.10-slim
WORKDIR /app
COPY src/app.py .
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN opentelemetry-bootstrap -a install
EXPOSE 8080
CMD ["opentelemetry-instrument", "python" ,"app.py"]

May 04 '25 08:05 NimrodAvni78

Thanks for the report. I tried to reproduce your issue, but wasn't able to only get traces with native frames from libpython:

As you see python frames in some cases, it seems like a sampling issue. Did you try to change the sampling frequency (-samples-per-second for OTel eBPF profiler deaults to 20) to something different? Unfortunately, OTel eBPF profiler uses by default a sampling frequency of 20Hz. Ideally you want to use a sampling frequency that is a prime number to avoid oversampling of samples, that are scheduled at the prime factors of your sampling frequency.

May 05 '25 06:05 florianl

hey @florianl thanks for the response iv'e also tried changing samples-per-second to 19 and it still persisted it didnt happen for you even after starting the application, and then restarting the profiling agent?

May 06 '25 15:05 NimrodAvni78

it didnt happen for you even after starting the application, and then restarting the profiling agent?

Order of execution doesn't matter here. Regardless whether OTel eBPF profiler is started before or after the Python application or vice versa.

May 06 '25 15:05 florianl