Memory Usage Spike

[danishubin]

Hi,

I was able to get Parca up and running and it works great. As a next step we wanted to set resource requests and limits (our goal was to set the limit to 500mib). The problem we are running into, however, is that there are almost always memory spikes in the beginning of the parca-agent pod deployment (v0.39), jumping to between 550mib and 750mib. Then, after a couple minutes, it drops back down to a reasonable level (between 300-400). This means we can't set the resource limit to 500 since it gets the OOM termination error. Of course we can increase the memory limit to ~800mib but ideally we would be able to remove these spikes.

Do you have any idea why they might be happening? Could this be related https://github.com/parca-dev/parca-agent/issues/2995? Or is it something else?

Additionally, this seems to only be happening in parca-agent pods that are profiling a clickhouse binary. Could the large binary size cause this memory spike?

My current setup: Agent:

  - args: 
    - --http-address=:7071
    - --node=$(NODE_NAME)
    - --config-path=/etc/parca-agent/parca-agent.yaml
    - --remote-store-address=parca.parca.svc.{{ .Values.region }}.k8s.:7070
    - --remote-store-insecure
    - --log-level=debug
    - --profiling-cpu-sampling-frequency=19
   ...
    resources:
      requests:
        memory: 100Mi
        cpu: 10m
      limits:
        memory: 500Mi
        cpu: 100m

Example Deployment (showing results for just one Parca-Agent pod): CPU Usage: Memory Spike: