Take Thread-Dump or Heap-Dump of K8s Pod

[rr-paras-patel]

How do we take thread-dump, heap-dump and Container capture in certain condition before pod die for debugging purpose for certain bugs or scenario in kubernetes ? Might be we can utilize some event hooks while pod is getting killed.

[missedone]

and sometime it gets even harder to create the heap dump when we have resources request/limits specified on the container, the dump tool itself consume a lot resource at a peak and even exceed the resource limit threshold which trigger kubernetes restart that pod, therefore we could lose the heap dump. is there any tool/solution for proper dump generating?

[chennv4]

try kill -3 1 it will print the thread dump in output without killing the pod

[snagaram3]

Kubectl -n $NameSpace exec -it $pod /bin/sh

Using Jmap, Jstack jmap -dump:format=b,file=/tmp/heapdump 1 jstack 1 > /tmp/threaddump

[dragonsoft-vincent]

https://danlebrero.com/2018/11/20/how-to-do-java-jvm-heapdump-in-kubernetes/

[madhifallah]

I was looking for a way to generate a threadump right before pod get killed/restart , I found this :

    lifecycle:
      preStop:
        exec:
          command:
            - "/bin/sh"
            - "-c"
            - "jstack 1 > /var/log/jvm/jvm-thread-$(date +'%Y%m%d%H%M').log" 
            - "sleep 2m"

source : https://habeeb-umo.medium.com/capturing-and-uploading-jvm-thread-dumps-or-other-logs-in-a-kubernetes-cluster-with-fluent-bit-5fc2a5e6b5fd

[aantn]

This thread is a little old, but we added Java troubleshooting actions to Robusta.dev to do this.

It's all open source.

[Yagyansh]

@aantn

How can I trigger Robusta action to take a heap dump when a POD goes OOM or restarts and save the dump somewhere before the POD comes up? Because from your smart actions I can see crash_loop trigger but that is when the pod is already in crash loop state, by then it won't be possible to take the dump.

Any option where the termination signal is read?

[aantn]

@Yagyansh the way OOM Kills work, you don't really get much warning.

Would it work if you fired a Prometheus alert when the pod was at 90% memory usage and ran the heap dump action from there?