CPU usage is pretty high on only one node and query is slow

[njalan]

Trino version is 394. I found that one node cpu is high out of 8 workers. Below are the top thread take the cpu: LANG=C ps -eT -o%cpu,pid,tid,ppid,comm | grep -v CPU | sort -n -r | head -20 35.5 22151 22165 1 C2 CompilerThre 33.9 22151 27230 1 C2 CompilerThre 13.5 22151 27713 1 C1 CompilerThre 10.6 22151 22192 1 GC Thread#9 10.6 22151 22188 1 GC Thread#5 10.6 22151 22186 1 GC Thread#3 10.6 22151 22185 1 GC Thread#2 10.6 22151 22184 1 GC Thread#1 10.5 22151 22195 1 GC Thread#12 10.5 22151 22194 1 GC Thread#11 10.5 22151 22193 1 GC Thread#10 10.5 22151 22191 1 GC Thread#8 10.5 22151 22190 1 GC Thread#7 10.5 22151 22189 1 GC Thread#6 10.5 22151 22187 1 GC Thread#4 10.5 22151 22153 1 GC Thread#0 9.8 22151 26104 1 20220921_130104 9.5 22151 24133 1 20220921_130055 9.4 22151 24165 1 20220921_130053 9.3 22151 22983 1 20220921_130058

top -Hp 22151

top - 20:55:19 up 26 days, 5:47, 1 user, load average: 117.77, 113.58, 63.38 Threads: 615 total, 32 running, 583 sleeping, 0 stopped, 0 zombie %Cpu0 : 41.4 us, 50.4 sy, 0.0 ni, 8.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 47.4 us, 48.9 sy, 0.0 ni, 2.3 id, 0.0 wa, 0.0 hi, 1.5 si, 0.0 st %Cpu2 : 28.8 us, 60.6 sy, 0.0 ni, 7.6 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st %Cpu3 : 27.5 us, 64.9 sy, 0.0 ni, 7.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 30.8 us, 58.6 sy, 0.0 ni, 7.5 id, 0.0 wa, 0.0 hi, 3.0 si, 0.0 st %Cpu5 : 31.6 us, 61.7 sy, 0.0 ni, 6.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu6 : 28.4 us, 59.0 sy, 0.0 ni, 10.4 id, 0.0 wa, 0.0 hi, 2.2 si, 0.0 st %Cpu7 : 26.9 us, 65.7 sy, 0.0 ni, 7.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu8 : 32.6 us, 59.1 sy, 0.0 ni, 6.8 id, 0.0 wa, 0.0 hi, 1.5 si, 0.0 st %Cpu9 : 41.7 us, 50.8 sy, 0.0 ni, 7.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu10 : 31.8 us, 60.6 sy, 0.0 ni, 5.3 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st %Cpu11 : 36.4 us, 56.8 sy, 0.0 ni, 6.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu12 : 28.0 us, 63.6 sy, 0.0 ni, 6.1 id, 0.0 wa, 0.0 hi, 2.3 si, 0.0 st %Cpu13 : 35.6 us, 59.8 sy, 0.0 ni, 4.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu14 : 43.2 us, 48.5 sy, 0.0 ni, 6.8 id, 0.0 wa, 0.0 hi, 1.5 si, 0.0 st %Cpu15 : 50.8 us, 43.9 sy, 0.0 ni, 5.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 13186292+total, 74778560 free, 39433488 used, 17650880 buff/cache KiB Swap: 3145724 total, 3115508 free, 30216 used. 91553264 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22492 root 20 0 124.8g 35.7g 23196 R 73.8 28.4 4:00.99 C2 CompilerThre 22165 root 20 0 124.8g 35.7g 23196 D 37.7 28.4 4:01.74 C2 CompilerThre 22245 root 20 0 124.8g 35.7g 23196 D 30.0 28.4 0:56.85 20220921_125457 25745 root 20 0 124.8g 35.7g 23196 R 23.8 28.4 0:23.35 C2 CompilerThre 24170 root 20 0 124.8g 35.7g 23196 S 21.5 28.4 0:09.65 20220921_125507 22262 root 20 0 124.8g 35.7g 23196 D 19.2 28.4 0:54.17 20220921_125427 22651 root 20 0 124.8g 35.7g 23196 R 18.5 28.4 3:49.26 C2 CompilerThre 24142 root 20 0 124.8g 35.7g 23196 D 14.6 28.4 0:07.63 20220921_125507 22271 root 20 0 124.8g 35.7g 23196 D 13.8 28.4 0:35.48 task-management 22265 root 20 0 124.8g 35.7g 23196 D 13.1 28.4 0:45.00 20220921_125508 24453 root 20 0 124.8g 35.7g 23196 D 13.1 28.4 0:22.24 20220921_125508 22599 root 20 0 124.8g 35.7g 23196 D 12.3 28.4 0:12.11 http-worker-280 22275 root 20 0 124.8g 35.7g 23196 S 11.5 28.4 0:39.22 Task-20220921_1 22568 root 20 0 124.8g 35.7g 23196 D 11.5 28.4 0:14.20 Task-20220921_1 22606 root 20 0 124.8g 35.7g 23196 R 11.5 28.4 0:35.40 task-notificati 25599 root 20 0 124.8g 35.7g 23196 R 11.5 28.4 0:01.82 20220921_125508 22282 root 20 0 124.8g 35.7g 23196 D 10.8 28.4 0:12.95 http-worker-104 22537 root 20 0 124.8g 35.7g 23196 D 10.8 28.4 0:14.16 http-worker-220 22552 root 20 0 124.8g 35.7g 23196 D 10.8 28.4 0:11.49 Task-20220921_1 22609 root 20 0 124.8g 35.7g 23196 R 10.8 28.4 0:35.95 Task-20220921_1 22241 root 20 0 124.8g 35.7g 23196 D 10.0 28.4 0:57.34 20220921_125507 22572 root 20 0 124.8g 35.7g 23196 D 10.0 28.4 0:11.63 http-worker-262 22642 root 20 0 124.8g 35.7g 23196 D 10.0 28.4 0:40.82 Task-20220921_1 22449 root 20 0 124.8g 35.7g 23196 D 9.2 28.4 0:13.31 Task-20220921_1 22491 root 20 0 124.8g 35.7g 23196 D 9.2 28.4 0:11.60 Task-20220921_1 22761 root 20 0 124.8g 35.7g 23196 D 9.2 28.4 0:06.14 page-buffer-cli 22764 root 20 0 124.8g 35.7g 23196 S 9.2 28.4 0:04.82 page-buffer-cli

Below is the jstack:

jstack 22151 | grep 57dc -A30

"C2 CompilerThread2" #9 daemon prio=9 os_prio=0 cpu=249980.15ms elapsed=684.40s tid=0x00007efca44bad70 nid=0x57dc runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Compiling: 58698 % 4 org.eclipse.jetty.http.HttpParser::parseLine @ 801 (2072 bytes)

"http-worker-176" #176 prio=5 os_prio=0 cpu=12681.45ms elapsed=684.40s tid=0x00007efb74006610 nid=0x57dd runnable [0x00007efabf6f4000] java.lang.Thread.State: RUNNABLE at java.nio.HeapByteBuffer.([email protected]/HeapByteBuffer.java:64) at java.nio.ByteBuffer.allocate([email protected]/ByteBuffer.java:363) at sun.nio.cs.StreamDecoder.([email protected]/StreamDecoder.java:239) at sun.nio.cs.StreamDecoder.forInputStreamReader([email protected]/StreamDecoder.java:95) at java.io.InputStreamReader.([email protected]/InputStreamReader.java:130) at java.nio.file.Files.newBufferedReader([email protected]/Files.java:2922) at java.nio.file.Files.newBufferedReader([email protected]/Files.java:2955) at jdk.internal.platform.CgroupUtil.lambda$readStringValue$1([email protected]/CgroupUtil.java:67) at jdk.internal.platform.CgroupUtil$$Lambda$36/0x0000000800c5f300.run([email protected]/Unknown Source) at java.security.AccessController.executePrivileged([email protected]/AccessController.java:807) at java.security.AccessController.doPrivileged([email protected]/AccessController.java:569) at jdk.internal.platform.CgroupUtil.readStringValue([email protected]/CgroupUtil.java:69) at jdk.internal.platform.CgroupSubsystemController.getStringValue([email protected]/CgroupSubsystemController.java:65) at jdk.internal.platform.CgroupSubsystemController.getLongValue([email protected]/CgroupSubsystemController.java:124) at jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getLongValue([email protected]/CgroupV1Subsystem.java:189) at jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getCpuShares([email protected]/CgroupV1Subsystem.java:245) at jdk.internal.platform.CgroupMetrics.getCpuShares([email protected]/CgroupMetrics.java:75) at com.sun.management.internal.OperatingSystemImpl$ContainerCpuTicks.getContainerCpuLoad([email protected]/OperatingSystemImpl.java:86) at com.sun.management.internal.OperatingSystemImpl.getCpuLoad([email protected]/OperatingSystemImpl.java:271) at com.sun.management.OperatingSystemMXBean.getSystemCpuLoad([email protected]/OperatingSystemMXBean.java:153) at io.trino.server.StatusResource.getStatus(StatusResource.java:92) at jdk.internal.reflect.GeneratedMethodAccessor371.invoke(Unknown Source) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke([email protected]/Method.java:568) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)