[3.2.0] High CPU when a single request hits the Gateway on Kubernetes

[HamzaOralK]

Description: High CPU when a single request hits the Gateway.

Followed troubleshooting documentation and findings are like this:

usage result: %CPU CPU NI S TIME PID TID ... 78.1 - 0 R 00:17:27 167 356 77.9 - 0 R 00:17:25 167 357 ...

dump result according to thread ids:

"HTTPS-Listener I/O dispatcher-1" #190 prio=5 os_prio=0 cpu=1043696.51ms elapsed=1336.89s tid=0x00007fdc54094000 nid=0x164 runnable  [0x00007fdc1b606000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPoll.wait([email protected]/Native Method)
        at sun.nio.ch.EPollSelectorImpl.doSelect([email protected]/EPollSelectorImpl.java:120)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:124)
        - locked <0x00000000eafef8d8> (a sun.nio.ch.Util$2)
        - locked <0x00000000eafef778> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:136)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
        at java.lang.Thread.run([email protected]/Thread.java:834)

"HTTPS-Listener I/O dispatcher-2" #191 prio=5 os_prio=0 cpu=1041420.55ms elapsed=1336.89s tid=0x00007fdc5409d000 nid=0x165 runnable  [0x00007fdc1b505000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPoll.wait([email protected]/Native Method)
        at sun.nio.ch.EPollSelectorImpl.doSelect([email protected]/EPollSelectorImpl.java:120)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:124)
        - locked <0x00000000eaff0950> (a sun.nio.ch.Util$2)
        - locked <0x00000000eaff08f8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:136)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
        at java.lang.Thread.run([email protected]/Thread.java:834)

This two threads consuming a lot of resources and I couldn't find a specific reason.

Steps to reproduce: Deploy helm chart on AWS m5a.large worker nodes.

Affected Product Version: API-M version 3.2.0

Environment details (with versions): OS: AWS Linux Env (Docker/K8s): K8s 1.18 Optional Fields Related Issues: Suggested Labels: Suggested Assignees: