Consuming High CPU in 3.2.1

[peraiahkarnam]

HI ,

1. we have more than 150 microservices and more than 1050 instances .
2. spring boot admin server 2.2.2 version is running and cpu consumption is below 5 % .
3. As part of the spring boot upgrade , we upgraded spring boot admin server to 3.2.1 version with java 17 .
4. Deployed SBA with default values , we see cpu consumption is more than 99 % .
5. Added below properties with custom values , But still we see cpu consumption is more than 99 % .

"spring.boot.admin.client.connect-timeout":"20000",
"spring.boot.admin.monitor.default-timeout":"20000",
"spring.boot.admin.client.period":"900000",
"spring.boot.admin.client.read-timeout":"20000",
"spring.boot.admin.monitor.status-interval":"300000",
"spring.boot.admin.monitor.status-lifetime":"300000"

Thanks in advance , help would be appreciated .

[erikpetzold]

Hi @peraiahkarnam

it's very hard to reproduce and analyze this issue. I tried with 200 instances and admin server is at very low cpu usage.

We have some ideas:

• you could try to analyze the app with jprofiler
• maybe the memory is too low and the garbage collector is running all the time
  • probably you have some monitoring for this, SBA can also monitor itself and show these things at "Insights" page
• is there some error leading to a retry loop? Have you checked the logs?
• is SBA showing the instances? Is discovery/registration successfull?
• Does it work if only some of the instances are monitored?
• can you provide a thread dump?

Additionially it would be nice if you could provide some more information:

• how/where do you run the services? docker, kubernetes, vm, ...
• how is registration/discovery working?
• which Java version do you use with SBA 2?

[peraiahkarnam]

HI @erikpetzold ,

Hi @peraiahkarnam

it's very hard to reproduce and analyze this issue. I tried with 200 instances and admin server is at very low cpu usage.

We have other environment where 233 instances running and cpu is very low like 1% .

We have some ideas:

• you could try to analyze the app with jprofiler

we are analyzing with dyantrace .

• maybe the memory is too low and the garbage collector is running all the time

  • probably you have some monitoring for this, SBA can also monitor itself and show these things at "Insights" page

4GB memory we applied

• is there some error leading to a retry loop? Have you checked the logs?

We dont see any loop.

• is SBA showing the instances? Is discovery/registration successfull?

services are registering slowly successfully but few of them are in RESTRICTED mode

• Does it work if only some of the instances are monitored?

we 1055 instances

• can you provide a thread dump?

I can not , due to policies

Additionially it would be nice if you could provide some more information:

• how/where do you run the services? docker, kubernetes, vm, ...

VM

• how is registration/discovery working?

services are registering slowly successfully but few of them are in RESTRICTED mode

• which Java version do you use with SBA 2?

java 11

pls do let me know if any

[erikpetzold]

Sorry we cannot help you, this seems to be an issue related to your setup and with the little amount of information you gave us we cannot analyze anything here.

If you have any specific thing that points to spring boot admin being the problem please open a new ticket and provide as much information as possible.

You could also try to update in smaller steps, e.g. from 2.2 to 2.7 first.

[peraiahkarnam]

@erikpetzold , Thank you for your time .

We are good now with cpu % , and it is comparable with previous versions .

we added below properties as part of the sb upgrade and updated default values on requirement.

"spring.boot.admin.monitor.status-interval":"90000", "spring.boot.admin.monitor.default-timeout":"20000"

Hope this will help you to help others as well !!!

Have a great day !!