runtime: findrunnableGCWorker consumed much higher CPU under 64c AWS instance (ARM64), comparing to 32c

[innerr]

Go version

go1.19.13 linux/arm64

Output of go env in your module/workspace:

GOARCH="arm64"

What did you do?

• We put one binary (developed/compiled by golang) in an AWS EKS instance, it runs as a gRpc server.
• Lately we changed this instance type, increasing CPU cores from 32c to 64c.

What did you see happen?

• We found that this process had much higher P999 latency comparing to previous.
• From flame graph, We found out findrunnableGCWorker consumed much higher CPU comparing to previous.

Here are the graphs:

• ARM64 32c:
• ARM64 64c:

What did you expect to see?

The latency of this server process shouldn't change much when we changed from 32c instance to 64c instance.