tobychui
[HELP] High CPU usage when high traffic load
What happened?
Hello!
There is a problem that worries me a little.
When sending a synthetic load to the server (GET / HTTP/1.1), the CPU consumption increases many times (reaching 80% of the active consumption of all cores).
I am trying through wrk with 1024 simultaneous connections per second, 100 threads.
Can you please tell me which module should I implement a debugger in to see where the server's "bottleneck" is?
Describe what have you tried A clear and concise description of what you expect to see and what you have tried to debug it.
Describe the networking setup you are using Here are some example, commonly asked questions from our maintainers:
- Are you using the docker build of Zoraxy? [yes (with docker setup & networking config attach) /no] no
- Your Zoraxy version? [e.g. 3.0.4] 3.2.5
- Are you using Cloudflare? [yes/no] no
- Are your system hosted under a NAT router? [e.g. yes, with subnet is e.g. 192.168.0.0/24 and include port forwarding config if any] noi
- DNS record (if any) n/a
Additional context Add any other context or screenshots about the feature request here.
It would be great if you could do some profiling to help narrow down the culprit
It would be great if you could do some profiling to help narrow down the culprit
Hi! I understand that this is what needs to be done, but my attempts to insert a profiler to see what the matter was were unsuccessful, so I asked for help in finding the main preload module where I can insert a profiler to find the cause
Sorry, must've missed that in the email.
Not entirely sure how it's done w/ go, but in Rust I would profile the process w/ ptrace and then dump the output into a tool like this https://github.com/flamegraph-rs/flamegraph to generate a flamegraph I could use to identify which functions are eating up the most CPU time, from there I would look closer at those functions (either the implementation itself, or where it's being used (for instance, locking a mutex might be eating up a lot of time, but that's not because the mutex needs to be optimized it's more because whatever is using the mutexs needs to be looked at)) to identify possible culprits.
You can then test any ideas you have with benchmarking.
As an example, I used this strategy to optimize event notification in my music player https://github.com/AnthonyMichaelTDM/mecomp/pull/210
As for what to actually profile, I'd see if you get the same behavior with other synthetic workloads:
- hitting an endpoint that proxies to a remote origin
- hitting an endpoint for a virtual directory
- ... that has nothing behind it (close connection)
- ... from a blacklisted IP
- ... from a whitelisted IP
- etc.
If the behavior is the same everywhere, it might be the routing code or access rule checking, otherwise if it's only in one or two situations, then it's probably whatever those situations cause to get run.
As mentioned by @AnthonyMichaelTDM , there are many routine in a request handler that might lead to this issue. Are you using slow geoIP resolve mode? That trades off RAM usage with CPU time.
If it is not the above's, I guess this is just the downside of dynamic routing design used in Zoraxy. Compiled / static routing configured proxy implementation is much better in efficiency but lack the flexibility for dynamically adjusting the proxy endpoints in runtime and require a restart every time a config changed.
