Grayproxy high cpu load

[edillmann]

Hi,

Grayproxy is using high cpu load when disc queuing is configured, even with a very small throughput (2 msg/s)

Regards, Eric

[andviro]

Hi. I'll look into it. Any particular details, Go version, docker/no docker, command line flags?

[edillmann]

Hi

go version 1.10 / no docker

grayproxy -in udp://0.0.0.0:12201 -out tcp://10.0.10.27:12201 -dataDir=/var/tmp without any message flowing through give high cpu

grayproxy -in udp://0.0.0.0:12201 -out tcp://10.0.10.27:12201 does not show this problem.

[andviro]

Yep, there seems to be a problem with current implementation. Will look into it asap.

[andviro]

Tuned go-diskqueue parameters a little in 4880498d. On my PC it amounts to 10 percent load or so.

[bangert]

i think i have the same problem. we have used grayproxy for a while now, but so far without diskqueue.

with diskqueue enable the cpu utilisation is fine, except when grayproxy looses the connection to its upstream.

i have about 1 msg/s going through grayproxy. cpu utilisation is about 0.4 percent. when i remove upstream, and grayproxy needs to queue to disk utilisation goes upwards of 70%. iowait also goes to 10%. i will investigate this further.

[andviro]

Yeah, there's that. I'm searching for decent embedded queue/ring buffer or something, that works well out-of the box. Will be grateful for recommendations.

[bangert]

in #3 i pointed out https://github.com/beeker1121/goque https://github.com/peterbourgon/diskv looks like it also may work (and has an interesting storage format). it would require incremental keys, though.

i have no experience with either of them, though.

[andviro]

Yes, thanks. I hoped that you found something battle-tested by now. I beginning to think that something more brutal and simple might work, like https://github.com/cloudflare/buffer.