Broker can stall when stress testing 40 * 10000 messages

[mochi-co]

When running ./mqtt-stresser -broker tcp://localhost:1883 -num-clients=40 -num-messages=10000 the broker appears to occasionally stall, although it does not freeze, and it continues to process other messages as expected.

Testing against the latest https://github.com/fhmq/hmq we find only marginal performance difference.

This raises questions about whether there is a benefit to continue using circular buffers (which are difficult to maintain and control) now that the performance of channels has been improved significantly, or if the circular buffers mechanism should be replaced with a worker pool. This would also alleviate issues discussed in #95 and could potentially reduce the overall number of goroutines as mentioned in #80.

Discussion invited :)

[alexsporn]

Probably a worker per client makes sense, so that slow clients cannot slow down others. Channels are a good alternative to the circular buffer, but they will also stall once full if you don't allow dropping of messages when the buffer fills up.

Maybe a priority queue per client would be the best approach. This way you could prioritize pending control packets over new messages, since new messages will generate even more pressure in form of more control packets.

[mochi-co]

Just a note here that to anyone who comes across this issue - I've begun a fairly significant refactor of the broker which is looking quite promising, so I would caution against making any sizeable PRs for the time being. I hope to resolve some of the longer standing issues we've been experiencing.

[mochi-co]

This, and all issues related this, are resolved in the upcoming v2.0.0 release. I hope to get it ready for pre-release in the near future