nsq
nsq copied to clipboard
nsqio
nsqd: server side message throttling
Note: I'm primarily capturing this idea, I have no short term plan to implement this but want to put it out there.
I've long wanted an ability to set a server side max-rate that messages would be handed out. This would make nsq even more operationally friendly for handling situations where you might want to queue a burst but do work at a steady rate so as not to overwhelm downstream systems (or where you want to mitigate temporary downstream capacity by manually throttling up the processing rate). I'm thinking of a nsqd controlled feature (exposed through nsqadmin) that is managed at runtime.
Related Existing Features:
The built in consumer backoff in go-nsq and pynsq provide some aspect of this, but it's not dynamically controllable, and it's designed primarily around pausing processing due to hard failures not at controlling a steady state.
Existing controls for consumer RDY count and max-in-flight address concurrency but not processing rate. It does however control an upper bound on processing rate because a round trip time is involved in message processing (ie: if you have a RTT of 100ms and a in-flight of 1, you can only process 10 messages/sec even if processing is instantaneous.)
Challenges:
As described this is an aspect of global state that nsqd so far manages to avoid in aiming for distributed decentralized messaging. This could be mitigated slightly by thinking about it in terms of a setting for rate per channel per nsqd. That would be a consistent value regardless of the cluster size growing or shrinking.
Amazing ! I need that, some time , we want to control the processing rate where the downstream systems having a dangours load. Do we have a time plan about that?
+1
would be incredibly useful for our usage. per-channel would satisfy what i'd want to do with the feature.
I'm also :+1:, it would be a great operational tool.
I think the best option, before something like #379 lands, is for it to behave like paused state in terms of individual nsqd persistence and be adjustable via nsqadmin
The amount of message traffic a client can handle should be determined by the client, not the channel producer. The main reason being that many clients that connect to the channel may be able to handle differing amounts of traffic. You would therefore be able to starve one client in order to not overwhelm another.
@devdazed this idea/feature isn't be about the producer setting consumption rate limits, this is about administrative/operational controls to throttle consumers. (To that extent i'm not sure it would make sense for the consumer to be able to set this either as i'm thinking of a more nsqd level setting that applies to all consumers on a channel). I agree that the producer should continue to be agnostic about anything related to specific channels. That nature of decoupling consumers from produce is important.
So you see this as something set at runtime, possible from the admin panel? To throttle a specific channel as some sort of damage control. Or do you see this as something the client send when it connects to tell the producer to throttle the channel?
The former is what i was thinking of. (Thought there might be good use cases for the later; i hadn't really thought about that)
I'm a :+1: on the former, I think having operational control at that level is a great idea.
The latter case is what I was attempting to prevent. In that case, a client should not be able to control how other clients receive messages. There could be a lot of contention in the event that one client's configuration differs. Any setting that controls how a client receives messages really should be client side because hardware and use-cases could differ. One instance I can think of is when there is two clients consuming a single channel but one is on a 32 core machine and the other on a 4 core machine.
:+1: We have a real scenario where a Consumer can only call a downstream service every so often (let's say 1/s). We have multiple nsqds and multiple Consumers on this channel for redundancy but overall we only want to process 1/s for the whole cluster. I think if we multiplied the rate limit by the number of nsqd nodes we'd be okay; meaning we can keep this at the nsqd level and not have to coordinate anything, although coordinate would be ideal if nsqds ever talk to each other. This use case lends itself toward an nsqd controlled throttling mechanism.
Something like tokenbucket might be a good start.
I'd be interested in picking this up if there's still interest. We'd intended on setting a RDY 1 and sleeping in the handler to achieve rate limiting :ghost: (boo).
@judwhite I think we absolutely want to do this, however for your use case wouldn't it make more sense for the downstream system to enforce its limit?
@mreiferson Yes, they should utilize some kind of rate limiting or resource pooling. Not my :circus_tent:, not my :hear_no_evil: :see_no_evil: :speak_no_evil:.
+1
would be incredibly useful for our usage. per-channel would satisfy what i'd want to do with the feature.
I think this is really useful! Proposal add in nsqadmin.
I cannot reset RDY # when consuming, it's no use to send RDY # again, had to disconnect and reconnect...