Call for comments - Should ConsumerGroup support a back pressure mechanism and how?

[dantswain]

What I mean here by back pressure is dwelling between fetch calls. This is an important part of many production systems where a stream processor may need to pause consuming from an upstream in order to allow some downstream condition to clear.

I can think of two alternative approaches:

1. The consumer could dwell in the handle_message_set call:

use GenConsumer

def handle_message_set(message_set, state) do
  backpressure(need_backpressure?())
   
   # normal message handling below
end

defp need_backpressure?() do
  # call out to check something downstream
end

defp backpressure(false), do: :ok
defp backpressure(true) do
  :timer.sleep(Application.get_env(:my_app, :backpressure_msec)
  backpressure(need_backpressure?())
end

2. Defer this behavior to pull-based consumers (see #230). A pull-based consumer can implement any kind of dwell/fetch loop it needs.

On the other hand, I think back pressure is an important concept for people to understand and if the library is written in a way that encourages people to consider this in their systems, then that would be A Good Thing. A couple ways that this could be implemented:

1. A back pressure callback option given to GenConsumer on startup. Similar to the way that the partition assignment callback is provided. The callback would be expected to return a number of milliseconds that we should dwell before calling fetch again. The default callback could simply return 0.

2. An actual behavior callback in GenConsumer. This would work similarly to (1) except it would get @behaviour compile-time benefits. We can provide the default implementation via the use macro.

3. Allow the handle_message_set function to return a {sync_status, consumer_state, dwell_msec} form. This would behave similarly to how GenServer's various handle_* functions can return a timeout.

After writing all of this out, I am leaning towards supporting back pressure and doing so via a behavior callback. I think this would help encourage people to consider this in their designs while still being unobtrusive for people who don't need it.

[joshuawscott]

Having worked with a number of backpressure systems, I feel as though it may be overcomplicating things to provide backpressure as part of the consumer group behaviour. When I've used these systems in the past (and currently), I've discovered having behaviours that aren't required leads to very hard-to-follow code (you have to read & understand the ordering in the underlying module to understand how your code works), so I would prefer if we just leave backpressure to the user to handle.