José Valim

You should be able to call `GenServer.stop(BroadwayPIpeline)` but keep in mind that starting and stopping Kafka partitions is asking for trouble because Kafka doesn't deal with nodes quickly going down...

Btw, this discussion made it clear there are different desired approaches to handling failure. If anyone wants to submit pull requests documenting individual techniques, it will be very welcome!

Correct. This is a specific request for more documentation to be added via PRs. :)

Yes, if we depend on master we should be able to rely on the process_name API for the same result. A PR would be welcome!

If you can encapsulate all of the operations, mentioned above, through the channel, then I don't see an issue with this approach. Although note that you will also need an...

Hi @hassox! Our biggest concern with sharing connections is how to reason about failures. If we are going to do sharing, I would probably still prefer to have Broadway itself...

Look at max_demand. By default we send 10 items to each processor. So if you have a prefetch_count of 5, it is likely that all 5 are going to the...

As soon as things arrive it starts sending them down. IRC, prefetch_count is about how many items in flights, not how many items to get before anything starts.

Yes, it is related. Before we get to this point though, have you confirmed that bumping the producer concurrency is beneficial for you? Most times the bottleneck is not in...

Maybe we should add an option to resize a Broadway Pipeline without shutting the prodiucers down (everything else would be restarted though).