Introduce fan-in producers

[josevalim]

This feature will allow developers to invoke a Broadway pipeline from ad-hoc processes. For example, if you have a Cowboy handler, you may want to use Broadway features from each request process.

Fan-in producers will be specified by passing a tuple as a producer option:

producer: {:fan_in, ...opts}

If a fan_in producer is used, Broadway.producer_names() will return an empty list. Broadway.push_messages can be used to push messages to a fan_in producer. The prepare_messages and handle_message callbacks will be invoked from the caller itself and then the messages will be sent to the batchers, if any.

Questions

Today producers are specified with producer: [module: {Mod, opts}, ...]. I have chosen for producer: {:fan_in, ...}] to be its own tuple exactly because when fan_in is used, there are no actual producers. Another option is to specify fan_in as a separate key:

fan_in: [...opts]

The :producer and :fan_in keys would be mutually exclusive, only one of them can be given at a time.

Any other suggestions for the definition API?

[jhalmeida]

I like the producer: {:fan_in, ...} implementation since it conceptually links the entry point for a developer regardless of whether it's a producer or a fan_in producer, which seems easier to reason about to me. The distinction with producer_names() returning an empty list could be confusing though. Could producer_names() include the :fan_in process name or pid, indicating the distinction?

[rewritten]

Using DummyProducer has the same user-facing effect, if I understand it correctly, as one can send a message using Broadway.push_message, and the message will be handled. I wonder if we can just have an alias or almost equivalent NoneProducer that produces nothing, or just document the usage of DummyProducer for this use case?

Note: I might be completely wrong about the equivalence though... :)

[josevalim]

No longer in the plans for now :) And yes, a DummyProducer would do just as fine!