conduit
conduit copied to clipboard
Design: Handling invalid data/errors
Feature description
We need to come up with a good design for handling invalid data / errors. If an error happens while processing a message the pipeline currently stops entirely. We need to allow the user to choose a different behavior, e.g. ignore errors, log and continue, reroute to dead letter queue. The design should define how the user specifies the behavior in the API or config file (if we have pipeline config files by then) and how it works behind the scenes.