kaskada
kaskada copied to clipboard
bug: invalid materializations affect subsequent runs
Description If an invalid materialization is created and causes sparrow to return an error during query, it will cause the async process to fail and exit, meaning materializations that have not yet run will not run. This highlights a few issues:
- A failure in a single materialization should not affect other materializations
- Materializations should likely be run in parallel
- Materializations that cause an
Error
in sparrow must be handled in a way that surfaces the error to the user
To Reproduce Create a materialization that causes an error in sparrow. A way to do so without inserting an arbitrary panic in Sparrow is (unless it has since been patched) to supply an invalid topic name to a Pulsar destination.
Actual Behavior Materialization panics, causing the process to exit and other materialization to not be run.
Expected Behavior It's difficult to know what to expect now, since we have no way to surface errors to users. At the least, invalid mats should not affect other mats from running.
related to #80