datayoga
datayoga copied to clipboard
datayoga-io
streaming data pipeline platform
Currently, we validate the block only against its JSON Schema (in the parent `Block`). We should be able to validate the block based on internal logic.
When we get the same key in the same batch, the whole batch will probably fail. First, add a test that verifies it and fails. Then, add a dedup mechanism...
including examples and limitations
the following input fails: {"a": [1,2,3]} fails with Interface error can not bind. Need to catch this exception and cast to string, or enter as null
Support both: ``` if: expression ``` and ``` if: expression: myexpression language: language ``` should also be applied in the relevant blocks as well. use a default for language in...
1. We can choose either one of them (jmespath, sql) as the default. 2. In job level, we can override the default for this job (optional). 3. In the block...
Should be README files with explanation for each example.
So we won't include those by default. We will have requirements per block in the near future, though.