singer-tools
singer-tools copied to clipboard
singer-check-tap ignores missing properties
Problem: singer-check-tap ignores RECORDs with missing properties, even though their SCHEMA does not allow said properties to be null
Expected behavior: singer-check-tap declares output invalid, or at least warns about records being at odds with the schema
Example:
{"type": "SCHEMA", "stream": "stream", "schema": {"type": "object", "properties": {"id": {"type": "number"}}}, "key_properties": ["id"]}
{"type": "RECORD", "stream": "stream", "record": {}, "time_extracted": "2021-02-17T22:15:35.591070Z" }
In this simple example, the schema declares id
's type to be string
(and only string
). In fact, it declares it to be a key_property
. The record, however, is empty - it doesn't contain id
nor any other property. singer-check-tap considers this valid (returns success) as evidenced by this output:
Checking stdin for valid Singer-formatted data
The output is valid.
It contained 2 messages for 1 streams.
1 schema messages
1 record messages
0 state messages
Details by stream:
+--------+---------+---------+
| stream | records | schemas |
+--------+---------+---------+
| stream | 1 | 1 |
+--------+---------+---------+