singer-tools icon indicating copy to clipboard operation
singer-tools copied to clipboard

singer-check-tap ignores missing properties

Open lukaepti opened this issue 3 years ago • 0 comments

Problem: singer-check-tap ignores RECORDs with missing properties, even though their SCHEMA does not allow said properties to be null

Expected behavior: singer-check-tap declares output invalid, or at least warns about records being at odds with the schema

Example:

{"type": "SCHEMA", "stream": "stream", "schema": {"type": "object", "properties": {"id": {"type": "number"}}}, "key_properties": ["id"]}
{"type": "RECORD", "stream": "stream", "record": {}, "time_extracted": "2021-02-17T22:15:35.591070Z" } 

In this simple example, the schema declares id's type to be string (and only string). In fact, it declares it to be a key_property. The record, however, is empty - it doesn't contain id nor any other property. singer-check-tap considers this valid (returns success) as evidenced by this output:

Checking stdin for valid Singer-formatted data
The output is valid.
It contained 2 messages for 1 streams.

      1 schema messages
      1 record messages
      0 state messages

Details by stream:
+--------+---------+---------+
| stream | records | schemas |
+--------+---------+---------+
| stream | 1       | 1       |
+--------+---------+---------+

lukaepti avatar Feb 17 '21 22:02 lukaepti