meltano
Log warning when tap outputs invalid SCHEMA or RECORD message
Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/2488
Originally created by @DouweM on 2020-12-30 21:32:48
For example:
- a
SCHEMAmessage with properties without atype/anyOf: https://gitlab.com/meltano/meltano/-/issues/2485 - a
SCHEMAmessage with multiple mutually incompatible types for one property: https://gitlab.com/meltano/tap-facebook/-/issues/7 - a
RECORDmessage not matching theSCHEMAfor the same stream
Some targets may handle be able to handle these misformed messages correctly by working around their lack of compliance with the Singer spec (https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md#schema-message), but Meltano can log to aid in debugging when targets do fail.
Once we have https://gitlab.com/meltano/meltano/-/issues/2469, users will be able to work around these issues by manually providing a correct schema using https://meltano.com/docs/plugins.html#schema-extra. Of course, fixing the tap is still preferable, but bandaids have their use.
Message validation should be handled asynchronously and should not slow down throughput of messages from tap to target. We'll probably want a setting to opt-in/out.
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.
