seatunnel
seatunnel copied to clipboard
apache
[Umbrella] [Dirty data] This is new feature for dirty data
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Search before asking
- [X] I had searched in the issues and found no similar issues.
Describe the proposal
Dirty data is data that is not meaningful to the business, has an illegal format, or is out of sync. A single piece of data is dirty if an exception occurs during writing to the target data source. So any data that fails to write is classified as dirty. For example, data of type VARCHAR on the source side is written to the target column of type INT, resulting in data that can not be written because of improper conversion. You can control whether dirty data is allowed during synchronization when task configuration is synchronized, and support for controlling the number of dirty data bars, that is, when the dirty data exceeds the specified number, the task fails to exit.
Task list
Support for defining dirty data and its impact on tasks
- [ ] When do not allow the dirty data, if produce dirty data synchronization task execution process, the task will fail
- [ ] When allowing dirty data and set its threshold, synchronization task will ignore the dirty data (that is, won't write to the target side), and normal execution.
Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
Can you show me how to achieve this feature？
I will output the detailed design later
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
It would be better if it could support specified tolerance ratio of the dirty data.
It would be better if it could support specified tolerance ratio of the dirty data. Now is not good support and need to provide the collector features first.
flink:https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/metrics/ spark:https://spark.apache.org/docs/2.4.5/rdd-programming-guide.html#shared-variables
When allowing dirty data and set its threshold, synchronization task will ignore the dirty data (that is, won't write to the target side), and normal execution.
If we want achieve this feature, two way: 1. the sink connector should make sure they are not throw exception when meet dirty data. 2. Or we recreate sink connector again. Both have some problem, 1 will be hard to do with all connector, 2 will have many connection to create and close.
When allowing dirty data and set its threshold, synchronization task will ignore the dirty data (that is, won't write to the target side), and normal execution.
If we want achieve this feature, two way: 1. the sink connector should make sure they are not throw exception when meet dirty data. 2. Or we recreate sink connector again. Both have some problem, 1 will be hard to do with all connector, 2 will have many connection to create and close.
I think you can use the first way.Whether can be obtained at a higher level anomalies and then use the accumulator?
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.