dolphinscheduler
dolphinscheduler copied to clipboard
[DSIP-79][Task] Add Datavines task to better support data quality
Search before asking
- [X] I had searched in the DSIP and found no similar DSIP.
Motivation
DataVines is an easy-to-use data quality service platform that supports multiple metric. https://github.com/datavane/datavines
- Datavines supports executing multiple metrics in one job.
- Datavines supports execution status dashboard and data quality report.
- Datavines supports plug-in extensions for components such as metric, data sources, error data storage, and execution engines.
- Jdbc engines can be used to execute data quality tasks instead of solely relying on Spark engines.
Design Detail
Sript mode
-
config data quality job in datavines
-
get the job config scipt file
-
Add datavines job node in workflow, and configure the script
API Mode
-
config data quality job in datavines
-
get the jobId
-
Add datavines job node in workflow, and configure the datavines api address and jobId
Compatibility, Deprecation, and Migration Plan
No response
Test Plan
No response
Code of Conduct
- [X] I agree to follow this project's Code of Conduct