spark-expectations icon indicating copy to clipboard operation
spark-expectations copied to clipboard

[FEATURE] Performance improvement

Open pinnapuvijay opened this issue 1 year ago • 0 comments

Is your feature request related to a problem? Please describe. There is an opportunity to improve the performance so the pipelines run times don't get impacted much. The "write error records" and "action" steps are significantly taking longer time than other steps during the runtime.

Describe the solution you'd like The "write error records" and "action" steps to be tuned so they run faster.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

pinnapuvijay avatar Feb 12 '24 15:02 pinnapuvijay