carbondata icon indicating copy to clipboard operation
carbondata copied to clipboard

Published 20 hours ago • verified publisher icon apache

[CARBONDATA-4347] Improve performance when delete empty partition directory

Open shenjiayu17 opened this issue 2 years ago • 19 comments

Why is this PR needed?

When delete multiple segments, carbondata tries to judge if partition directory is empty and delete it on each segments, but partiton path may be the same between two segments. So there is repetitive action.

What changes were proposed in this PR?

Collect a set of partition directories that are from segments to be deleted, judge empty and delete after all segments deleted.

Does this PR introduce any user interface change?

  • No

Is any new testcase added?

  • No

shenjiayu17 avatar Jul 18 '22 04:07 shenjiayu17

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4658/

CarbonDataQA2 avatar Jul 18 '22 05:07 CarbonDataQA2

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6401/

CarbonDataQA2 avatar Jul 18 '22 05:07 CarbonDataQA2

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/787/

CarbonDataQA2 avatar Jul 18 '22 05:07 CarbonDataQA2

retest this please

shenjiayu17 avatar Jul 19 '22 03:07 shenjiayu17

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/788/

CarbonDataQA2 avatar Jul 19 '22 03:07 CarbonDataQA2

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6402/

CarbonDataQA2 avatar Jul 19 '22 03:07 CarbonDataQA2

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4659/

CarbonDataQA2 avatar Jul 19 '22 03:07 CarbonDataQA2

retest this please

shenjiayu17 avatar Jul 19 '22 03:07 shenjiayu17

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/789/

CarbonDataQA2 avatar Jul 19 '22 04:07 CarbonDataQA2

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6403/

CarbonDataQA2 avatar Jul 19 '22 04:07 CarbonDataQA2

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4660/

CarbonDataQA2 avatar Jul 19 '22 04:07 CarbonDataQA2

retest this please

shenjiayu17 avatar Jul 19 '22 08:07 shenjiayu17

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4662/

CarbonDataQA2 avatar Jul 19 '22 11:07 CarbonDataQA2

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6405/

CarbonDataQA2 avatar Jul 19 '22 11:07 CarbonDataQA2

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/791/

CarbonDataQA2 avatar Jul 19 '22 11:07 CarbonDataQA2

retest this please

shenjiayu17 avatar Jul 30 '22 01:07 shenjiayu17

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4665/

CarbonDataQA2 avatar Jul 30 '22 03:07 CarbonDataQA2

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6408/

CarbonDataQA2 avatar Jul 30 '22 03:07 CarbonDataQA2

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/794/

CarbonDataQA2 avatar Jul 30 '22 04:07 CarbonDataQA2

Please fix CI error

xubo245 avatar Feb 09 '23 03:02 xubo245