carbondata icon indicating copy to clipboard operation
carbondata copied to clipboard

Published 20 hours ago • verified publisher icon apache

[CARBONDATA-3770] Improve partition count star query performance

Open Zhangshunyu opened this issue 4 years ago • 28 comments

Why is this PR needed?

To improve pure partition count star performance

  1. currently the count(*) with filter whose culumns are all partition columns will load datamaps of these partitions including block info/minmax info, but it is no need to load them ,we can just read it from valid index files directly as the rowCount stored inside, and cache these info.

  2. For no-sort partition table, minmax is almost no using but cost time.

What changes were proposed in this PR?

The detail of query flow as following if it is pure partition count star Step 1. check whether it is pure partition count star by filter Step 2. read tablestatus to get all valid segments, remove the segment file cache of invalid segment and expired segment Step 3. use multi-thread to read segment files which not in cache and cache index files list of each segment into memory. If its index files already is eixst in cache, not require to read again. Step 4. use multi-thread to prune segment and partition to get pruned index file list, which can prune most index files and reduce the files num. Step 5. read the count from pruned index file directly and cache it, get from cache if exist in the index_file <-> rowCount map.

Does this PR introduce any user interface change?

  • No

Is any new testcase added?

  • No

Zhangshunyu avatar Apr 09 '20 08:04 Zhangshunyu

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/977/

CarbonDataQA1 avatar Apr 09 '20 10:04 CarbonDataQA1

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2690/

CarbonDataQA1 avatar Apr 09 '20 11:04 CarbonDataQA1

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/980/

CarbonDataQA1 avatar Apr 09 '20 13:04 CarbonDataQA1

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2693/

CarbonDataQA1 avatar Apr 09 '20 14:04 CarbonDataQA1

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/994/

CarbonDataQA1 avatar Apr 10 '20 12:04 CarbonDataQA1

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2707/

CarbonDataQA1 avatar Apr 10 '20 12:04 CarbonDataQA1

@Zhangshunyu : What do you mean pure partition ? is it just normal "partition" ? Also mention what was the bottleneck before in the description.

ajantha-bhat avatar Apr 11 '20 03:04 ajantha-bhat

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1000/

CarbonDataQA1 avatar Apr 11 '20 05:04 CarbonDataQA1

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2712/

CarbonDataQA1 avatar Apr 11 '20 06:04 CarbonDataQA1

@ajantha-bhat We find that select count() for some partitons is time costly and worse than parquet, as currently the count() with filter whose culumns are all partition columns will load all datamaps of these partitions including block info/minmax info, but it is no need to load them ,we can just read it from valid index files directly using partition prune as the rowCount stored inside index files, and we can cache these info. For no-sort partition table, minmax is almost no using but cost time.

Zhangshunyu avatar Apr 13 '20 02:04 Zhangshunyu

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2714/

CarbonDataQA1 avatar Apr 13 '20 05:04 CarbonDataQA1

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1002/

CarbonDataQA1 avatar Apr 13 '20 05:04 CarbonDataQA1

@Indhumathi27 : we are already matching partition first, before the loading min max (your old PR) That was done only for select * flow, not for count(*) flow ?

ajantha-bhat avatar Apr 13 '20 07:04 ajantha-bhat

@ajantha-bhat It was done for select with filter on partition columns., where we will load index for matched partiitons

Indhumathi27 avatar Apr 13 '20 07:04 Indhumathi27

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2722/

CarbonDataQA1 avatar Apr 13 '20 11:04 CarbonDataQA1

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1010/

CarbonDataQA1 avatar Apr 13 '20 11:04 CarbonDataQA1

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1033/

CarbonDataQA1 avatar Apr 15 '20 08:04 CarbonDataQA1

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2746/

CarbonDataQA1 avatar Apr 15 '20 09:04 CarbonDataQA1

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2781/

CarbonDataQA1 avatar Apr 18 '20 09:04 CarbonDataQA1

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1068/

CarbonDataQA1 avatar Apr 18 '20 09:04 CarbonDataQA1

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1733/

CarbonDataQA1 avatar Jul 23 '20 07:07 CarbonDataQA1

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3475/

CarbonDataQA1 avatar Jul 23 '20 08:07 CarbonDataQA1

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3476/

CarbonDataQA1 avatar Jul 23 '20 10:07 CarbonDataQA1

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1734/

CarbonDataQA1 avatar Jul 23 '20 10:07 CarbonDataQA1

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2354/

CarbonDataQA1 avatar Sep 16 '20 08:09 CarbonDataQA1

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4095/

CarbonDataQA1 avatar Sep 16 '20 08:09 CarbonDataQA1

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4096/

CarbonDataQA1 avatar Sep 16 '20 11:09 CarbonDataQA1

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2355/

CarbonDataQA1 avatar Sep 16 '20 11:09 CarbonDataQA1