[CARBONDATA-754] improve performance when order by prefix columns of mdk + limit

[mayunSaicmotor]

the improvement scenario is for order by prefix columns of mdk + limit

1. order by prefix columns of mdk asc + limit
2. order by prefix columns of mdk desc + limit
3. order by prefix columns of mdk asc + limit + filter
4. order by prefix columns of mdk desc + limit + filter

the logical is to leverage the mdk sort feature to get the sorted data. The performance is much better. for example, order by prefix columns of mdk + limit on 20,000,000 data, the performance can be from 10s to about 1s. if do not want to use this feature, can also set CarbonCommonConstants.ORDER_BY_MDK_OPTIMIZATION_FLG = false.

[asfbot]

Can one of the admins verify this patch?

[asfbot]

Can one of the admins verify this patch?

[asfbot]

Can one of the admins verify this patch?

[CarbonDataQA]

Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1435/

[asfbot]

Can one of the admins verify this patch?

[asfbot]

Can one of the admins verify this patch?

[asfbot]

Can one of the admins verify this patch?

[CarbonDataQA]

Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1439/

[asfbot]

Can one of the admins verify this patch?

[asfbot]

Can one of the admins verify this patch?

[asfbot]

Can one of the admins verify this patch?

[CarbonDataQA]

Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1452/

[CarbonDataQA]

Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1455/

[CarbonDataQA]

Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1456/

[CarbonDataQA]

Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1457/

[CarbonDataQA]

Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1494/

[asfbot]

Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/carbondata-pr-spark-1.6/64/

Build result: ABORTED

[...truncated 158.49 KB...] at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.MojoFailureException: Timed out after 0 seconds waiting for forked process to complete. at org.scalatest.tools.maven.AbstractScalaTestMojo.runForkingOnce(AbstractScalaTestMojo.java:319) at org.scalatest.tools.maven.AbstractScalaTestMojo.runScalaTest(AbstractScalaTestMojo.java:242) at org.scalatest.tools.maven.TestMojo.execute(TestMojo.java:106) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) ... 31 more[ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn -rf :carbondata-sparkBuild was abortedAborted by chenliang613channel stoppedSetting status of 8119fa3b2a016375671a8d580da1d9f9c7ed4638 to FAILURE with url https://builds.apache.org/job/carbondata-pr-spark-1.6/64/ and message: 'Build finished. 'Using context: Jenkins (Spark 1.6): Maven clean install Test FAILed.

[asfbot]

Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/carbondata-pr-spark-2.1/241/

Failed Tests: 1

carbondata-pr-spark-2.1/org.apache.carbondata:carbondata-spark-common-test: 1

• org.apache.carbondata.spark.testsuite.deleteTable.TestDeleteTableNewDDL.drop table and create table with dictionary exclude string scenario

Test Failed.

[CarbonDataQA]

Can one of the admins verify this patch?

[asfgit]

Can one of the admins verify this patch?

[asfgit]

Can one of the admins verify this patch?

[CarbonDataQA]

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/86/

[CarbonDataQA]

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/625/

[ravipesala]

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1256/

[CarbonDataQA]

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3098/

[CarbonDataQA]

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4017/

[CarbonDataQA]

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7164/

[CarbonDataQA]

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5940/

[CarbonDataQA]

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7435/

[CarbonDataQA]

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/93/