carbondata icon indicating copy to clipboard operation
carbondata copied to clipboard

Published 20 hours ago • verified publisher icon apache

[CARBONDATA-2865] Fix StackOverflowException during compaction in migrated store

Open kunal642 opened this issue 6 years ago • 12 comments

Problem: When compaction is fired on migrated partition table, PartitionSpec object is unnecessarily being serialized to executor which is causing StackOverFlowException.

Solution: Make PartitionSpec as transient to avoid serialization.

Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily:

  • [x] Any interfaces changed?

  • [x] Any backward compatibility impacted?

  • [x] Document update required?

  • [x] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change.

  • [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

kunal642 avatar Aug 28 '18 13:08 kunal642

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6444/

ravipesala avatar Aug 28 '18 13:08 ravipesala

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/56/

CarbonDataQA avatar Aug 28 '18 14:08 CarbonDataQA

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8127/

CarbonDataQA avatar Aug 28 '18 15:08 CarbonDataQA

retest sdv please

jackylk avatar Aug 29 '18 06:08 jackylk

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6449/

ravipesala avatar Aug 29 '18 07:08 ravipesala

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/14/

CarbonDataQA avatar Sep 05 '18 14:09 CarbonDataQA

@kunal642 why does the serialization of ‘PartitionSpec’ cause stackoverflow?

xuchuanyin avatar Sep 07 '18 15:09 xuchuanyin

@xuchuanyin Actually i was never able to find why this stackoverflow exception is thrown but while debugging i realised that a lot of recursive call is occurring when trying to serialize 'PartitionSpec'. After a little bit of searching i found a article which suggested that collections are serialized in a recursive manner. I tried this solution as i wasn't getting any other clue.

I would appreciate any suggestion regarding this.

kunal642 avatar Oct 09 '18 12:10 kunal642

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/61/

CarbonDataQA avatar Jul 24 '19 09:07 CarbonDataQA

Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/60/

CarbonDataQA avatar Jul 24 '19 09:07 CarbonDataQA

Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/61/

CarbonDataQA1 avatar Feb 03 '20 07:02 CarbonDataQA1

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1949/

CarbonDataQA1 avatar Feb 12 '20 04:02 CarbonDataQA1