incubator-livy icon indicating copy to clipboard operation
incubator-livy copied to clipboard

Published 20 hours ago • verified publisher icon apache

[LIVY-336] Livy should not spawn one thread per job to track the job on Yarn

Open meisam opened this issue 7 years ago • 6 comments

  • Moved YARN related code from SparkYarnApp to a new class YarnInterface
  • Removed "yarnAppMinotorThreads" that are created for each Spark application
  • Created one thread that polls YARN to get ApplicationIds and
  • Created a pool of threads that gets ApplicationAttempts and ContainerReports from YARN
    • because there is no API to get ApplicationAttempts and ContainerReports in bulk
    • the thread pool is actually used by Scala Futures
  • Updated test cases in SparkYarnAppSpec to match the new design

Task-Url: https://issues.cloudera.org/browse/LIVY-336

meisam avatar Aug 16 '17 17:08 meisam

@meisam this PR looks old and inactive. May you please close it? Thanks.

mgaido91 avatar Jan 24 '19 18:01 mgaido91

IIRC there was a huge discussion on the mailing list about this... this would be a good feature to have if it's brought up to date.

vanzin avatar Jan 24 '19 18:01 vanzin

Sure! I'll update it. Is the IIRC channel public? If yes, can you point me to it?

meisam avatar Jan 24 '19 18:01 meisam

IIRC = if I remember correctly.

vanzin avatar Jan 24 '19 18:01 vanzin

I would love for this to actually get in. We are having problems right now with Livy thread starvation if more than 20 jobs are running at the same time.

Azuaron avatar Jun 07 '19 19:06 Azuaron

Hi @meisam , Now we need this part. If there are no updates, I will continue your work in the new PR and keep your previous commits.

runzhiwang avatar Sep 18 '19 12:09 runzhiwang