[LIVY-336][SERVER] Livy should not spawn one thread per job to track the job on Yarn

[runzhiwang]

What changes were proposed in this pull request?

Instead of spawning a monitor thread for every session, create a centralized thread to monitor all apps.

How was this patch tested?

Existed ITs and UTs.

[codecov-io]

Codecov Report

Merging #242 into master will increase coverage by 0.14%. The diff coverage is 77.14%.

@@             Coverage Diff              @@
##             master     #242      +/-   ##
============================================
+ Coverage     68.09%   68.23%   +0.14%     
- Complexity      939      949      +10     
============================================
  Files           102      102              
  Lines          5883     5935      +52     
  Branches        893      904      +11     
============================================
+ Hits           4006     4050      +44     
- Misses         1302     1310       +8     
  Partials        575      575

Impacted Files	Coverage Δ	Complexity Δ
...rver/src/main/scala/org/apache/livy/LivyConf.scala	96.07% <100%> (+0.05%)	21 <0> (ø)	:arrow_down:
...ain/scala/org/apache/livy/utils/SparkYarnApp.scala	76.07% <76.47%> (+1.07%)	50 <7> (+10)	:arrow_up:
...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala	52.94% <0%> (-2.95%)	7% <0%> (ø)
...c/src/main/java/org/apache/livy/rsc/RSCClient.java	73.49% <0%> (+1.2%)	20% <0%> (ø)	:arrow_down:
...ala/org/apache/livy/scalaapi/LivyScalaClient.scala	86.66% <0%> (+3.33%)	7% <0%> (ø)	:arrow_down:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 319386a...3144ff1. Read the comment docs.

[runzhiwang]

@jahstreet Thank you for your comments. I have updated the PR.

[jahstreet]

@runzhiwang Yeah, I see, will go through once again during weekend. Good job man!

[jerryshao]

I still need a close look to see the detailed implementations.

[jerryshao]

I'm OK with the current changes, @yiheng would you please also take a look.

[jerryshao]

Looks ok from my side, @yiheng please help to review again.

[runzhiwang]

@jahstreet Thank you very much, I have updated according to your comments.

[jahstreet]

@runzhiwang can you test your patch on a Yarn cluster?

[runzhiwang]

@jahstreet OK.

[runzhiwang]

@jahstreet Hi, I have test the patch, and it works fine.

[sanchay0]

Hey @runzhiwang what's the status of this PR? Are you going to merge this in? I'm having issues with dangling threads with a lot of jobs running simultaneously and this fix will be great to have!

[runzhiwang]

@sanchay0 Thanks for your attention. I have completed the code. And It just need more review.