[LIVY-471][Server] Adding new API set to support resource uploading

[jerryshao]

What changes were proposed in this pull request?

Adding a new API set to support resource uploading in session creation.

1. GET [sessions|batches]/id to request new session id.
2. POST [sessions|batches]/id/resource to upload resources. This step can be passed if we don't need to upload resources.
3. POST [sessions|batches]/id to create new session with specified id. the POST request format is the same as old API.

This PR adding a local ResourcesManager to manager all uploaded resources. Resources are properly managed and deleted after session finished.

This is still WIP, needs to debug and add new tests.

How was this patch tested?

new UT.

[jerryshao]

@ajbozarth @pmsgd please take a look at this solution. I would say this solution is more straightforward and comprehensive.

[ajbozarth]

FYI I have been taking a look, I just am really busy atm and don't have time for a deep review right now (or maybe even this week). But this is on my todo list

[codecov-io]

Codecov Report

Merging #95 into master will increase coverage by 0.01%. The diff coverage is 73.01%.

@@             Coverage Diff              @@
##             master      #95      +/-   ##
============================================
+ Coverage     71.35%   71.36%   +0.01%     
- Complexity      794      815      +21     
============================================
  Files            97       98       +1     
  Lines          5428     5528     +100     
  Branches        809      829      +20     
============================================
+ Hits           3873     3945      +72     
- Misses         1030     1037       +7     
- Partials        525      546      +21

Impacted Files	Coverage Δ	Complexity Δ
...e/livy/server/interactive/InteractiveSession.scala	67.38% <100%> (+0.2%)	42 <0> (ø)	:arrow_down:
...rver/src/main/scala/org/apache/livy/LivyConf.scala	93.65% <100%> (+0.05%)	21 <0> (ø)	:arrow_down:
...cala/org/apache/livy/sessions/SessionManager.scala	84.52% <100%> (+0.97%)	26 <3> (+3)	:arrow_up:
...la/org/apache/livy/server/batch/BatchSession.scala	85.55% <100%> (+0.16%)	13 <0> (ø)	:arrow_down:
...server/interactive/InteractiveSessionServlet.scala	65.85% <50%> (-0.54%)	6 <0> (ø)
.../scala/org/apache/livy/server/SessionServlet.scala	65.85% <56.75%> (-2.24%)	34 <5> (+5)
...ala/org/apache/livy/sessions/ResourceManager.scala	71.42% <71.42%> (ø)	12 <12> (?)
.../main/scala/org/apache/livy/sessions/Session.scala	73.21% <86.66%> (-0.38%)	16 <0> (ø)
...apache/livy/server/batch/BatchSessionServlet.scala	85.18% <87.5%> (-0.53%)	3 <0> (ø)
...main/scala/org/apache/livy/server/LivyServer.scala	36.09% <0%> (-1.19%)	9% <0%> (ø)
... and 4 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ca4cad2...d592461. Read the comment docs.

[jerryshao]

Any comment @ajbozarth ?

[ajbozarth]

Sorry I haven't looked at this yet, I was out sick for two weeks and have been playing catch up all week. I may have time next week to take a look. I haven't forgotten about this.

[ajbozarth]

Did a look through for style issues and didn't find any, so as long as it passes style check then style LGTM.

I haven't had a chance to check out and run the code yet but will do so once you address my current comments.

[thoralf-gutierrez]

Any update @ajbozarth or @jerryshao ?

[pepelu0]

any progress?

[mashuai191]

any update on this? is this still planed to be added?

[fifthpostulate]

This functionality would be extremely useful.