[LIVY-527] added autocompletion api and implementation for ipython

[wangqiaoshi]

What changes were proposed in this pull request?

add complete function in PySparkJobProcessor interface, so ReplDriver can get completion message from python

How was this patch tested?

post /sessions/{sessionId}/completion set kind=pyspark

[wangqiaoshi]

should set spark.pyspark.python ${ipython-path}

[codecov-io]

Codecov Report

Merging #119 into master will decrease coverage by 2.44%. The diff coverage is 0%.

@@             Coverage Diff              @@
##             master     #119      +/-   ##
============================================
- Coverage     70.94%   68.49%   -2.45%     
+ Complexity      922      893      -29     
============================================
  Files           100      100              
  Lines          5500     5606     +106     
  Branches        828      840      +12     
============================================
- Hits           3902     3840      -62     
- Misses         1062     1214     +152     
- Partials        536      552      +16

Impacted Files	Coverage Δ	Complexity Δ
...scala/org/apache/livy/repl/PythonInterpreter.scala	47.53% <0%> (-0.6%)	9 <0> (ø)
.../main/java/org/apache/livy/rsc/ReplJobResults.java	0% <0%> (-100%)	0% <0%> (-2%)
...ain/java/org/apache/livy/rsc/driver/Statement.java	0% <0%> (-100%)	0% <0%> (-6%)
...in/java/org/apache/livy/rsc/driver/AddFileJob.java	0% <0%> (-100%)	0% <0%> (-3%)
...ava/org/apache/livy/rsc/driver/StatementState.java	0% <0%> (-93.34%)	0% <0%> (-9%)
...rc/main/java/org/apache/livy/rsc/BaseProtocol.java	52.12% <0%> (-22.35%)	1% <0%> (ø)
.../java/org/apache/livy/rsc/driver/SparkEntries.java	46.87% <0%> (-17.19%)	7% <0%> (-3%)
...main/java/org/apache/livy/rsc/ContextLauncher.java	66.01% <0%> (-15.62%)	13% <0%> (-5%)
...c/src/main/java/org/apache/livy/rsc/RSCClient.java	75.15% <0%> (-8.7%)	20% <0%> (-6%)
...ain/scala/org/apache/livy/utils/SparkYarnApp.scala	73.23% <0%> (-6.34%)	33% <0%> (ø)
... and 20 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d87a348...1cd30e1. Read the comment docs.

[vanzin]

Please follow the guidelines in the contribution guide (linked in the PR template). Mainly:

• file a bug with a description of the problem / feature
• provide a proper description of your PR (which will be the commit message when merged).

[wangqiaoshi]

Fix the bug : The completion function doesn't work in pyspark kind.

description: when pyspark run in ipython environment , kind pyspark completion work. this pr add complete function in PySparkJobProcessor interface, so ReplDriver can get completion message from python

How was this patch tested?

1. post /sessions/{sessionId}/completion set kind=pyspark

[vanzin]

Please follow the contribution guide. It's linked in the PR template. In case you missed it: https://livy.incubator.apache.org/community/

You're supposed to replace the PR template with your description of the changes. And file a bug.

[Tagar]

ping @wangqiaoshi - that's a great improvement! do you mind fixing those minor issues after the last review thank you!

[wangqiaoshi]

ping @wangqiaoshi - that's a great improvement! do you mind fixing those minor issues after the last review thank you!

ok

[wangqiaoshi]

Some style issues, otherwise looks ok.

Can a test be written for this (e.g. in InteractiveIT)? Not sure how easy it is to hit the completion API from tests.

I have a doubt, InteractiveIT isn't find Kind . Because module livy - integration - the test not introduced livy - core - the parent, but Kind in module livy - core - the parent. I can't run InteractiveIT

[vanzin]

Sorry, totally missed your question. I'm also not sure I understand the question.

It would be really good to have some test for this. It doesn't need to be an integration tests, but it needs to be tested somehow. Otherwise the changes look good.

[balamuruganvg]

What's the plan to merge this?

[jepma]

Yes, it would be nice to implement this feature 👍

[Tagar]

ping @wangqiaoshi

[cyrilledelabre]

Hi @wangqiaoshi , any plan to merge this ? is still something missing ? thanks!

[tsurushun]

Hi @wangqiaoshi, you would be doing a great favor for a lot of people if you could merge this. Thanks!

[mnoji]

Hello @wangqiaoshi , Please merge your great job!!

[Jassy1994]

Hello @wangqiaoshi I use jupyter pyspark kernel in livy yarn-cluster mode, and merged this patch in my livy。but jupyter notebook still cannot auto complete. I find jupyter auto completion seems not use livy autocompletion api，May you have some advices about this situation? Thanks

[rragundez]

Any updates on merging this feature @wangqiaoshi ?

[agsachin]

Any updates on merging this feature @wangqiaoshi @vanzin ?