Hyukjin Kwon

@lyssg mind linking the JIRA into the PR title please? See also https://spark.apache.org/contributing.html

cc @JoshRosen FYI

oh yeah, probably it's a good idea to name them

Alright, let's merge this in once the tests pass. @LucaCanali mind fixing the lint failure? ``` starting mypy annotations test... annotations failed mypy checks: python/pyspark/sql/tests/test_dataframe.py:1113: error: Redundant cast to "str"...

Merged to master.

There have not been updates here after https://github.com/py4j/py4j/issues/374#issuecomment-513463892. PR is very welcome!

@Devender2806 I believe you are already able to do what you're able to do in Java side. It would be great to elaborate what this issue targets to add preferably...

@Tagar, mind if I ask a code bit you want to run? will try to explain it or take a look. I don't think it's an issue for py4j nor...

> how we can call a third-party Java library's method (that was distributed into executors through --jars argument) - from Spark's mapPartitions() or map() functions? I think you need to...

I think that's because we don't use Py4J in executors within Spark .. I think it makes less sense to have it in executor side only because users might use...