RANGER-3787 Non-daemon threads started by ElasticSearchAuditDestination cause Spark application hanging

Open zhouyifan279 opened this issue 3 years ago • 3 comments

Issue

https://issues.apache.org/jira/browse/RANGER-3787

When using kyuubi-spark-authz plugin (which extends RangerBasePlugin) to save audit log to ElasticSearch in Spark application, I found that if Spark application was submitted in local or client mode, it hanged forever even after main thread exited. But if I turn off xasecure.audit.destination.elasticsearch, Spark application exits normally.

Solution

By default, if ThreadFactory is not set, RestHighLevelClient uses Executors.defaultThreadFactory() to create http connection threads, which are non-daemon threads. We provide a ThreadFactory which creates daemon threads to override the default behavior.

Jun 14 '22 03:06 zhouyifan279

cc @mneethiraj

Jun 14 '22 03:06 zhouyifan279

Ranger following the review board process for the contributions.

Please attache your patch at "https://reviews.apache.org/r/".

For more details please check the link : https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=55151244

Jun 14 '22 04:06 bhavikpatel9977

@bhavikpatel9977 Thanks for your information. Review request is created: https://reviews.apache.org/r/74022/ .

Jun 14 '22 13:06 zhouyifan279