Spark pod not getting terminated after completion of spark job

[Amey2400]

Describe the bug Using SynpaseML package for applying LightGBM algorithm in a spark application. This spark application is running in the k8s environment. We have observed that even after the completion of all execution steps, the spark pod does not get terminated when the LightGBM package is used.

Expected behavior After completion of all the steps, all the threads should be terminated and the spark pod should get terminated.

Info (please complete the following information):

• SynapseML Version: v0.9.4
• SynapseML Package: com.microsoft.azure:synapseml_2.12:0.9.4
• Spark Version 3.1.2
• Spark Platform Amazon k8s

[fonhorst]

I experience the same problem with:

• SynapseML Version: v0.9.5
• Spark Version 3.2.0
• Spark Platform on-premise k8s

The problem is not regular and happens from time to time.

[chengs]

I have the problem also.

SynapseML Version: v0.9.5
Spark Version 3.2.0
Spark Platform on Google Cloud DataProc

Likely, it appears from time to time.

[fox-gamer]

Same problem leading to unnecessary costs when using auto-scaling:

• SynapseML version: v0.9.5
• Spark version 3.2.0
• Azure Kubernetes Service (AKS)

[AlexandreOuellet]

I don't know if related, but I have the same issue with sagemaker : I see the pyspark script finish, but the ProcessingJob times out

I am using a LightGBMRegressor, if that makes any difference.

• SynapseML version: v0.9.5
• Spark Version 3.1.
• Sagemaker ProcessingJob