Error while executing many jobs to the same spark cluster

[valentinbeuze]

I'm using Azure Databricks Runtime 5.4 (includes Apache Spark 2.4.3, Scala 2.11) with azure_cosmosdb_spark_2_4_0_2_11_1_4_0_uber.jar

I try to execute many jobs to the same spark cluster Each job connects to the same cosmos db database The first job executed successfully, but the other one fails with the following error :

java.lang.IllegalStateException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask rejected from com.microsoft.azure.documentdb.DocumentClient$DocumentDBThreadPoolExecutor [Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

I think it is about the cosmos db connection... Any help ? Thanks

[abhsh]

I am facing the exact same issue.