pyspark-asyncactions icon indicating copy to clipboard operation
pyspark-asyncactions copied to clipboard

Published 20 hours ago • verified publisher icon zero323

Base cores on sparkContext.defaultParallelism

Open benstrum opened this issue 5 months ago • 0 comments

The code and docs show that the number of threads is limited to 2 if spark.driver.cores is not set. There are often times when this value is not set and thus gets set to 2. Suggestions include the following:

  • base the value on sparkContext.defaultParallelism
  • or allow the value to be configured during initialization

benstrum avatar Sep 26 '24 11:09 benstrum