linkis icon indicating copy to clipboard operation
linkis copied to clipboard

[Question] How to set the running parameters of Spark

Open casionone opened this issue 1 year ago • 1 comments

How to set the running parameters of Spark? such as

  • spark.executor.cores,
  • spark.executor.memory

casionone avatar Aug 03 '22 13:08 casionone

  1. if you run by linkis-cli such as sh ./bin/linkis-cli -engineType spark-2.4.3 -codeType sql -confMap wds.linkis.yarnqueue=q02 -confMap spark.executor.instances=3 -code "select count(*) from testdb.test;" -submitUser hadoop -proxyUser hadoop see detail https://linkis.apache.org/zh-CN/docs/1.1.3/user_guide/linkiscli-manual/

  2. system level add new system level configuration for creator in linkis web console 企业微信截图_16595320571217

  3. set for single task in dss (like eg 1) 企业微信截图_16595322452353

casionone avatar Aug 03 '22 13:08 casionone