[Feature] support spark dynamicAllocation

[CCweixiao]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Problem Description

希望可以支持spark.dynamicAllocation.enabled，引擎空闲时可以回收yarn资源

I hope spark.dynamicAllocation.enabled can be supported, and yarn resources can be recycled when the engine is idl

Description

希望可以支持spark.dynamicAllocation.enabled，引擎空闲时可以回收yarn资源

I hope spark.dynamicAllocation.enabled can be supported, and yarn resources can be recycled when the engine is idl

Use case

No response

solutions

No response

Anything else

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

[wForget]

This seems to be a Spark feature, just add the relevant SparkConfs, what do you want to do in Linkis?

[CCweixiao]

This seems to be a Spark feature, just add the relevant SparkConfs, what do you want to do in Linkis?

动态资源参数配置可以生效，只是Linkis在做资源检查的时候，不会动态变化每个实例实际占用的资源大小。 这里的统计值是不会变化的 {"DriverAndYarnResource":{"loadInstanceResource":{"memory":4294967296,"cores":1,"instances":1},"yarnResource":{"queueMemory":322122547200,"queueCores":60,"queueInstances":0,"queueName":"medium.linkis","applicationId":""}}}

The resource parameter configuration can take effect, but Link does not dynamically change the actual resource size of each instance when checking resources. The stats here won't change

[wForget]

This seems to be a Spark feature, just add the relevant SparkConfs, what do you want to do in Linkis?

动态资源参数配置可以生效，只是Linkis在做资源检查的时候，不会动态变化每个实例实际占用的资源大小。 这里的统计值是不会变化的 {"DriverAndYarnResource":{"loadInstanceResource":{"memory":4294967296,"cores":1,"instances":1},"yarnResource":{"queueMemory":322122547200,"queueCores":60,"queueInstances":0,"queueName":"medium.linkis","applicationId":""}}}

The resource parameter configuration can take effect, but Link does not dynamically change the actual resource size of each instance when checking resources. The stats here won't change

got it, it's needed