Parameter validation failed: Invalid type for parameter Configurations[1].Properties.spark.shuffle.service.enabled, value: True, type: <class 'bool'>, valid types: <class 'str'>

[seunggabi]

Describe the feature

Parameter validation failed:
Invalid type for parameter Configurations[1].Properties.spark.shuffle.service.enabled, value: True, type: <class 'bool'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.dynamicAllocation.enabled, value: True, type: <class 'bool'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.dynamicAllocation.minExecutors, value: 1, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.dynamicAllocation.maxExecutors, value: 200, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.dynamicAllocation.initialExecutors, value: 1, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.executor.memoryOverhead, value: 2048, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.yarn.nodemanager.resource.memory-mb, value: 12288, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.driver.cores, value: 4, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.executor.cores, value: 5, type: <class 'int'>, valid types: <class 'str'>

Use Case

[
  {
    "Classification": "spark-env",
    "Configurations": [
      {
        "Classification": "export",
        "Properties": {
          "JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
        }
      }
    ]
  },
  {
    "Classification": "spark-defaults",
    "Properties": {
      "spark.executorEnv.JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64",
      "spark.shuffle.service.enabled": true,
      "spark.dynamicAllocation.enabled": true,
      "spark.dynamicAllocation.minExecutors": 1,
      "spark.dynamicAllocation.maxExecutors": 200,
      "spark.dynamicAllocation.initialExecutors": 1,
      "spark.executor.memoryOverhead": 2048,
      "yarn.nodemanager.resource.memory-mb": 12288,
      "spark.maxRemoteBlockSizeFetchToMem": "2g",
      "spark.driver.maxResultSize": "40g",
      "spark.driver.cores": 4,
      "spark.driver.memory": "16g",
      "spark.executor.cores": 5,
      "spark.executor.memory": "4g"
    }
  }
]

Proposed Solution

boolean, int is okay (internal to string)

Other Information

No response

Acknowledgements

• [ ] I may be able to implement this feature request
• [ ] This feature might incur a breaking change

CLI version used

aws-cli/1.22.97

Environment details (OS name and version, etc.)

Python/3.9.13 Darwin/21.3.0 botocore/1.24.42

[tim-finnigan]

Hi @seunggabi thanks for reaching out. Could you elaborate a bit more on the feature you're requesting here? It would help to know which service/command you're using and what your use case is.

[seunggabi]

@tim-finnigan okay I will comment & pr this weekend.

[tim-finnigan]

Hi @seunggabi did you have any update on this?

[seunggabi]

@tim-finnigan I'm too late.. Sry I transfered my job.

this issue depends on boto lib.. can I change boto?

https://github.com/boto/botocore/blob/f2b0dbb800b8dc2a3541334d5ca1190faf900150/botocore/validate.py#L202-L245

[tim-finnigan]

Hi @seunggabi I think we need more context on what you're trying to do. And I suggest creating an issue or PR in the botocore repository if your feature request is related to that codebase.

[seunggabi]

@tim-finnigan thx for your solution. I promise this, this weekend I resolve issue & create boto issue.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.