aws-cli
aws-cli copied to clipboard
Parameter validation failed: Invalid type for parameter Configurations[1].Properties.spark.shuffle.service.enabled, value: True, type: <class 'bool'>, valid types: <class 'str'>
Describe the feature
Parameter validation failed:
Invalid type for parameter Configurations[1].Properties.spark.shuffle.service.enabled, value: True, type: <class 'bool'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.dynamicAllocation.enabled, value: True, type: <class 'bool'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.dynamicAllocation.minExecutors, value: 1, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.dynamicAllocation.maxExecutors, value: 200, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.dynamicAllocation.initialExecutors, value: 1, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.executor.memoryOverhead, value: 2048, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.yarn.nodemanager.resource.memory-mb, value: 12288, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.driver.cores, value: 4, type: <class 'int'>, valid types: <class 'str'>
Invalid type for parameter Configurations[1].Properties.spark.executor.cores, value: 5, type: <class 'int'>, valid types: <class 'str'>
Use Case
[
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"Properties": {
"JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
}
}
]
},
{
"Classification": "spark-defaults",
"Properties": {
"spark.executorEnv.JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64",
"spark.shuffle.service.enabled": true,
"spark.dynamicAllocation.enabled": true,
"spark.dynamicAllocation.minExecutors": 1,
"spark.dynamicAllocation.maxExecutors": 200,
"spark.dynamicAllocation.initialExecutors": 1,
"spark.executor.memoryOverhead": 2048,
"yarn.nodemanager.resource.memory-mb": 12288,
"spark.maxRemoteBlockSizeFetchToMem": "2g",
"spark.driver.maxResultSize": "40g",
"spark.driver.cores": 4,
"spark.driver.memory": "16g",
"spark.executor.cores": 5,
"spark.executor.memory": "4g"
}
}
]
Proposed Solution
boolean, int is okay (internal to string)
Other Information
No response
Acknowledgements
- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
CLI version used
aws-cli/1.22.97
Environment details (OS name and version, etc.)
Python/3.9.13 Darwin/21.3.0 botocore/1.24.42
Hi @seunggabi thanks for reaching out. Could you elaborate a bit more on the feature you're requesting here? It would help to know which service/command you're using and what your use case is.
@tim-finnigan okay I will comment & pr this weekend.
Hi @seunggabi did you have any update on this?
@tim-finnigan I'm too late.. Sry I transfered my job.
this issue depends on boto
lib..
can I change boto?
https://github.com/boto/botocore/blob/f2b0dbb800b8dc2a3541334d5ca1190faf900150/botocore/validate.py#L202-L245
Hi @seunggabi I think we need more context on what you're trying to do. And I suggest creating an issue or PR in the botocore repository if your feature request is related to that codebase.
@tim-finnigan thx for your solution. I promise this, this weekend I resolve issue & create boto issue.
⚠️COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.