sagemaker-python-sdk icon indicating copy to clipboard operation
sagemaker-python-sdk copied to clipboard

EstimatorBase._json_encode_hyperparameters(hyperparams) breaks sagemaker_training.py

Open osdf opened this issue 2 years ago • 0 comments

Describe the bug

https://github.com/aws/sagemaker-training-toolkit/blob/master/src/sagemaker_training/logging_config.py#L29 needs an integer for level. Through https://github.com/aws/sagemaker-python-sdk/blob/dev/src/sagemaker/estimator.py#L685 the value in hyperparams[CONTAINER_LOG_LEVEL_PARAM_NAME] (which is logging.INFO, the integer value 20) gets converted to a string ('20'), and hence https://github.com/aws/sagemaker-training-toolkit/blob/master/src/sagemaker_training/logging_config.py#L37 breaks.

Similarly (if one fixes the '20' issue by hand), https://github.com/aws/sagemaker-training-toolkit/blob/master/src/sagemaker_training/files.py#L136 won't identify that code needs to be downloaded from s3, because the entry for https://github.com/aws/sagemaker-python-sdk/blob/dev/src/sagemaker/model.py#L65 is wrapped into a string, having the form "'s3://....'".

To reproduce Run with script-mode, using newest sagemaker python sdk (2.76.0)

Expected behavior It works.

System information A description of your system. Please provide:

  • SageMaker Python SDK version: 2.76.0
  • Framework name: None
  • Framework version: None
  • Python version: 3.8
  • CPU or GPU: CPU and GPU
  • Custom Docker image (Y/N): Yes.

Additional context Add any other context about the problem here.

osdf avatar Feb 22 '22 21:02 osdf