sagemaker-python-sdk icon indicating copy to clipboard operation
sagemaker-python-sdk copied to clipboard

Args not getting updated properly with multiple trainingsteps in pipeline

Open clausagerskov opened this issue 1 year ago • 2 comments

Describe the bug When running a sagemaker pytorch pipeline with two separate pytorch training steps with hyperparameters passed to pytorch estimators, if one sets a hyperparam named "modeA" to e.g. True and pass to trainingstep 1, and then set a hyperparam with the same name "modeA" to False and pass to trainingstep 2, when doing argparser to get the args.modeA in the second the value of the hyperparam is still True.

To reproduce Create a pipeline with two pytorch estimator trainingsteps, give both a bool hyperparam named "modeA", first pass as true to trainingstep1 then false to trainingstep2.

System information A description of your system. Please provide:

  • SageMaker Python SDK version: 2.103
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): PyTorch
  • Framework version: 0.23-1
  • Python version: 3.8
  • CPU or GPU: both
  • Custom Docker image (Y/N): N

Additional context Add any other context about the problem here.

clausagerskov avatar Aug 15 '22 18:08 clausagerskov

the same instance type is used in the trainingsteps

clausagerskov avatar Aug 15 '22 18:08 clausagerskov

Could you share the code snippet to reproduce this issue?

navaj0 avatar Aug 30 '22 17:08 navaj0

Did not heard anything back from the CX, close for now; feel free to re-open.

jerrypeng7773 avatar Dec 19 '22 22:12 jerrypeng7773