amazon-sagemaker-examples
amazon-sagemaker-examples copied to clipboard
Published
20 hours ago •
aws
aws
[Bug Report]
Link to the notebook Train an TensorFlow model with a SageMaker Training Job and track it using SageMaker Experiments Describe the bug When executing the notebook the model training (8th cell in the notebook) fails with
ParamValidationError: Parameter validation failed:
Unknown parameter in ProfilerConfig: "DisableProfiler", must be one of: S3OutputPath, ProfilingIntervalInMilliseconds, ProfilingParameters
Bugs replicated in SageMaker Studio domains in ap-southeast-1 and us-east-2
To reproduce Run the notebook step by step
Logs Error trace:
INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.
INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.
INFO:sagemaker:Creating training-job with name: tensorflow-training-2022-12-20-10-27-40-801
---------------------------------------------------------------------------
ParamValidationError Traceback (most recent call last)
<ipython-input-8-952c129da21f> in <module>
30 )
31
---> 32 est.fit()
/opt/conda/lib/python3.7/site-packages/sagemaker/workflow/pipeline_context.py in wrapper(*args, **kwargs)
270 return _StepArguments(retrieve_caller_name(self_instance), run_func, *args, **kwargs)
271
--> 272 return run_func(*args, **kwargs)
273
274 return wrapper
/opt/conda/lib/python3.7/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
1128
1129 experiment_config = check_and_get_run_experiment_config(experiment_config)
-> 1130 self.latest_training_job = _TrainingJob.start_new(self, inputs, experiment_config)
1131 self.jobs.append(self.latest_training_job)
1132 if wait:
/opt/conda/lib/python3.7/site-packages/sagemaker/estimator.py in start_new(cls, estimator, inputs, experiment_config)
2046 train_args = cls._get_train_args(estimator, inputs, experiment_config)
2047
-> 2048 estimator.sagemaker_session.train(**train_args)
2049
2050 return cls(estimator.sagemaker_session, estimator._current_job_name)
/opt/conda/lib/python3.7/site-packages/sagemaker/session.py in train(self, input_mode, input_config, role, job_name, output_config, resource_config, vpc_config, hyperparameters, stop_condition, tags, metric_definitions, enable_network_isolation, image_uri, algorithm_arn, encrypt_inter_container_traffic, use_spot_instances, checkpoint_s3_uri, checkpoint_local_path, experiment_config, debugger_rule_configs, debugger_hook_config, tensorboard_output_config, enable_sagemaker_metrics, profiler_rule_configs, profiler_config, environment, retry_strategy)
625 self.sagemaker_client.create_training_job(**request)
626
--> 627 self._intercept_create_request(train_request, submit, self.train.__name__)
628
629 def _get_train_request( # noqa: C901
/opt/conda/lib/python3.7/site-packages/sagemaker/session.py in _intercept_create_request(self, request, create, func_name)
4654 func_name (str): the name of the function needed intercepting
4655 """
-> 4656 return create(request)
4657
4658
/opt/conda/lib/python3.7/site-packages/sagemaker/session.py in submit(request)
623 LOGGER.info("Creating training-job with name: %s", job_name)
624 LOGGER.debug("train request: %s", json.dumps(request, indent=4))
--> 625 self.sagemaker_client.create_training_job(**request)
626
627 self._intercept_create_request(train_request, submit, self.train.__name__)
/opt/conda/lib/python3.7/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
528 )
529 # The "self" in this scope is referring to the BaseClient.
--> 530 return self._make_api_call(operation_name, kwargs)
531
532 _api_call.__name__ = str(py_operation_name)
/opt/conda/lib/python3.7/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
922 endpoint_url=endpoint_url,
923 context=request_context,
--> 924 headers=additional_headers,
925 )
926 resolve_checksum_context(request_dict, operation_model, api_params)
/opt/conda/lib/python3.7/site-packages/botocore/client.py in _convert_to_request_dict(self, api_params, operation_model, endpoint_url, context, headers, set_user_agent_header)
989 )
990 request_dict = self._serializer.serialize_to_request(
--> 991 api_params, operation_model
992 )
993 if not self._client_config.inject_host_prefix:
/opt/conda/lib/python3.7/site-packages/botocore/validate.py in serialize_to_request(self, parameters, operation_model)
379 )
380 if report.has_errors():
--> 381 raise ParamValidationError(report=report.generate_report())
382 return self._serializer.serialize_to_request(
383 parameters, operation_model
ParamValidationError: Parameter validation failed:
Unknown parameter in ProfilerConfig: "DisableProfiler", must be one of: S3OutputPath, ProfilingIntervalInMilliseconds, ProfilingParameters
SageMaker Python SDK version: 2.125.0 Boto3 version: 1.26.33
output of pip list:
Package Version
------------------------------------ -----------------
absl-py 1.3.0
aiobotocore 2.4.1
aiohttp 3.8.3
aioitertools 0.11.0
aiosignal 1.3.1
alabaster 0.7.12
anaconda-client 1.7.2
anaconda-project 0.8.3
ansi2html 1.8.0
anyio 3.6.2
argh 0.26.2
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
asn1crypto 1.3.0
astroid 2.12.13
astropy 4.0
astunparse 1.6.3
async-timeout 4.0.2
asynctest 0.13.0
atomicwrites 1.3.0
attrs 22.1.0
autopep8 1.4.4
autovizwidget 0.20.0
awscli 1.27.24
Babel 2.11.0
backcall 0.1.0
backports.shutil-get-terminal-size 1.0.0
beautifulsoup4 4.8.2
bitarray 1.2.1
bkcharts 0.2
bleach 5.0.1
bokeh 1.4.0
boto 2.49.0
boto3 1.26.33
botocore 1.29.33
Bottleneck 1.3.2
brotlipy 0.7.0
cached-property 1.5.2
cachetools 5.2.0
certifi 2022.9.24
cffi 1.15.0
chardet 3.0.4
charset-normalizer 2.0.4
Click 7.0
cloudpickle 2.2.0
clyent 1.2.2
colorama 0.4.3
conda 22.9.0
conda-package-handling 1.8.1
contextlib2 0.6.0.post1
cryptography 38.0.4
cycler 0.10.0
Cython 0.29.15
cytoolz 0.10.1
dash 2.7.0
dash-core-components 2.0.0
dash-html-components 2.0.0
dash-table 5.0.0
dask 2022.2.0
decorator 4.4.1
defusedxml 0.6.0
diff-match-patch 20181111
dill 0.3.6
distributed 2022.2.0
distro 1.8.0
docker 6.0.1
docker-compose 1.29.2
dockerpty 0.4.1
docopt 0.6.2
docutils 0.16
dparse 0.6.2
entrypoints 0.3
et-xmlfile 1.0.1
fastcache 1.1.0
fastjsonschema 2.16.2
filelock 3.0.12
flake8 3.7.9
Flask 1.1.1
flatbuffers 22.12.6
frozenlist 1.3.3
fsspec 2022.11.0
future 0.18.2
gast 0.4.0
gevent 1.4.0
glob2 0.7
gmpy2 2.0.8
google-auth 2.15.0
google-auth-oauthlib 0.4.6
google-pasta 0.2.0
greenlet 0.4.15
grpcio 1.51.1
h5py 2.10.0
hdijupyterutils 0.20.0
HeapDict 1.0.1
html5lib 1.0.1
hypothesis 5.5.4
idna 2.8
imageio 2.6.1
imagesize 1.2.0
importlib-metadata 4.13.0
intervaltree 3.0.2
ipykernel 5.1.4
ipython 7.34.0
ipython_genutils 0.2.0
ipywidgets 7.5.1
isort 4.3.21
itsdangerous 1.1.0
jdcal 1.4.1
jedi 0.18.2
jeepney 0.4.2
Jinja2 3.1.2
jmespath 1.0.1
joblib 0.14.1
json5 0.9.1
jsonschema 3.2.0
jupyter 1.0.0
jupyter_client 7.4.8
jupyter-console 6.1.0
jupyter_core 4.12.0
jupyter-dash 0.4.2
jupyter-server 1.23.3
jupyterlab 1.2.21
jupyterlab-pygments 0.2.2
jupyterlab-server 1.0.6
keras 2.11.0
keyring 21.1.0
kiwisolver 1.1.0
lazy-object-proxy 1.4.3
libarchive-c 2.8
libclang 14.0.6
lief 0.9.0
llvmlite 0.39.1
locket 0.2.0
lxml 4.9.1
Markdown 3.4.1
MarkupSafe 2.1.1
matplotlib 3.1.3
matplotlib-inline 0.1.6
mccabe 0.6.1
mistune 0.8.4
mkl-fft 1.0.15
mkl-random 1.1.0
mkl-service 2.3.0
mock 4.0.1
more-itertools 8.2.0
mpmath 1.1.0
msgpack 0.6.1
multidict 6.0.3
multipledispatch 0.6.0
multiprocess 0.70.14
nbclassic 0.4.8
nbclient 0.7.2
nbconvert 6.5.4
nbformat 5.7.0
nest-asyncio 1.5.6
networkx 2.4
nltk 3.7
nose 1.3.7
notebook 6.5.2
notebook_shim 0.2.2
numba 0.56.4
numexpr 2.7.1
numpy 1.21.6
numpydoc 0.9.2
oauthlib 3.2.2
olefile 0.46
openpyxl 3.0.3
opt-einsum 3.3.0
packaging 20.1
pandas 1.3.5
pandocfilters 1.4.2
parso 0.8.3
partd 1.1.0
path 13.1.0
pathlib2 2.3.5
pathos 0.3.0
pathtools 0.1.2
patsy 0.5.1
pep8 1.7.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.3.0
pip 22.3.1
pkginfo 1.5.0.1
platformdirs 2.6.0
plotly 5.8.2
pluggy 0.13.1
ply 3.11
pox 0.3.2
ppft 1.7.6.6
prometheus-client 0.7.1
prompt-toolkit 3.0.3
protobuf 3.19.6
protobuf3-to-dict 0.1.5
psutil 5.6.7
ptyprocess 0.6.0
pure-sasl 0.6.2
py 1.11.0
pyarrow 10.0.1
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycodestyle 2.5.0
pycosat 0.6.3
pycparser 2.19
pycrypto 2.6.1
pycurl 7.43.0.5
pydocstyle 4.0.1
pyflakes 2.1.1
pyfunctional 1.4.3
Pygments 2.13.0
PyHive 0.6.5
pykerberos 1.2.1
pylint 2.15.8
pyodbc 4.0.0-unsupported
pyOpenSSL 22.1.0
pyparsing 2.4.6
pyrsistent 0.15.7
PySocks 1.7.1
pytest 5.3.5
pytest-arraydiff 0.3
pytest-astropy 0.8.0
pytest-astropy-header 0.1.2
pytest-doctestplus 0.5.0
pytest-openfiles 0.4.0
pytest-remotedata 0.3.2
python-dateutil 2.8.2
python-dotenv 0.21.0
python-jsonrpc-server 0.3.4
python-language-server 0.31.7
pytz 2019.3
PyWavelets 1.1.1
pyxdg 0.26
PyYAML 6.0
pyzmq 24.0.1
QDarkStyle 2.8
QtAwesome 0.6.1
qtconsole 4.6.0
QtPy 1.9.0
regex 2022.10.31
requests 2.28.1
requests-kerberos 0.12.0
requests-oauthlib 1.3.1
retrying 1.3.4
rope 0.16.0
rsa 4.9
Rtree 0.9.3
ruamel_yaml 0.15.87
s3fs 0.4.2
s3transfer 0.6.0
sagemaker 2.125.0
sagemaker-data-insights 0.3.3
sagemaker-datawrangler 0.3.8
sagemaker-scikit-learn-extension 2.5.0
sagemaker-studio-analytics-extension 0.0.14
sagemaker-studio-sparkmagic-lib 0.1.4
sasl 0.2.1
schema 0.7.5
scikit-image 0.16.2
scikit-learn 0.22.1
scipy 1.4.1
seaborn 0.10.0
SecretStorage 3.1.2
Send2Trash 1.8.0
setuptools 59.3.0
simplegeneric 0.8.1
singledispatch 3.4.0.3
six 1.14.0
smclarify 0.3
smdebug-rulesconfig 1.0.1
sniffio 1.3.0
snowballstemmer 2.0.0
sortedcollections 1.1.2
sortedcontainers 2.1.0
soupsieve 1.9.5
sparkmagic 0.20.0
Sphinx 2.4.0
sphinxcontrib-applehelp 1.0.1
sphinxcontrib-devhelp 1.0.1
sphinxcontrib-htmlhelp 1.0.2
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.2
sphinxcontrib-serializinghtml 1.1.3
sphinxcontrib-websupport 1.2.0
spyder 4.0.1
spyder-kernels 1.8.1
SQLAlchemy 1.3.13
statsmodels 0.11.0
sympy 1.5.1
tables 3.6.1
tabulate 0.9.0
tblib 1.6.0
tenacity 8.1.0
tensorboard 2.11.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorflow 2.11.0
tensorflow-estimator 2.11.0
tensorflow-io-gcs-filesystem 0.29.0
termcolor 2.1.1
terminado 0.8.3
testpath 0.4.4
texttable 1.6.7
thrift 0.13.0
thrift-sasl 0.4.3
tinycss2 1.2.1
toml 0.10.2
tomli 2.0.1
tomlkit 0.11.6
toolz 0.10.0
tornado 6.2
tqdm 4.42.1
traitlets 5.6.0
typed-ast 1.5.4
typing_extensions 4.4.0
ujson 5.6.0
unicodecsv 0.14.1
urllib3 1.26.13
watchdog 0.10.2
wcwidth 0.1.8
webencodings 0.5.1
websocket-client 0.59.0
Werkzeug 2.2.2
wheel 0.34.2
widgetsnbextension 3.5.1
wrapt 1.11.2
wurlitzer 2.0.0
xlrd 1.2.0
XlsxWriter 1.2.7
xlwt 1.3.0
yapf 0.28.0
yarl 1.8.2
zict 1.0.0
zipp 3.11.0
