tensorboard Tensorboard starting but not showing scalars when used on AWS Sagemaker

Environment information (required)

I am running Tensorboard in a Jupyter Lab, on AWS Sagemaker.
Tensorboard is installed via Pytorch.
Tensorboard version: 2.10.1

Diagnostics

Diagnostics output

--- check: autoidentify
INFO: diagnose_tensorboard.py version 516a2f9433ba4f9c3a4fdb0f89735870eda054a1

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=10, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='ip-xxx.eu-west-1.compute.internal', release='5.10.102-99.473.amzn2.x86_64', version='#1 SMP Wed Mar 2 19:14:12 UTC 2022', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: True
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
WARNING: no installation among: ['tb-nightly', 'tensorboard', 'tensorflow-tensorboard']
WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu', 'tf-nightly-gpu-2.0-preview']
WARNING: no installation among: ['tensorflow-estimator', 'tensorflow-estimator-2.0-preview', 'tf-estimator-nightly']

--- check: tensorboard_python_version
Traceback (most recent call last):
  File "tb_diag.py", line 528, in main
    suggestions.extend(check())
  File "tb_diag.py", line 81, in wrapper
    result = fn()
  File "tb_diag.py", line 277, in tensorboard_python_version
    from tensorboard import version
ModuleNotFoundError: No module named 'tensorboard'

--- check: tensorflow_python_version
Traceback (most recent call last):
  File "tb_diag.py", line 528, in main
    suggestions.extend(check())
  File "tb_diag.py", line 81, in wrapper
    result = fn()
  File "tb_diag.py", line 284, in tensorflow_python_version
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

--- check: tensorboard_data_server_version
INFO: no data server installed

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/home/ec2-user/SageMaker/envs/a2s-aws/bin/tensorboard\n'

--- check: addrinfos
socket.has_ipv6 = True
socket.AF_UNSPEC = <AddressFamily.AF_UNSPEC: 0>
socket.SOCK_STREAM = <SocketKind.SOCK_STREAM: 1>
socket.AI_ADDRCONFIG = <AddressInfo.AI_ADDRCONFIG: 32>
socket.AI_PASSIVE = <AddressInfo.AI_PASSIVE: 1>
Loopback flags: <AddressInfo.AI_ADDRCONFIG: 32>
Loopback infos: [(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::1', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0))]
Wildcard flags: <AddressInfo.AI_PASSIVE: 1>
Wildcard infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('0.0.0.0', 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 0, 0, 0))]

--- check: readable_fqdn
INFO: socket.getfqdn(): 'ip-xxx.eu-west-1.compute.internal'

--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=101312705, st_dev=66306, st_nlink=2, st_uid=1000, st_gid=1000, st_size=28, st_atime=1667911951, st_mtime=1667913983, st_ctime=1667913983)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (0): []; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
alabaster==0.7.12
anaconda-client==1.7.2
anaconda-project @ file:///tmp/build/80754af9/anaconda-project_1610472525955/work
anyio==3.5.0
appdirs==1.4.4
argh==0.26.2
argon2-cffi @ file:///tmp/build/80754af9/argon2-cffi_1613036642480/work
asn1crypto @ file:///tmp/build/80754af9/asn1crypto_1596577642040/work
astroid @ file:///tmp/build/80754af9/astroid_1613500851753/work
astropy @ file:///tmp/build/80754af9/astropy_1606922938720/work
async-generator==1.10
atomicwrites==1.4.0
attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work
autopep8 @ file:///tmp/build/80754af9/autopep8_1613246362269/work
autovizwidget==0.19.1
awscli==1.27.2
Babel==2.9.1
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
backports.shutil-get-terminal-size @ file:///tmp/build/80754af9/backports.shutil_get_terminal_size_1608222128777/work
bcrypt==4.0.1
beautifulsoup4 @ file:///home/linux1/recipes/ci/beautifulsoup4_1610988766420/work
bitarray @ file:///tmp/build/80754af9/bitarray_1611254370778/work
bkcharts==0.2
black==19.10b0
bleach==4.1.0
bokeh==2.4.2
boto==2.49.0
boto3==1.26.2
botocore==1.29.2
Bottleneck==1.3.2
brotlipy==0.7.0
cached-property==1.5.2
certifi==2021.10.8
cffi==1.14.0
chardet==3.0.4
charset-normalizer==2.0.12
click @ file:///home/linux1/recipes/ci/click_1610990599742/work
cloudpickle @ file:///tmp/build/80754af9/cloudpickle_1598884132938/work
clyent==1.2.2
colorama==0.4.3
conda==4.8.4
conda-package-handling==1.7.0
contextlib2==0.6.0.post1
coverage==6.3.2
cryptography==36.0.2
cycler==0.10.0
Cython @ file:///tmp/build/80754af9/cython_1614014840177/work
cytoolz==0.11.0
dask @ file:///tmp/build/80754af9/dask-core_1613253194046/work
decorator @ file:///home/ktietz/src/ci/decorator_1611930055503/work
defusedxml @ file:///home/ktietz/src/ci_mi/defusedxml_1612808095953/work
diff-match-patch @ file:///tmp/build/80754af9/diff-match-patch_1594828741838/work
dill==0.3.4
distributed @ file:///tmp/build/80754af9/distributed_1613253474444/work
distro==1.8.0
docker==6.0.1
docker-compose==1.29.2
dockerpty==0.4.1
docopt==0.6.2
docutils==0.15.2
dparse==0.5.1
entrypoints==0.3
environment-kernels==1.1.1
et-xmlfile==1.0.1
fastcache==1.1.0
filelock @ file:///home/linux1/recipes/ci/filelock_1610993975404/work
flake8 @ file:///tmp/build/80754af9/flake8_1601911421857/work
Flask @ file:///home/ktietz/src/ci/flask_1611932660458/work
Flask-Cors==3.0.10
fsspec @ file:///tmp/build/80754af9/fsspec_1602684995936/work
future==0.18.2
gevent @ file:///tmp/build/80754af9/gevent_1610995761849/work
gitdb==4.0.9
GitPython==3.1.29
glob2 @ file:///home/linux1/recipes/ci/glob2_1610991677669/work
gmpy2==2.0.8
google-pasta==0.2.0
greenlet @ file:///tmp/build/80754af9/greenlet_1611958357961/work
gssapi==1.7.3
h5py==2.10.0
hdijupyterutils==0.19.1
HeapDict==1.0.1
html5lib @ file:///tmp/build/80754af9/html5lib_1593446221756/work
idna==2.9
imagecodecs @ file:///tmp/build/80754af9/imagecodecs_1611258163564/work
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imagesize @ file:///home/ktietz/src/ci/imagesize_1611921604382/work
importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1602276842396/work
iniconfig @ file:///home/linux1/recipes/ci/iniconfig_1610983019677/work
intervaltree @ file:///tmp/build/80754af9/intervaltree_1598376443606/work
ipykernel @ file:///tmp/build/80754af9/ipykernel_1596206598566/work/dist/ipykernel-5.3.4-py3-none-any.whl
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1645888813068/work
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
ipywidgets @ file:///tmp/build/80754af9/ipywidgets_1610481889018/work
isort @ file:///tmp/build/80754af9/isort_1612243941122/work
itsdangerous==1.1.0
jdcal==1.4.1
jedi @ file:///tmp/build/80754af9/jedi_1606932558467/work
jeepney @ file:///tmp/build/80754af9/jeepney_1606148855031/work
Jinja2==3.0.3
jmespath==0.10.0
joblib @ file:///tmp/build/80754af9/joblib_1613502643832/work
json5==0.9.5
jsonschema @ file:///tmp/build/80754af9/jsonschema_1602607155483/work
jupyter==1.0.0
jupyter-client @ file:///tmp/build/80754af9/jupyter_client_1601311786391/work
jupyter-console @ file:///tmp/build/80754af9/jupyter_console_1598884538475/work
jupyter-core @ file:///tmp/build/80754af9/jupyter_core_1612213308260/work
jupyter-server==1.15.6
jupyter-server-mathjax==0.2.6
jupyterlab==3.3.2
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
jupyterlab-server==2.11.2
jupyterlab-widgets @ file:///tmp/build/80754af9/jupyterlab_widgets_1609884341231/work
keyring @ file:///tmp/build/80754af9/keyring_1611778520348/work
kiwisolver @ file:///tmp/build/80754af9/kiwisolver_1612282414123/work
krb5==0.3.0
lazy-object-proxy @ file:///tmp/build/80754af9/lazy-object-proxy_1606850042850/work
libarchive-c @ file:///home/linux1/recipes/ci/python-libarchive-c_1610974153025/work
llvmlite==0.34.0
locket==0.2.1
lxml==4.8.0
MarkupSafe==2.1.1
matplotlib @ file:///tmp/build/80754af9/matplotlib-suite_1613407855456/work
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1631080358261/work
mccabe==0.6.1
mistune @ file:///tmp/build/80754af9/mistune_1594373098390/work
mkl-fft==1.2.1
mkl-random==1.1.1
mkl-service==2.3.0
mock @ file:///tmp/build/80754af9/mock_1607622725907/work
more-itertools @ file:///tmp/build/80754af9/more-itertools_1613676688952/work
mpmath==1.1.0
msgpack @ file:///tmp/build/80754af9/msgpack-python_1612287175283/work
multipledispatch==0.6.0
multiprocess==0.70.12.2
mypy-extensions==0.4.3
nbclassic==0.3.7
nbclient @ file:///tmp/build/80754af9/nbclient_1613676291033/work
nbconvert @ file:///tmp/build/80754af9/nbconvert_1601914821128/work
nbdime==3.1.1
nbformat==5.2.0
nest-asyncio @ file:///tmp/build/80754af9/nest-asyncio_1613680548246/work
networkx @ file:///tmp/build/80754af9/networkx_1598376031484/work
nltk==3.7
nose @ file:///tmp/build/80754af9/nose_1606773131901/work
notebook==6.4.10
notebook-shim==0.1.0
numba @ file:///tmp/build/80754af9/numba_1600102479638/work
numexpr @ file:///tmp/build/80754af9/numexpr_1609354613445/work
numpy==1.21.5
numpydoc @ file:///tmp/build/80754af9/numpydoc_1605117425582/work
olefile==0.46
opencv-python==4.5.5.64
openpyxl @ file:///tmp/build/80754af9/openpyxl_1610651698508/work
packaging==21.3
pandas==1.2.2
pandocfilters @ file:///tmp/build/80754af9/pandocfilters_1605120451932/work
paramiko==2.11.0
parso==0.7.0
partd==1.1.0
path @ file:///tmp/build/80754af9/path_1614022219955/work
pathlib2 @ file:///tmp/build/80754af9/pathlib2_1607024979554/work
pathos==0.2.8
pathspec==0.7.0
pathtools==0.1.2
patsy==0.5.1
pep8==1.7.1
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
pid==3.0.4
Pillow==9.0.1
pip==22.0.4
pkginfo==1.7.0
plotly==5.6.0
pluggy==0.13.1
ply==3.11
pox==0.3.0
ppft==1.6.6.4
prometheus-client @ file:///tmp/build/80754af9/prometheus_client_1606344362066/work
prompt-toolkit @ file:///tmp/build/80754af9/prompt-toolkit_1602688806899/work
protobuf==3.15.2
protobuf3-to-dict==0.1.5
psutil @ file:///tmp/build/80754af9/psutil_1612298016854/work
psycopg2-binary==2.9.3
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
py @ file:///tmp/build/80754af9/py_1607971587848/work
py4j==0.10.9.5
pyasn1==0.4.8
pycodestyle @ file:///home/ktietz/src/ci_mi/pycodestyle_1612807597675/work
pycosat==0.6.3
pycparser==2.20
pycrypto==2.6.1
pycurl==7.43.0.5
pydocstyle @ file:///tmp/build/80754af9/pydocstyle_1598885001695/work
pyerfa @ file:///tmp/build/80754af9/pyerfa_1613848714600/work
pyflakes @ file:///home/ktietz/src/ci_ipy2/pyflakes_1612551159640/work
pygal==2.4.0
Pygments @ file:///tmp/build/80754af9/pygments_1613849567472/work
pyinstrument==3.4.2
pyinstrument-cext==0.2.4
pykerberos==1.2.1
pylint @ file:///tmp/build/80754af9/pylint_1614008007455/work
pyls-black @ file:///tmp/build/80754af9/pyls-black_1607553132291/work
pyls-spyder @ file:///tmp/build/80754af9/pyls-spyder_1613849700860/work
PyNaCl==1.5.0
pyodbc===4.0.0-unsupported
pyOpenSSL==19.1.0
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
pyrsistent @ file:///tmp/build/80754af9/pyrsistent_1600141707582/work
PySocks==1.7.1
pyspark==3.3.0
pyspnego==0.5.0
pytest==6.2.2
python-dateutil @ file:///home/ktietz/src/ci/python-dateutil_1611928101742/work
python-dotenv==0.21.0
python-jsonrpc-server @ file:///tmp/build/80754af9/python-jsonrpc-server_1600278539111/work
python-language-server @ file:///tmp/build/80754af9/python-language-server_1607972495879/work
pytz @ file:///tmp/build/80754af9/pytz_1612215392582/work
PyWavelets @ file:///tmp/build/80754af9/pywavelets_1601658308664/work
pyxdg @ file:///tmp/build/80754af9/pyxdg_1603822279816/work
PyYAML==5.4.1
pyzmq==20.0.0
QDarkStyle==2.8.1
QtAwesome @ file:///tmp/build/80754af9/qtawesome_1602272867890/work
qtconsole @ file:///tmp/build/80754af9/qtconsole_1612458529756/work
QtPy==1.9.0
regex==2022.3.15
requests==2.27.1
requests-kerberos==0.14.0
rope @ file:///tmp/build/80754af9/rope_1602264064449/work
rsa==4.7.2
Rtree==0.9.4
ruamel_yaml==0.15.87
s3transfer==0.6.0
safety==1.10.3
sagemaker==2.116.0
sagemaker-nbi-agent @ file:///tmp/sagemaker_nbi_agent
sagemaker-pyspark==1.4.5
schema==0.7.5
scikit-image==0.17.2
scikit-learn @ file:///tmp/build/80754af9/scikit-learn_1598376882706/work
scipy @ file:///tmp/build/80754af9/scipy_1614022789548/work
seaborn @ file:///tmp/build/80754af9/seaborn_1608578541026/work
SecretStorage @ file:///tmp/build/80754af9/secretstorage_1614022884812/work
Send2Trash==1.8.0
setuptools==46.4.0.post20200518
shap==0.40.0
simplegeneric==0.8.1
singledispatch @ file:///tmp/build/80754af9/singledispatch_1614023095590/work
six==1.14.0
slicer==0.0.7
smdebug==1.0.12
smdebug-rulesconfig==1.0.1
smmap==5.0.0
sniffio==1.2.0
snowballstemmer @ file:///tmp/build/80754af9/snowballstemmer_1611258885636/work
sortedcollections @ file:///tmp/build/80754af9/sortedcollections_1611172717284/work
sortedcontainers @ file:///tmp/build/80754af9/sortedcontainers_1606865132123/work
soupsieve @ file:///tmp/build/80754af9/soupsieve_1614023247537/work
sparkmagic==0.15.0
Sphinx @ file:///tmp/build/80754af9/sphinx_1614023429948/work
sphinxcontrib-applehelp @ file:///home/ktietz/src/ci/sphinxcontrib-applehelp_1611920841464/work
sphinxcontrib-devhelp @ file:///home/ktietz/src/ci/sphinxcontrib-devhelp_1611920923094/work
sphinxcontrib-htmlhelp @ file:///home/ktietz/src/ci/sphinxcontrib-htmlhelp_1611920974801/work
sphinxcontrib-jsmath @ file:///home/ktietz/src/ci/sphinxcontrib-jsmath_1611920942228/work
sphinxcontrib-qthelp @ file:///home/ktietz/src/ci/sphinxcontrib-qthelp_1611921055322/work
sphinxcontrib-serializinghtml @ file:///home/ktietz/src/ci/sphinxcontrib-serializinghtml_1611920755253/work
sphinxcontrib-websupport @ file:///tmp/build/80754af9/sphinxcontrib-websupport_1597081412696/work
spyder @ file:///tmp/build/80754af9/spyder_1612463486277/work
spyder-kernels @ file:///tmp/build/80754af9/spyder-kernels_1614030583822/work
SQLAlchemy @ file:///tmp/build/80754af9/sqlalchemy_1612467559264/work
statsmodels @ file:///tmp/build/80754af9/statsmodels_1614023743307/work
sympy @ file:///tmp/build/80754af9/sympy_1608137658835/work
tables==3.6.1
tblib @ file:///tmp/build/80754af9/tblib_1597928476713/work
tenacity==8.0.1
terminado==0.9.2
testpath @ file:///home/ktietz/src/ci/testpath_1611930608132/work
textdistance @ file:///tmp/build/80754af9/textdistance_1612461398012/work
texttable==1.6.4
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
three-merge @ file:///tmp/build/80754af9/three-merge_1607553261110/work
tifffile @ file:///tmp/build/80754af9/tifffile_1610739638720/work
toml @ file:///tmp/build/80754af9/toml_1592853716807/work
toolz @ file:///home/linux1/recipes/ci/toolz_1610987900194/work
tornado @ file:///tmp/build/80754af9/tornado_1606942283357/work
tqdm==4.46.0
traitlets @ file:///home/ktietz/src/ci/traitlets_1611929699868/work
typed-ast @ file:///tmp/build/80754af9/typed-ast_1610484562261/work
typing_extensions==4.1.1
ujson @ file:///tmp/build/80754af9/ujson_1611259506266/work
unicodecsv==0.14.1
urllib3==1.26.8
watchdog @ file:///tmp/build/80754af9/watchdog_1612471130876/work
wcwidth @ file:///tmp/build/80754af9/wcwidth_1593447189090/work
webencodings==0.5.1
websocket-client==0.59.0
Werkzeug==2.0.3
wheel==0.34.2
widgetsnbextension==3.5.1
wrapt==1.12.1
wurlitzer @ file:///tmp/build/80754af9/wurlitzer_1594751868473/work
xlrd @ file:///tmp/build/80754af9/xlrd_1608072521494/work
XlsxWriter @ file:///home/ktietz/src/ci_ipy2/xlsxwriter_1612556818988/work
xlwt==1.3.0
yapf @ file:///tmp/build/80754af9/yapf_1593528177422/work
zict==2.0.0
zipp @ file:///tmp/build/80754af9/zipp_1604001098328/work
zope.event==4.5.0
zope.interface @ file:///tmp/build/80754af9/zope.interface_1606940237377/work

Issue description

I start tensorboard via tensorboard --logdir trainings/
Tensorboard starts correctly.
Then I open the UI via https://xxx.notebook.eu-west-1.sagemaker.aws/proxy/6006/
The UI shows up, the runs are listed, but there is no scalar or hparam.
The scalars plots are there, but empty.
The terminal returns:

I1108 12:55:28.033817 140157277591296 _internal.py:224] 127.0.0.1 - - [08/Nov/2022 12:55:28] "POST /experiment/defaultExperimentId/data/plugin/timeseries/timeSeries HTTP/1.1" 500 -
E1108 12:55:28.034482 140157277591296 _internal.py:224] Error on request:
Traceback (most recent call last):
  File "/home/ec2-user/SageMaker/envs/a2s-aws/lib/python3.10/site-packages/werkzeug/serving.py", line 100, in read_chunk_len
    _len = int(line.strip(), 16)
ValueError: invalid literal for int() with base 16: '------WebKitFormBoundaryUrgwwCzpGNRnuCAe'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/SageMaker/envs/a2s-aws/lib/python3.10/site-packages/werkzeug/formparser.py", line 140, in wrapper
    return f(self, stream, *args, **kwargs)
  File "/home/ec2-user/SageMaker/envs/a2s-aws/lib/python3.10/site-packages/werkzeug/formparser.py", line 290, in _parse_multipart
    form, files = parser.parse(stream, boundary, content_length)
  File "/home/ec2-user/SageMaker/envs/a2s-aws/lib/python3.10/site-packages/werkzeug/formparser.py", line 418, in parse
    for data in iterator:
  File "/home/ec2-user/SageMaker/envs/a2s-aws/lib/python3.10/site-packages/werkzeug/wsgi.py", line 691, in _make_chunk_iter
    item = _read(buffer_size)
  File "/home/ec2-user/SageMaker/envs/a2s-aws/lib/python3.10/site-packages/werkzeug/serving.py", line 113, in readinto
    self._len = self.read_chunk_len()
  File "/home/ec2-user/SageMaker/envs/a2s-aws/lib/python3.10/site-packages/werkzeug/serving.py", line 102, in read_chunk_len
    raise OSError("Invalid chunk header") from e
OSError: Invalid chunk header

This is the same error than described here, 2 years ago, on tensorboard 2.4:
https://stackoverflow.com/questions/66220426/tensorboard-not-working-with-after-upgrading-tensorflow-from-2-1-2-to-2-3-0

I can also confirm that tensorboard 2.3 runs and return scalars, in the same context.

Nov 08 '22 13:11 ghost

The request is triggering the logic in werkzeug that parses chunked HTTP requests but the request is presumably not actually chunked encoded.

Which browser are you using?

If you're using Chrome, would it be ok for you could use the "Copy as cURL" option in the network tab of the developer console and paste the contents here? https://developer.chrome.com/blog/replay-a-network-request-in-curl/

Nov 10 '22 15:11 bmd3k

Hello @bmd3k, here is the guilty request (sensitive info replaced by xxx):

curl 'https://xxx.notebook.eu-west-1.sagemaker.aws/proxy/6006/experiment/defaultExperimentId/data/plugin/timeseries/timeSeries' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'Accept-Language: en-GB,en;q=0.9,fr;q=0.8,de;q=0.7' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryWA6twK652Fzuc5rB' \
  -H 'Cookie: _xsrf=xxx; domain=prod; region=eu-west-1; expiryTime=1668196217000; authToken=xxx; JSESSIONID=xxx; redirectURL="https://console.aws.amazon.com/sagemaker/home?region=eu-west-1&notebookState=xxx%3D%3D#/notebook-instances/openNotebook/xxx"' \
  -H 'Origin: https://xxx.notebook.eu-west-1.sagemaker.aws' \
  -H 'Referer: https://xxx.notebook.eu-west-1.sagemaker.aws/proxy/6006/?darkMode=true' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36' \
  -H 'X-TensorBoard-Feature-Flags: {"enabledColorGroup":true,"enabledColorGroupByRegex":true,"enabledExperimentalPlugins":[],"enabledLinkedTime":false,"enabledCardWidthSetting":true,"enabledScalarDataTable":false,"forceSvg":false,"enableDarkModeOverride":null,"defaultEnableDarkMode":true,"isAutoDarkModeAllowed":true,"inColab":false,"metricsImageSupportEnabled":true,"enableTimeSeriesPromotion":false}' \
  -H 'X-XSRF-Protected: 1' \
  -H 'sec-ch-ua: "Chromium";v="106", "Google Chrome";v="106", "Not;A=Brand";v="99"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  --data-raw $'------WebKitFormBoundaryWA6twK652Fzuc5rB\r\nContent-Disposition: form-data; name="requests"\r\n\r\n[{"plugin":"scalars","tag":"epoch"}]\r\n------WebKitFormBoundaryWA6twK652Fzuc5rB--\r\n' \
  --compressed

And I can confirm that the server struggle with this WebKitForm:

ValueError: invalid literal for int() with base 16: '------WebKitFormBoundaryWA6twK652Fzuc5rB'

EDIT:
I looked at the same package when running locally, and it seems similar. I don't get why it doesn't work on AWS.
I also compared werkzeug version and it's 2.2.2 in both cases, locally and on AWS.

Nov 11 '22 08:11 ghost

Might or might not relevant: in your diagnose shows Werkzeug==2.0.3 not 2.2.2 I'm trying to reproduce but jupyter is running for me. Here's my env versions:

$ pip freeze --all | grep Werkzeug
Werkzeug==2.2.2

$ pip freeze --all | grep jupyter
jupyter-server==1.23.1
jupyter-server-proxy==3.2.1
jupyter_client==7.4.4
jupyter_core==5.0.0
jupyterlab==3.3.2
jupyterlab-pygments==0.2.2
jupyterlab-server==2.11.2
jupyterlab-widgets==3.0.3

This is my request:

curl 'http://localhost:8888/proxy/6006/' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'Accept-Language: en-US,en;q=0.9,hi;q=0.8,ja;q=0.7' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: keep-alive' \
  -H 'Cookie: _xsrf=...; username-localhost-8888="2|1:0|10:1668116972|23:username-localhost-8888|44:..."' \
  -H 'Pragma: no-cache' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36' \
  -H 'sec-ch-ua: "Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  --compressed

I don't have "data-raw"

Nov 11 '22 17:11 japie1235813

@japie1235813 : I did a fresh reinstall of my conda on an AWS instance and I can confirm Werkzeug==2.2.2. The problem is still there. But, like you, I don't have the issue if I run it locally, it seems tied to AWS, somehow.

Nov 23 '22 10:11 ghost

Facing the same issue with tensorboard 2.11.0 and Werkzeug 2.2.2 on AWS Sagemaker.