hparams table not getting displayed when many hparams are beeing used

[asorie]

Consider Stack Overflow for getting support using TensorBoard—they have a larger community with better searchability:

https://stackoverflow.com/questions/tagged/tensorboard

Do not use this template for for setup, installation, or configuration issues. Instead, use the “installation problem” issue template:

https://github.com/tensorflow/tensorboard/issues/new?template=installation_problem.md

To report a problem with TensorBoard itself, please fill out the remainder of this template.

Environment information (required)

Diagnostics output

--- check: autoidentify
INFO: diagnose_tensorboard.py version 4725c70c7ed724e2d1b9ba5618d7c30b957ee8a4

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)
INFO: os.name: nt
INFO: os.uname(): N/A
INFO: sys.getwindowsversion(): sys.getwindowsversion(major=10, minor=0, build=14393, platform=2, service_pack='')

--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: 'C:\\tensorflow_anduin'

--- check: installed_packages
INFO: installed: tensorboard==2.0.0
INFO: installed: tensorflow-gpu==2.0.0
INFO: installed: tensorflow-estimator==2.0.0

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.0.0'

--- check: tensorflow_python_version
2019-10-08 14:40:42.620638: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
INFO: tensorflow.__version__: '2.0.0'
INFO: tensorflow.__git_version__: 'v2.0.0-rc2-26-g64c3d382ca'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'C:\\tensorflow_anduin\\Scripts\\tensorboard.exe\r\n'

--- check: readable_fqdn
INFO: socket.getfqdn(): '...'

--- check: stat_tensorboardinfo
INFO: directory: C:\Users\halle\AppData\Local\Temp\.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=3096224744103339, st_dev=2217911477, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1570538160, st_mtime=1570538160, st_ctime=1562760637)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['C:\\tensorflow_anduin\\lib\\site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.7.1
adal==1.2.2
asn1crypto==0.24.0
astor==0.8.0
astroid==2.2.5
avro-python3==1.9.1
azure-common==1.1.23
azure-graphrbac==0.53.0
azure-keyvault==1.1.0
azure-mgmt-authorization==0.51.1
azure-mgmt-containerregistry==2.7.0
azure-mgmt-keyvault==1.1.0
azure-mgmt-msi==0.2.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-resource==2.2.0
azure-mgmt-storage==3.1.1
azure-nspkg==3.0.2
azure-storage-blob==1.5.0
azure-storage-common==1.4.2
blinker==1.4
boto3==1.9.238
botocore==1.12.238
cachetools==3.1.1
certifi==2019.9.11
cffi==1.12.3
chardet==3.0.4
Click==7.0
click-completion==0.5.1
clipboard==0.0.4
colorama==0.3.9
cryptography==2.7
cycler==0.10.0
docker==3.7.3
docker-pycreds==0.4.0
docutils==0.15.2
Flask==1.1.1
flatten-json==0.1.7
gast==0.2.2
gitdb2==2.0.6
GitPython==2.1.14
google-api-core==1.14.2
google-auth==1.6.3
google-cloud-core==1.0.3
google-cloud-kms==1.2.1
google-cloud-storage==1.20.0
google-pasta==0.1.7
google-resumable-media==0.4.1
googleapis-common-protos==1.6.0
grpc-google-iam-v1==0.12.3
grpcio==1.22.0
h5py==2.9.0
httplib2==0.14.0
humanize==0.5.1
idna==2.8
imageio==2.5.0
isodate==0.6.0
isort==4.3.21
itsdangerous==1.1.0
Jinja2==2.10.1
jmespath==0.9.4
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
lazy-object-proxy==1.4.1
lockfile==0.12.2
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.1
mccabe==0.6.1
missinglink==19.9.26557
missinglink-kernel==19.9.26893
missinglink-sdk==19.9.26893
ml-core==19.9.3999
ml-crypto==0.7.811
ml-legit==19.9.8734
msgpack==0.6.2
msrest==0.6.10
msrestazure==0.6.2
mypy==0.711
mypy-extensions==0.4.1
natsort==6.0.0
netifaces==0.10.9
numpy==1.17.2
oauthlib==3.1.0
opt-einsum==2.3.2
pandas==0.25.1
patsy==0.5.1
pep8==1.7.1
Pillow==6.1.0
pip==19.2.3
ply==3.11
protobuf==3.8.0
psutil==5.6.3
puremagic==1.5
pyasn1==0.4.7
pyasn1-modules==0.2.6
pycparser==2.19
pycryptodome==3.6.6
Pygments==2.4.2
PyJWT==1.7.1
pylint==2.3.1
pyparsing==2.4.0
pyperclip==1.7.0
pypiwin32==223
python-dateutil==2.8.0
pytz==2019.2
pywin32==225
PyYAML==5.1.1
requests==2.22.0
requests-oauthlib==1.2.0
retrying==1.3.3
rope==0.14.0
rsa==4.0
s3transfer==0.2.1
scipy==1.3.0
sentry-sdk==0.11.2
setuptools==41.0.1
shellingham==1.3.1
six==1.12.0
smmap2==2.0.5
sseclient==0.0.24
statsmodels==0.10.1
tensorboard==2.0.0
tensorflow-estimator==2.0.0
tensorflow-gpu==2.0.0
termcolor==1.1.0
terminaltables==3.1.0
tqdm==4.32.2
typed-ast==1.4.0
urllib3==1.24.3
wcwidth==0.1.7
websocket-client==0.56.0
Werkzeug==0.16.0
wheel==0.33.4
wrapt==1.11.2

• Browser: Chrome 76.0.3809.132

Issue description

If I use many hparams (eg. 14) in tensorboard the table doenst display any results but the table head gets displayed correcly.

But when I delete some of the rows in HPARAMS section the row in the hparams table and the accuracy gets displayed correcly.

HPARAMS = [HP_BATCH_SIZE,
                HP_OPTIMIZER,
                HP_W_PARAM_0,
                HP_W_PARAM_1,
                HP_W_PARAM_2,
                HP_W_PARAM_3,
                HP_CONV1_FILTER,
                HP_CONV1_KERNEL,
                HP_CONV2_FILTER,
                HP_CONV2_KERNEL,
                HP_CONV3_FILTER,
                HP_CONV3_KERNEL,
                HP_Conv_UP_1_UNITS,
                HP_Conv_UP_2_UNITS]

with file_writer.as_default():
        hp.hparams_config(
            hparams=HPARAMS,
            metrics=METRICS,
        )
        hp.hparams(hparams)

[gowthamkpr]

@Asorie Can you please try running this script in your notebook and let me know if you are facing the same issue?

[asorie]

I tried the script , but at section 4. the line %tensorboard --logdir logs/hparam_tuning procudes an error:

ERROR: Failed to launch TensorBoard (exited with 1).
Contents of stderr:
Traceback (most recent call last):
  File "/usr/local/bin/tensorboard", line 10, in <module>
    sys.exit(run_main())
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/main.py", line 64, in run_main
    app.run(tensorboard.main, flags_parser=tensorboard.configure)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/program.py", line 220, in main
    server = self._make_server()
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/program.py", line 299, in _make_server
    self.assets_zip_provider)
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/backend/application.py", line 160, in standard_tensorboard_wsgi
    flags, plugin_loaders, data_provider, assets_zip_provider, multiplexer)
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/backend/application.py", line 228, in TensorBoardWSGIApp
    return TensorBoardWSGI(tbplugins, flags.path_prefix)
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/backend/application.py", line 279, in __init__
    raise ValueError('Duplicate plugins for name %s' % plugin.plugin_name)
ValueError: Duplicate plugins for name projector

[gowthamkpr]

Its because there might be multiple versions of Tensorboard in your system. Please find my github gist here

I am able to see all the hyperparameters on Tensorboard using Tensorflow 2.0. There might be an issue with your tensorboard. Please try to run the same script in your system and see if you can see hparams displayed or no. Thanks!

[asorie]

The script works. I think the problem is, that I tried to add new HP and write the logs to an already used tensorboard.

[gowthamkpr]

Yes. So, I think the problem here is resolved?

[asorie]

Not really. I think tensorboard should look for the HP used and add new to the table if a new HP was found.

[asorie]

If this line: HP_OPTIMIZER = hp.HParam('optimizer', hp.Discrete(['adam', 'sgd'])) gets changed to: HP_OPTIMIZER = hp.HParam('optimizer', hp.Discrete(['adam', 'sgd', 'RMSprop'])) and then trained to the same logdir, tensorboard doenst add this new model to the hparams table.

So it isn't possible to dynamically change the possible hparams in the same logdir?

[Yannick947]

I think I'm facing the same issue. Any updates here?

[bersbersbers]

This issue, in particular, https://github.com/tensorflow/tensorboard/issues/2743#issuecomment-542057891, very much reminds me of #3597. There, the problem is that mixed-type (string + float, meaning some models use a string value, others a numerical value) parameters are all cast to string, but the filter in list_session_groups.py doesn't take that casting into account - it looks for 2.0 and doesn't find "2.0". As a result, only models with string parameter values are found - the other ones just don't show up. I have never used hp.HParam myself, so I cannot say if the two HP_OPTIMIZERs are seen as different types, but it sure feels like a similar issue.

[NumberChiffre]

I'm having the same issue, I am using torch + PPO in rllib and only half of my hyperparams show on tensorboard

[vrublack]

I've had the same issue with TensorboardX, the reason was that the metric name contained a whitespace.

[ghost]

I met the same issue, when the number of hparams is getting large the issue appears.

[pinkponk]

Still an issue for me. Really annoying. Anyone have a solution?

I guess I will try to write Hparams structure to other file and replace that every time I change something. Not sure this works though

[arcra]

The original issue description here suggests the issue appears when "many hparams" are used. Then later it seems to be that users are trying to "add new HP and write the logs to an already used tensorboard".

So I'm not sure I'm understanding what the issue is. Are you logging more hparams data to the same log dir, and you want TB to read it? Does starting tensorboard again like tensorboard --logdir path/to/logs show everything you want to see? Do you have a small example to reproduce the issue?

[bersbersbers]

@arcra it probably covers only one aspect of this issue, but https://github.com/tensorflow/tensorboard/issues/3597#issuecomment-1490793918 has very specific repro steps that I created "only" 7 months ago ("only" compared to the 4 years that this issue has been open).