tensorboard
tensorboard copied to clipboard
Published
20 hours ago •
tensorflow
tensorflow
Tensorboard 2.10.0 does not combine mutliple event files automatically.
Environment information (required)
Diagnostics
Diagnostics output
--- check: autoidentify
INFO: diagnose_tensorboard.py version df7af2c6fc0e4c4a5b47aeae078bc7ad95777ffa
--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='scigpu16', release='5.4.17-2136.318.7.1.el8uek.x86_64', version='#2 SMP Thu Apr 13 17:33:39 PDT 2023', machine='x86_64')
INFO: sys.getwindowsversion(): N/A
--- check: package_management
INFO: has conda-meta: True
INFO: $VIRTUAL_ENV: None
--- check: installed_packages
INFO: installed: tensorboard==2.10.0
WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu', 'tf-nightly-gpu-2.0-preview']
WARNING: no installation among: ['tensorflow-estimator', 'tensorflow-estimator-2.0-preview', 'tf-estimator-nightly']
INFO: installed: tensorboard-data-server==0.6.1
--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.10.0'
--- check: tensorflow_python_version
Traceback (most recent call last):
File "diagnose_tensorboard.py", line 511, in main
suggestions.extend(check())
File "diagnose_tensorboard.py", line 81, in wrapper
result = fn()
File "diagnose_tensorboard.py", line 267, in tensorflow_python_version
import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'
--- check: tensorboard_data_server_version
INFO: data server binary: '/home/comp/21482098/.local/lib/python3.8/site-packages/tensorboard_data_server/bin/server'
INFO: data server binary version: b'rustboard 0.6.1'
--- check: tensorboard_binary_path
INFO: which tensorboard: b'/usr/local/bin/tensorboard\n'
--- check: addrinfos
socket.has_ipv6 = True
socket.AF_UNSPEC = <AddressFamily.AF_UNSPEC: 0>
socket.SOCK_STREAM = <SocketKind.SOCK_STREAM: 1>
socket.AI_ADDRCONFIG = <AddressInfo.AI_ADDRCONFIG: 32>
socket.AI_PASSIVE = <AddressInfo.AI_PASSIVE: 1>
Loopback flags: <AddressInfo.AI_ADDRCONFIG: 32>
Loopback infos: [(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::1', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0))]
Wildcard flags: <AddressInfo.AI_PASSIVE: 1>
Wildcard infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('0.0.0.0', 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 0, 0, 0))]
--- check: readable_fqdn
INFO: socket.getfqdn(): 'scigpu16.comp.hkbu.edu.hk'
--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=4831850181, st_dev=2051, st_nlink=2, st_uid=1066, st_gid=501, st_size=6, st_atime=1683603189, st_mtime=1683617521, st_ctime=1683617521)
INFO: mode: 0o40777
--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/home/comp/21482098/.local/lib/python3.8/site-packages']; bad_roots (0): []
--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==1.4.0
alabaster @ file:///home/ktietz/src/ci/alabaster_1611921544520/work
anaconda-client==1.7.2
anaconda-navigator==2.0.3
anaconda-project @ file:///tmp/build/80754af9/anaconda-project_1610472525955/work
anyio @ file:///tmp/build/80754af9/anyio_1617783275907/work/dist
appdirs==1.4.4
argh==0.26.2
argon2-cffi @ file:///tmp/build/80754af9/argon2-cffi_1613037097816/work
asn1crypto @ file:///tmp/build/80754af9/asn1crypto_1596577642040/work
astroid @ file:///tmp/build/80754af9/astroid_1613500854201/work
astropy @ file:///tmp/build/80754af9/astropy_1617745353437/work
async-generator @ file:///home/ktietz/src/ci/async_generator_1611927993394/work
atomicwrites==1.4.0
attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work
autopep8 @ file:///tmp/build/80754af9/autopep8_1615918855173/work
Babel @ file:///tmp/build/80754af9/babel_1607110387436/work
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
backports.functools-lru-cache @ file:///tmp/build/80754af9/backports.functools_lru_cache_1618170165463/work
backports.shutil-get-terminal-size @ file:///tmp/build/80754af9/backports.shutil_get_terminal_size_1608222128777/work
backports.tempfile @ file:///home/linux1/recipes/ci/backports.tempfile_1610991236607/work
backports.weakref==1.0.post1
beautifulsoup4 @ file:///home/linux1/recipes/ci/beautifulsoup4_1610988766420/work
bitarray @ file:///tmp/build/80754af9/bitarray_1620827551536/work
bkcharts==0.2
black==19.10b0
bleach @ file:///tmp/build/80754af9/bleach_1612211392645/work
bokeh @ file:///tmp/build/80754af9/bokeh_1620779595936/work
boto==2.49.0
Bottleneck==1.3.2
brotlipy==0.7.0
cachetools==5.3.0
ccimport==0.3.7
certifi==2020.12.5
cffi @ file:///tmp/build/80754af9/cffi_1613246945912/work
chardet @ file:///tmp/build/80754af9/chardet_1607706746162/work
click @ file:///home/linux1/recipes/ci/click_1610990599742/work
cloudpickle @ file:///tmp/build/80754af9/cloudpickle_1598884132938/work
clyent==1.2.2
colorama @ file:///tmp/build/80754af9/colorama_1607707115595/work
conda==4.10.1
conda-build==3.21.4
conda-content-trust @ file:///tmp/build/80754af9/conda-content-trust_1617045594566/work
conda-package-handling @ file:///tmp/build/80754af9/conda-package-handling_1618262148928/work
conda-repo-cli @ file:///tmp/build/80754af9/conda-repo-cli_1620168426516/work
conda-token @ file:///tmp/build/80754af9/conda-token_1620076980546/work
conda-verify==3.4.2
ConfigArgParse==1.5.3
configer @ git+https://github.com/nghorbani/configer@8cd1e3e556d9697298907800a743e120be57ac36
configparser==5.2.0
contextlib2==0.6.0.post1
cryptography @ file:///tmp/build/80754af9/cryptography_1616769286105/work
cumm-cu114==0.2.8
cycler==0.10.0
Cython @ file:///tmp/build/80754af9/cython_1618435160151/work
cytoolz==0.11.0
dask @ file:///tmp/build/80754af9/dask-core_1617390489108/work
decorator @ file:///tmp/build/80754af9/decorator_1617916966915/work
defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
diff-match-patch @ file:///tmp/build/80754af9/diff-match-patch_1594828741838/work
distributed @ file:///tmp/build/80754af9/distributed_1620902833129/work
docutils @ file:///tmp/build/80754af9/docutils_1620827984873/work
entrypoints==0.3
et-xmlfile==1.0.1
fastcache==1.1.0
filelock @ file:///home/linux1/recipes/ci/filelock_1610993975404/work
fire==0.4.0
flake8 @ file:///tmp/build/80754af9/flake8_1615834841867/work
Flask @ file:///home/ktietz/src/ci/flask_1611932660458/work
freetype-py==2.2.0
fsspec @ file:///tmp/build/80754af9/fsspec_1617959894824/work
future==0.18.2
gdown==4.5.1
gevent @ file:///tmp/build/80754af9/gevent_1616770671827/work
glob2 @ file:///home/linux1/recipes/ci/glob2_1610991677669/work
gmpy2==2.0.8
google-auth==2.17.3
google-auth-oauthlib==0.4.6
greenlet @ file:///tmp/build/80754af9/greenlet_1611957705398/work
grpcio==1.54.0
h5py==2.10.0
HeapDict==1.0.1
html5lib @ file:///tmp/build/80754af9/html5lib_1593446221756/work
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
imageio @ file:///tmp/build/80754af9/imageio_1617700267927/work
imagesize @ file:///home/ktietz/src/ci/imagesize_1611921604382/work
importlib-metadata==6.6.0
iniconfig @ file:///home/linux1/recipes/ci/iniconfig_1610983019677/work
intervaltree @ file:///tmp/build/80754af9/intervaltree_1598376443606/work
ipykernel @ file:///tmp/build/80754af9/ipykernel_1596207638929/work/dist/ipykernel-5.3.4-py3-none-any.whl
ipython @ file:///tmp/build/80754af9/ipython_1617120885885/work
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
ipywidgets @ file:///tmp/build/80754af9/ipywidgets_1610481889018/work
isort @ file:///tmp/build/80754af9/isort_1616355431277/work
itsdangerous @ file:///home/ktietz/src/ci/itsdangerous_1611932585308/work
jdcal==1.4.1
jedi @ file:///tmp/build/80754af9/jedi_1606932564285/work
jeepney @ file:///tmp/build/80754af9/jeepney_1606148855031/work
Jinja2 @ file:///tmp/build/80754af9/jinja2_1612213139570/work
joblib @ file:///tmp/build/80754af9/joblib_1613502643832/work
json5==0.9.5
jsonschema @ file:///tmp/build/80754af9/jsonschema_1602607155483/work
jupyter==1.0.0
jupyter-client @ file:///tmp/build/80754af9/jupyter_client_1616770841739/work
jupyter-console @ file:///tmp/build/80754af9/jupyter_console_1616615302928/work
jupyter-core @ file:///tmp/build/80754af9/jupyter_core_1612213311222/work
jupyter-packaging @ file:///tmp/build/80754af9/jupyter-packaging_1613502826984/work
jupyter-server @ file:///tmp/build/80754af9/jupyter_server_1616083640759/work
jupyterlab @ file:///tmp/build/80754af9/jupyterlab_1619133235951/work
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
jupyterlab-server @ file:///tmp/build/80754af9/jupyterlab_server_1617134334258/work
jupyterlab-widgets @ file:///tmp/build/80754af9/jupyterlab_widgets_1609884341231/work
keyring @ file:///tmp/build/80754af9/keyring_1614616740399/work
kiwisolver @ file:///tmp/build/80754af9/kiwisolver_1612282420641/work
lark==1.1.2
lazy-object-proxy @ file:///tmp/build/80754af9/lazy-object-proxy_1616526917483/work
libarchive-c @ file:///tmp/build/80754af9/python-libarchive-c_1617780486945/work
llvmlite==0.36.0
locket==0.2.1
lxml @ file:///tmp/build/80754af9/lxml_1616443220220/work
Markdown==3.4.3
MarkupSafe==1.1.1
matplotlib @ file:///tmp/build/80754af9/matplotlib-suite_1613407855456/work
mccabe==0.6.1
mistune==0.8.4
mkl-fft==1.3.0
mkl-random @ file:///tmp/build/80754af9/mkl_random_1618853849286/work
mkl-service==2.3.0
mock @ file:///tmp/build/80754af9/mock_1607622725907/work
more-itertools @ file:///tmp/build/80754af9/more-itertools_1613676688952/work
mpmath==1.2.1
msgpack @ file:///tmp/build/80754af9/msgpack-python_1612287151062/work
multipledispatch==0.6.0
mypy-extensions==0.4.3
navigator-updater==0.2.1
nbclassic @ file:///tmp/build/80754af9/nbclassic_1616085367084/work
nbclient @ file:///tmp/build/80754af9/nbclient_1614364831625/work
nbconvert @ file:///tmp/build/80754af9/nbconvert_1601914830498/work
nbformat @ file:///tmp/build/80754af9/nbformat_1617383369282/work
nest-asyncio @ file:///tmp/build/80754af9/nest-asyncio_1613680548246/work
networkx @ file:///tmp/build/80754af9/networkx_1598376031484/work
ninja==1.10.2.3
nltk @ file:///tmp/build/80754af9/nltk_1618327084230/work
nose @ file:///tmp/build/80754af9/nose_1606773131901/work
notebook @ file:///tmp/build/80754af9/notebook_1616443462982/work
numba @ file:///tmp/build/80754af9/numba_1616774046117/work
numexpr @ file:///tmp/build/80754af9/numexpr_1618856167419/work
numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1618497241363/work
numpydoc @ file:///tmp/build/80754af9/numpydoc_1605117425582/work
oauthlib==3.2.2
olefile==0.46
open3d-python==0.3.0.0
openpyxl @ file:///tmp/build/80754af9/openpyxl_1615411699337/work
packaging @ file:///tmp/build/80754af9/packaging_1611952188834/work
pandas==1.2.4
pandocfilters @ file:///tmp/build/80754af9/pandocfilters_1605120460739/work
parso==0.7.0
partd @ file:///tmp/build/80754af9/partd_1618000087440/work
path @ file:///tmp/build/80754af9/path_1614022220526/work
pathlib2 @ file:///tmp/build/80754af9/pathlib2_1607024983162/work
pathspec==0.7.0
patsy==0.5.1
pccm==0.3.4
pep8==1.7.1
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
Pillow @ file:///tmp/build/80754af9/pillow_1617383569452/work
pip==21.0.1
pkginfo==1.7.0
pluggy @ file:///tmp/build/80754af9/pluggy_1615976321666/work
ply==3.11
portalocker==2.4.0
prometheus-client @ file:///tmp/build/80754af9/prometheus_client_1618088486455/work
prompt-toolkit @ file:///tmp/build/80754af9/prompt-toolkit_1616415428029/work
protobuf==3.19.6
psutil @ file:///tmp/build/80754af9/psutil_1612298023621/work
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
py @ file:///tmp/build/80754af9/py_1607971587848/work
pyasn1==0.5.0
pyasn1-modules==0.3.0
pybind11==2.9.2
pycodestyle @ file:///home/ktietz/src/ci_mi/pycodestyle_1612807597675/work
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pycurl==7.43.0.6
pydocstyle @ file:///tmp/build/80754af9/pydocstyle_1616182067796/work
pyerfa @ file:///tmp/build/80754af9/pyerfa_1619390903914/work
pyflakes @ file:///home/ktietz/src/ci_ipy2/pyflakes_1612551159640/work
pyglet==1.5.23
Pygments @ file:///tmp/build/80754af9/pygments_1615143339740/work
pylint @ file:///tmp/build/80754af9/pylint_1617135829881/work
pyls-black @ file:///tmp/build/80754af9/pyls-black_1607553132291/work
pyls-spyder @ file:///tmp/build/80754af9/pyls-spyder_1613849700860/work
pyodbc===4.0.0-unsupported
PyOpenGL==3.1.0
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
pyrender==0.1.45
pyrsistent @ file:///tmp/build/80754af9/pyrsistent_1600141720057/work
PySocks @ file:///tmp/build/80754af9/pysocks_1605305779399/work
pytest==6.2.3
python-dateutil @ file:///home/ktietz/src/ci/python-dateutil_1611928101742/work
python-jsonrpc-server @ file:///tmp/build/80754af9/python-jsonrpc-server_1600278539111/work
python-language-server @ file:///tmp/build/80754af9/python-language-server_1607972495879/work
pytz @ file:///tmp/build/80754af9/pytz_1612215392582/work
PyWavelets @ file:///tmp/build/80754af9/pywavelets_1601658317819/work
pyxdg @ file:///tmp/build/80754af9/pyxdg_1603822279816/work
PyYAML==5.4.1
pyzmq==20.0.0
QDarkStyle==2.8.1
QtAwesome @ file:///tmp/build/80754af9/qtawesome_1615991616277/work
qtconsole @ file:///tmp/build/80754af9/qtconsole_1616775094278/work
QtPy==1.9.0
regex @ file:///tmp/build/80754af9/regex_1617569202463/work
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-oauthlib==1.3.1
rope @ file:///tmp/build/80754af9/rope_1602264064449/work
rsa==4.9
Rtree @ file:///tmp/build/80754af9/rtree_1618420845272/work
ruamel-yaml-conda @ file:///tmp/build/80754af9/ruamel_yaml_1616016699510/work
scikit-image==0.18.1
scikit-learn @ file:///tmp/build/80754af9/scikit-learn_1614446682169/work
scipy @ file:///tmp/build/80754af9/scipy_1618855647378/work
seaborn @ file:///tmp/build/80754af9/seaborn_1608578541026/work
SecretStorage @ file:///tmp/build/80754af9/secretstorage_1614022784285/work
Send2Trash @ file:///tmp/build/80754af9/send2trash_1607525499227/work
setuptools==52.0.0.post20210125
Shapely==1.8.1.post1
simplegeneric==0.8.1
singledispatch @ file:///tmp/build/80754af9/singledispatch_1614366001199/work
sip==4.19.13
six @ file:///tmp/build/80754af9/six_1605205327372/work
smplx==0.1.28
sniffio @ file:///tmp/build/80754af9/sniffio_1614030475067/work
snowballstemmer @ file:///tmp/build/80754af9/snowballstemmer_1611258885636/work
sortedcollections @ file:///tmp/build/80754af9/sortedcollections_1611172717284/work
sortedcontainers @ file:///tmp/build/80754af9/sortedcontainers_1606865132123/work
soupsieve @ file:///tmp/build/80754af9/soupsieve_1616183228191/work
spconv-cu114==2.1.21
Sphinx @ file:///tmp/build/80754af9/sphinx_1620777493457/work
sphinxcontrib-applehelp @ file:///home/ktietz/src/ci/sphinxcontrib-applehelp_1611920841464/work
sphinxcontrib-devhelp @ file:///home/ktietz/src/ci/sphinxcontrib-devhelp_1611920923094/work
sphinxcontrib-htmlhelp @ file:///home/ktietz/src/ci/sphinxcontrib-htmlhelp_1611920974801/work
sphinxcontrib-jsmath @ file:///home/ktietz/src/ci/sphinxcontrib-jsmath_1611920942228/work
sphinxcontrib-qthelp @ file:///home/ktietz/src/ci/sphinxcontrib-qthelp_1611921055322/work
sphinxcontrib-serializinghtml @ file:///home/ktietz/src/ci/sphinxcontrib-serializinghtml_1611920755253/work
sphinxcontrib-websupport @ file:///tmp/build/80754af9/sphinxcontrib-websupport_1597081412696/work
spyder @ file:///tmp/build/80754af9/spyder_1616775618138/work
spyder-kernels @ file:///tmp/build/80754af9/spyder-kernels_1614030590686/work
SQLAlchemy @ file:///tmp/build/80754af9/sqlalchemy_1620712430742/work
statsmodels @ file:///tmp/build/80754af9/statsmodels_1614023746358/work
sympy @ file:///tmp/build/80754af9/sympy_1618252284338/work
tables==3.6.1
tblib @ file:///tmp/build/80754af9/tblib_1597928476713/work
tensorboard==2.10.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorboardX==2.5.1
termcolor==1.1.0
terminado==0.9.4
testpath @ file:///home/ktietz/src/ci/testpath_1611930608132/work
textdistance @ file:///tmp/build/80754af9/textdistance_1612461398012/work
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
three-merge @ file:///tmp/build/80754af9/three-merge_1607553261110/work
tifffile==2020.10.1
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
toolz @ file:///home/linux1/recipes/ci/toolz_1610987900194/work
torch==1.11.0+cu113
torchaudio==0.11.0+cu113
torchvision==0.12.0+cu113
tornado @ file:///tmp/build/80754af9/tornado_1606942300299/work
tqdm==4.60.0
traitlets @ file:///home/ktietz/src/ci/traitlets_1611929699868/work
trimesh==3.10.8
typed-ast @ file:///tmp/build/80754af9/typed-ast_1610484547928/work
typing-extensions @ file:///home/ktietz/src/ci_mi/typing_extensions_1612808209620/work
ujson @ file:///tmp/build/80754af9/ujson_1611259522456/work
unicodecsv==0.14.1
urllib3 @ file:///tmp/build/80754af9/urllib3_1615837158687/work
watchdog @ file:///tmp/build/80754af9/watchdog_1612471027849/work
wcwidth @ file:///tmp/build/80754af9/wcwidth_1593447189090/work
webencodings==0.5.1
Werkzeug @ file:///home/ktietz/src/ci/werkzeug_1611932622770/work
wheel==0.36.2
widgetsnbextension==3.5.1
wrapt==1.12.1
wurlitzer @ file:///tmp/build/80754af9/wurlitzer_1617224664226/work
xlrd @ file:///tmp/build/80754af9/xlrd_1608072521494/work
XlsxWriter @ file:///tmp/build/80754af9/xlsxwriter_1617224712951/work
xlwt==1.3.0
xmltodict==0.12.0
yapf @ file:///tmp/build/80754af9/yapf_1615749224965/work
zict==2.0.0
zipp @ file:///tmp/build/80754af9/zipp_1615904174917/work
zope.event==4.5.0
zope.interface @ file:///tmp/build/80754af9/zope.interface_1616357211867/work
Issue description
I train a neural network multiple times and get multiple event files with tensorboard. On one machine, it runs well by combining all events in a single loss curve. But on the other, it does not combine the events. Both the two machines have tensorboard version 2.10.0. How to further identity the problem?
Does not combine
Combine
Hi @HelloRicky123,
On the machine that didn't combine events, can you please try killing and restating the current TensorBoard process (as described in https://stackoverflow.com/questions/36896164/tensorflow-how-to-close-tensorboard-server)?
If it still doesn't work, please share the full tensorboard command, run the following inspection against the parent log directory on the machine with issue and paste the output here (more details):
tensorboard --inspect --logdir {your_logdir}
Thanks!
