tensorboard
tensorboard copied to clipboard
Published
20 hours ago •
tensorflow
tensorflow
Tensorboard frontend does not update when the log directory changes.
Environment information (required)
Diagnostics
Diagnostics output
--- check: autoidentify
INFO: diagnose_tensorboard.py version b5843ba83bb708385ff54baaab4b2c70c39f7a4f
--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='c8278555cf9a', release='4.19.69-1-MANJARO', version='#1 SMP PREEMPT Thu Aug 29 08:51:46 UTC 2019', machine='x86_64')
INFO: sys.getwindowsversion(): N/A
--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: None
--- check: installed_packages
INFO: installed: tensorboard==1.14.0
diagnose_tensorboard.py:197: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
logging.warn("no installation among: %s", sorted(family))
WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu', 'tf-nightly-gpu-2.0-preview']
WARNING: no installation among: ['tensorflow-estimator', 'tensorflow-estimator-2.0-preview', 'tf-estimator-nightly']
--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '1.14.0'
--- check: tensorflow_python_version
Traceback (most recent call last):
File "diagnose_tensorboard.py", line 419, in main
suggestions.extend(check())
File "diagnose_tensorboard.py", line 77, in wrapper
result = fn()
File "diagnose_tensorboard.py", line 236, in tensorflow_python_version
import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'
--- check: tensorboard_binary_path
INFO: which tensorboard: b'/usr/local/bin/tensorboard\n'
--- check: readable_fqdn
INFO: socket.getfqdn(): 'c8278555cf9a'
--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=19925017, st_dev=48, st_nlink=2, st_uid=0, st_gid=0, st_size=4096, st_atime=1569252142, st_mtime=1569252142, st_ctime=1569252142)
INFO: mode: 0o40777
--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/usr/local/lib/python3.7/site-packages']; bad_roots (0): []
--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.8.0
grpcio==1.23.0
Markdown==3.1.1
numpy==1.17.2
pip==19.2.3
protobuf==3.9.1
setuptools==41.2.0
six==1.12.0
tensorboard==1.14.0
Werkzeug==0.16.0
wheel==0.33.6
Issue description
Please describe the bug as clearly as possible. How can we reproduce the problem without additional resources (including external data files and proprietary Python modules)?
I am trying to run the tensorboard server in a docker container but there seems to be problems with updating the log directory. Tensorboard will load a single file and then be stuck in that state even if I manually go into the container and delete the files. For example, I have 4 folders each with a tfevents file and this is what the resulting frontend is stuck at:
An exception is raised in the Reloader thread during spin up of the server which seems related to the issue. Output of docker run:
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
TensorFlow installation not found - running with reduced feature set.
Exception in thread Reloader:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/application.py", line 430, in _reload
multiplexer.Reload()
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 240, in Reload
Worker()
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 218, in Worker
accumulator.Reload()
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 177, in Reload
for event in self._generator.Load():
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 89, in Load
for event in self._LoadInternal():
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 113, in _LoadInternal
for event in self._loader.Load():
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 95, in Load
yield event_pb2.Event.FromString(record)
google.protobuf.message.DecodeError: Error parsing message
TensorBoard 1.14.0 at http://c8278555cf9a:6006/ (Press CTRL+C to quit)
If i use the official tensorflow image everything works fine. However, the problem with that solution is that the image is too big at 1.2 gigs. I really just need a barebones tensorboard.
Steps to reproduce:
- Use this Dockerfile:
FROM python:3.7-slim
RUN pip install --upgrade pip
RUN pip install tensorboard==1.14
COPY diagnose_tensorboard.py .
VOLUME /logs
EXPOSE 6006
ENTRYPOINT tensorboard --logdir /logs
-
Build the image:
docker build -t my-tb . -
Run a container with mounted tfevent files:
docker run -it --rm -p 6006:6006 --name tb_test -v $(realpath tb_files):/logs my-tb /bin/bash -c "tensorboard --logdir /logs" -
Go to localhost:6006
-
Observe issue. Try deleting tfevent files in the container and notice that the frontend never updates.
The error google.protobuf.message.DecodeError: Error parsing message looks like there's some issue with the actual event file reading. Does that happen every time? It's possible there is some issue with exposing the event files within the container.
The other thing that would be useful is the full logs, from running TensorBoard with --verbosity 1.
It happens 99% of the time I would say. There was one time that it was oddly working. Another piece of information that may be useful is that it seemed to work on OSX but not on Linux machines. I'll get you the full logs .
The logs appear to be the same:
docker run -it --rm -p 6006:6006 --name tb_test -v $(realpath tb_files):/logs my-tb /bin/bash -c "tensorboard --logdir /logs --verbosity 1"
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
TensorFlow installation not found - running with reduced feature set.
Exception in thread Reloader:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/application.py", line 430, in _reload
multiplexer.Reload()
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 240, in Reload
Worker()
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 218, in Worker
accumulator.Reload()
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 177, in Reload
for event in self._generator.Load():
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 89, in Load
for event in self._LoadInternal():
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 113, in _LoadInternal
for event in self._loader.Load():
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 95, in Load
yield event_pb2.Event.FromString(record)
google.protobuf.message.DecodeError: Error parsing message
TensorBoard 1.14.0 at http://135b6062305d:6006/ (Press CTRL+C to quit)
Sorry my entry point was overriding the command that I sent. These are the logs with verbosity enabled:
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
TensorFlow installation not found - running with reduced feature set.
I1001 14:58:28.699232 140225656981312 plugin_event_multiplexer.py:97] Event Multiplexer initializing.
I1001 14:58:28.699336 140225656981312 plugin_event_multiplexer.py:112] Event Multiplexer done initializing
I1001 14:58:28.700659 140225656981312 application.py:447] Launching reload in a daemon thread
I1001 14:58:28.701358 140225077454592 application.py:426] TensorBoard reload process beginning
I1001 14:58:28.701571 140225077454592 plugin_event_multiplexer.py:182] Starting AddRunsFromDirectory: /logs
I1001 14:58:28.702064 140225077454592 io_wrapper.py:196] GetLogdirSubdirectories: Starting to list directories via walking.
I1001 14:58:28.705006 140225077454592 plugin_event_multiplexer.py:184] Adding run from directory /logs/job3
I1001 14:58:28.705146 140225077454592 plugin_event_multiplexer.py:142] Constructing EventAccumulator for /logs/job3
I1001 14:58:28.705740 140225077454592 plugin_event_multiplexer.py:184] Adding run from directory /logs/job4
I1001 14:58:28.705968 140225077454592 plugin_event_multiplexer.py:142] Constructing EventAccumulator for /logs/job4
I1001 14:58:28.706339 140225077454592 plugin_event_multiplexer.py:184] Adding run from directory /logs/job1
I1001 14:58:28.706508 140225077454592 plugin_event_multiplexer.py:142] Constructing EventAccumulator for /logs/job1
I1001 14:58:28.706799 140225077454592 plugin_event_multiplexer.py:184] Adding run from directory /logs/job2
I1001 14:58:28.707018 140225077454592 plugin_event_multiplexer.py:142] Constructing EventAccumulator for /logs/job2
I1001 14:58:28.707223 140225077454592 plugin_event_multiplexer.py:188] Done with AddRunsFromDirectory: /logs
I1001 14:58:28.707343 140225077454592 application.py:429] TensorBoard reload process: Reload the whole Multiplexer
I1001 14:58:28.707461 140225077454592 plugin_event_multiplexer.py:193] Beginning EventMultiplexer.Reload()
I1001 14:58:28.707665 140225077454592 plugin_event_multiplexer.py:238] Reloading runs serially (one after another) on the main thread.
I1001 14:58:28.707915 140225077454592 event_file_loader.py:40] Opening a record reader pointing at /logs/job3/events.out.tfevents.1565885285.foundations-job-f46dcb53-132f-4c37-b68c-afc21c68fdb2-5v77c
I1001 14:58:28.708627 140225077454592 event_file_loader.py:58] Loading events from /logs/job3/events.out.tfevents.1565885285.foundations-job-f46dcb53-132f-4c37-b68c-afc21c68fdb2-5v77c
Exception in thread Reloader:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/application.py", line 430, in _reload
multiplexer.Reload()
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 240, in Reload
Worker()
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 218, in Worker
accumulator.Reload()
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 177, in Reload
for event in self._generator.Load():
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 89, in Load
for event in self._LoadInternal():
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 113, in _LoadInternal
for event in self._loader.Load():
File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 95, in Load
yield event_pb2.Event.FromString(record)
google.protobuf.message.DecodeError: Error parsing message
TensorBoard 1.14.0 at http://d8e8463bce41:6006/ (Press CTRL+C to quit)
Does events.out.tfevents.1565885285.foundations-job-f46dcb53-132f-4c37-b68c-afc21c68fdb2-5v77c happen to be something you could share? If not, any chance you can try running TensorBoard on it outside the container and seeing if it produces the same exception?
Also, does it work if you omit the job3 directory entirely, i.e. do the other 3 folders load successfully in that case or do they hit the same error?
TF Events file (renamed as a .txt so i could upload it): events.out.tfevents.1565885285.foundations-job-f46dcb53-132f-4c37-b68c-afc21c68fdb2-5v77c.txt
I hit the same error (Now with the job4 dir) after removing job3. I took it a step further and removed job4 then job1 as they were the next to fail. Even with just the one job dir it still fails.
I created a fresh environment, installed tensorboard 1.14, then started the server on my PC. Same error.
➜ tb_image pipenv install tensorboard==1.14
Creating a virtualenv for this project…
Pipfile: /home/austin/repos/dessa/tb_image/Pipfile
Using /usr/bin/python (3.7.4) to create virtualenv…
⠏ Creating virtual environment...Already using interpreter /usr/bin/python
Using base prefix '/usr'
New python executable in /home/austin/repos/dessa/tb_image/.venv/bin/python
Installing setuptools, pip, wheel...
done.
✔ Successfully created virtual environment!
Virtualenv location: /home/austin/repos/dessa/tb_image/.venv
Creating a Pipfile for this project…
Installing tensorboard==1.14…
Adding tensorboard to Pipfile's [packages]…
✔ Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success!
Updated Pipfile.lock (c65257)!
Installing dependencies from Pipfile.lock (c65257)…
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 9/9 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
➜ tb_image pipenv run tensorboard --logdir tb_files --verbosity 1
/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
TensorFlow installation not found - running with reduced feature set.
I1001 15:42:19.447333 140081663223424 plugin_event_multiplexer.py:97] Event Multiplexer initializing.
I1001 15:42:19.447424 140081663223424 plugin_event_multiplexer.py:112] Event Multiplexer done initializing
I1001 15:42:19.448043 140081663223424 application.py:447] Launching reload in a daemon thread
I1001 15:42:19.448365 140081080043264 application.py:426] TensorBoard reload process beginning
I1001 15:42:19.448460 140081080043264 plugin_event_multiplexer.py:182] Starting AddRunsFromDirectory: /home/austin/repos/dessa/tb_image/tb_files
I1001 15:42:19.448612 140081080043264 io_wrapper.py:196] GetLogdirSubdirectories: Starting to list directories via walking.
I1001 15:42:19.450169 140081080043264 plugin_event_multiplexer.py:184] Adding run from directory /home/austin/repos/dessa/tb_image/tb_files/job2
I1001 15:42:19.450272 140081080043264 plugin_event_multiplexer.py:142] Constructing EventAccumulator for /home/austin/repos/dessa/tb_image/tb_files/job2
I1001 15:42:19.450345 140081080043264 plugin_event_multiplexer.py:188] Done with AddRunsFromDirectory: /home/austin/repos/dessa/tb_image/tb_files
I1001 15:42:19.450391 140081080043264 application.py:429] TensorBoard reload process: Reload the whole Multiplexer
I1001 15:42:19.450434 140081080043264 plugin_event_multiplexer.py:193] Beginning EventMultiplexer.Reload()
I1001 15:42:19.450495 140081080043264 plugin_event_multiplexer.py:238] Reloading runs serially (one after another) on the main thread.
I1001 15:42:19.450607 140081080043264 event_file_loader.py:40] Opening a record reader pointing at /home/austin/repos/dessa/tb_image/tb_files/job2/events.out.tfevents.1565884781.foundations-job-f46dcb53-132f-4c37-b68c-afc21c68fdb2-5v77c
I1001 15:42:19.450892 140081080043264 event_file_loader.py:58] Loading events from /home/austin/repos/dessa/tb_image/tb_files/job2/events.out.tfevents.1565884781.foundations-job-f46dcb53-132f-4c37-b68c-afc21c68fdb2-5v77c
Exception in thread Reloader:
Traceback (most recent call last):
File "/usr/lib64/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/backend/application.py", line 430, in _reload
multiplexer.Reload()
File "/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 240, in Reload
Worker()
File "/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 218, in Worker
accumulator.Reload()
File "/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 177, in Reload
for event in self._generator.Load():
File "/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 89, in Load
for event in self._LoadInternal():
File "/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 113, in _LoadInternal
for event in self._loader.Load():
File "/home/austin/repos/dessa/tb_image/.venv/lib/python3.7/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 95, in Load
yield event_pb2.Event.FromString(record)
google.protobuf.message.DecodeError: Error parsing message
TensorBoard 1.14.0 at http://austin-pc:6006/ (Press CTRL+C to quit)
I have the same problem. In my case, the files are stored on a network drive which is sometimes slow. Is it possible that there's some timing issue? Tensorboard Version: 1.15.0 Tensorflow is not installed, I'm using Tensorboard with Pytorch instead.
+1 same issue. Tensorboard does not update the directory for example if I first run tensorboard --logdir=./logs and then I run tensorboard --logdir=./logs2 It will stay at logs
