Open3D installation considerably slows down Tensorboard start up

[cgsaxner]

Checklist

• [X] I have searched for similar issues.
• [X] For Python issues, I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for master branch).

Describe the issue

I have a conda environment under Windows 10 for some deep learning application using PyTorch and Tensorboard. When I install Open3D within this environment, Tensorboard becomes very very slow to load, basically unusable. Each time I run tensorboard --logdir runs, I get the usual message of TensorFlow installation not found - running with reduced feature set., after which usually, the message TensorBoard 2.8.0 at http://localhost:6006/ (Press CTRL+C to quit) appears, and I can monitor my training in the browser. After installing Open3D, the second message never pops up, and the browser page can't be reached. I once left it running over night, and in the morning Tensorboard was loaded successfully, but otherwise so success. When I interrupt the start up using Ctrl+C, I get a traceback to some Open3d tensorboard plugin / tensorboard script (see below).

Steps to reproduce the bug

Create a conda environment using Python 3.7
Install PyTorch and Tensorboard using conda
Run `tensorboard --logdir runs` -> Tensorboard starts up normally within seconds
Install Open3d using  `conda install -c open3d-admin open3d` OR `pip install --user open3d` OR `pip install --user --pre https://storage.googleapis.com/open3d-releases-master/python-wheels/open3d-0.15.1-cp37-cp37m-win_amd64.whl`
Run `tensorboard --logdir runs` again -> Tensorboard takes forever to start up (> 1 hour)

Error message

Traceback (most recent call last): File "C:\Users\chris\Miniconda3\envs\Inpainting\Scripts\tensorboard-script.py", line 9, in <module> sys.exit(run_main()) File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\main.py", line 46, in run_main app.run(tensorboard.main, flags_parser=tensorboard.configure) File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\absl\app.py", line 312, in run _run_main(main, args) File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\absl\app.py", line 258, in _run_main sys.exit(main(argv)) File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\program.py", line 276, in main return runner(self.flags) or 0 File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\program.py", line 292, in _run_serve_subcommand server = self._make_server() File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\program.py", line 481, in _make_server deprecated_multiplexer, File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\backend\application.py", line 123, in TensorBoardWSGIApp plugin = loader.load(context) File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\plugins\base_plugin.py", line 362, in load return self.plugin_class(context) File "C:\Users\chris\AppData\Roaming\Python\Python37\site-packages\open3d\visualization\tensorboard_plugin\plugin.py", line 461, in __init__ self.data_reader = Open3DPluginDataReader(self._logdir) File "C:\Users\chris\AppData\Roaming\Python\Python37\site-packages\open3d\visualization\tensorboard_plugin\util.py", line 225, in __init__ self.reload_events() File "C:\Users\chris\AppData\Roaming\Python\Python37\site-packages\open3d\visualization\tensorboard_plugin\util.py", line 230, in reload_events self.event_mux.Reload() File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\backend\event_processing\plugin_event_multiplexer.py", line 255, in Reload Worker() File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\backend\event_processing\plugin_event_multiplexer.py", line 233, in Worker accumulator.Reload() File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\backend\event_processing\plugin_event_accumulator.py", line 177, in Reload for event in self._generator.Load(): File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\backend\event_processing\directory_watcher.py", line 88, in Load for event in self._LoadInternal(): File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\backend\event_processing\directory_watcher.py", line 118, in _LoadInternal for event in self._loader.Load(): File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\backend\event_processing\event_file_loader.py", line 189, in Load for event in super(EventFileLoader, self).Load(): File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\backend\event_processing\event_file_loader.py", line 163, in Load for record in super(LegacyEventFileLoader, self).Load(): File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\backend\event_processing\event_file_loader.py", line 138, in Load yield next(self._iterator) File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\backend\event_processing\event_file_loader.py", line 109, in __next__ self._reader.GetNext() File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\compat\tensorflow_stub\pywrap_tensorflow.py", line 234, in GetNext event_crc_calc = masked_crc32c(event_str) File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\compat\tensorflow_stub\pywrap_tensorflow.py", line 40, in masked_crc32c x = u32(crc32c(data)) File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\compat\tensorflow_stub\pywrap_tensorflow.py", line 165, in crc32c return crc_finalize(crc_update(CRC_INIT, data)) File "C:\Users\chris\Miniconda3\envs\Inpainting\lib\site-packages\tensorboard\compat\tensorflow_stub\pywrap_tensorflow.py", line 140, in crc_update table_index = (crc ^ b) & 0xFF KeyboardInterrupt

Expected behavior

Open3D does not slow down Tensorboard start up.

Open3D, Python and System information

- Operating system: Windows 10 64-bit
- Python version: Python 3.7.12
- Open3D version: 0.15.1
- PyTorch version: 1.9.1
- Tensorboard version: 2.8.0
- System architecture: 64bit
- Is this a remote workstation?: no
- How did you install Open3D?: tested with conda and pip (including latest development wheel)
- Compiler version (if built from source): na

Additional information

No response

[ssheorey]

Hi @cgsaxner thanks for reporting this. Can you try with an empty runs directory? Can you provide the minimal contents of the runs directory that slows down tensorboard?

[cgsaxner]

Hi @ssheorey, thanks for getting back to me! I tried with an empty runs directory, and indeed, Tensorboard starts up without a problem or noticable delay. Then, already adding one run file leads to a noticable delay of around half a minute for a 3 MB events file, and already around 5 minutes for a larger file (~400 MB). Start up time then increases for every added run, e.g., for 10 run files of mixed sizes start up takes around 20 Minutes. My original folder with which I first noticed the problem contains 50 runs and is ~7 GB large.

[ssheorey]

On Windows, tensorboard does not use fast data loading by default (--load_fast=true) https://github.com/tensorflow/tensorboard/issues/4784 I suspect this may be amplified by the Open3D plugin that needs to check each run to see if there is any Open3D data to display. I'll look into this.

As a workaround, can you install and run tensorboard in a separate virtual env (say tb_only)?

conda run -n tb_only tensorboard --logdir runs

[cgsaxner]

Hi! I installed tensorboard in a separate, clean env, and strangely, the issue persists! It looks like tensorboard still somehow finds the Open3D tensorboard Plugin outside of my conda environment (I again pasted the stack trace below, it's basically the same as above). Only completely removing Open3D 0.15 from my PC resolves the problem.

Traceback (most recent call last): File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\Scripts\tensorboard-script.py", line 9, in <module> sys.exit(run_main()) File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\main.py", line 46, in run_main app.run(tensorboard.main, flags_parser=tensorboard.configure) File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\absl\app.py", line 312, in run _run_main(main, args) File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\absl\app.py", line 258, in _run_main sys.exit(main(argv)) File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\program.py", line 276, in main return runner(self.flags) or 0 File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\program.py", line 292, in _run_serve_subcommand server = self._make_server() File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\program.py", line 481, in _make_server deprecated_multiplexer, File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\backend\application.py", line 123, in TensorBoardWSGIApp plugin = loader.load(context) File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\plugins\base_plugin.py", line 362, in load return self.plugin_class(context) File "C:\Users\chris\AppData\Roaming\Python\Python37\site-packages\open3d\visualization\tensorboard_plugin\plugin.py", line 461, in __init__ self.data_reader = Open3DPluginDataReader(self._logdir) File "C:\Users\chris\AppData\Roaming\Python\Python37\site-packages\open3d\visualization\tensorboard_plugin\util.py", line 225, in __init__ self.reload_events() File "C:\Users\chris\AppData\Roaming\Python\Python37\site-packages\open3d\visualization\tensorboard_plugin\util.py", line 230, in reload_events self.event_mux.Reload() File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\backend\event_processing\plugin_event_multiplexer.py", line 255, in Reload Worker() File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\backend\event_processing\plugin_event_multiplexer.py", line 233, in Worker accumulator.Reload() File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\backend\event_processing\plugin_event_accumulator.py", line 177, in Reload for event in self._generator.Load(): File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\backend\event_processing\directory_watcher.py", line 88, in Load for event in self._LoadInternal(): File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\backend\event_processing\directory_watcher.py", line 118, in _LoadInternal for event in self._loader.Load(): File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\backend\event_processing\event_file_loader.py", line 189, in Load for event in super(EventFileLoader, self).Load(): File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\backend\event_processing\event_file_loader.py", line 163, in Load for record in super(LegacyEventFileLoader, self).Load(): File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\backend\event_processing\event_file_loader.py", line 138, in Load yield next(self._iterator) File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\backend\event_processing\event_file_loader.py", line 109, in __next__ self._reader.GetNext() File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\compat\tensorflow_stub\pywrap_tensorflow.py", line 234, in GetNext event_crc_calc = masked_crc32c(event_str) File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\compat\tensorflow_stub\pywrap_tensorflow.py", line 40, in masked_crc32c x = u32(crc32c(data)) File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\compat\tensorflow_stub\pywrap_tensorflow.py", line 165, in crc32c return crc_finalize(crc_update(CRC_INIT, data)) File "C:\Users\chris\Miniconda3\envs\tensorboard_clean\lib\site-packages\tensorboard\compat\tensorflow_stub\pywrap_tensorflow.py", line 141, in crc_update crc = (CRC_TABLE[table_index] ^ (crc >> 8)) & _MASK KeyboardInterrupt

[weizhuowang]

I have the same issue, any solution figured out?

[sxyu]

yeah this is a huge problem! if Open3D is installed tensorboard becomes unusable completely I have to use a dedicated tensorboard env