kaolin-wisp icon indicating copy to clipboard operation
kaolin-wisp copied to clipboard

put wandb setup first in _setup_dashboards and arrange cast data pass…

Open barikata1984 opened this issue 2 years ago • 2 comments

…ed to wandb.Table into a double-nested list

barikata1984 avatar Nov 07 '23 04:11 barikata1984

confirmed

barikata1984 avatar Nov 07 '23 04:11 barikata1984

Why:

  1. training curves, such as total_loss and rgb_loss, are not plotted properly on a synced wandb dashboard: only a single point was plotted on it
  2. log_table() for _WandB did not work with the following error:
  File "/home/atsushi/workspace/kaolin-wisp/app/nerf/main_nerf.py", line 128, in <module>
    trainer.train()  # Run in headless mode
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 371, in train
    self.iterate()
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 330, in iterate
    self.end_epoch()    # determines if optimization keeps running
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 308, in end_epoch
    self.validate()
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/multiview_trainer.py", line 294, in validate
    self.tracker.log_table("args", record_dict, self.epoch)
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/tracker/tracker.py", line 227, in log_table
    dashboard.log_table(caption, data, step)
  File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/tracker/tracker.py", line 453, in log_table
    table = wandb.Table(columns=data.keys(), data=data.values())
  File "/home/atsushi/miniconda3/envs/wisp/lib/python3.9/site-packages/wandb/data_types.py", line 290, in __init__
    self._init_from_list(data, columns, optional, dtype)
  File "/home/atsushi/miniconda3/envs/wisp/lib/python3.9/site-packages/wandb/data_types.py", line 309, in _init_from_list
    assert isinstance(data, list), "data argument expects a `list` object"
AssertionError: data argument expects a `list` object

What:

  1. Configure a wandb dashboard first if it is enabled because wandb.init() should be called earlier than instantiating SummaryWrite according to the official description of wandb (link)
  2. Cast columns argument into a list and data argument into a 2D list because the columns argument should be iterable and, both the data argument and its element should be iterable as well based on the source (link)

P.S. PR #183 was my fault. Sorry about that. Please ignore it

barikata1984 avatar Nov 07 '23 05:11 barikata1984