kaolin-wisp
kaolin-wisp copied to clipboard
put wandb setup first in _setup_dashboards and arrange cast data pass…
…ed to wandb.Table into a double-nested list
confirmed
Why:
- training curves, such as
total_lossandrgb_loss, are not plotted properly on a synced wandb dashboard: only a single point was plotted on it log_table()for_WandBdid not work with the following error:
File "/home/atsushi/workspace/kaolin-wisp/app/nerf/main_nerf.py", line 128, in <module>
trainer.train() # Run in headless mode
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 371, in train
self.iterate()
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 330, in iterate
self.end_epoch() # determines if optimization keeps running
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/base_trainer.py", line 308, in end_epoch
self.validate()
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/multiview_trainer.py", line 294, in validate
self.tracker.log_table("args", record_dict, self.epoch)
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/tracker/tracker.py", line 227, in log_table
dashboard.log_table(caption, data, step)
File "/home/atsushi/workspace/kaolin-wisp/wisp/trainers/tracker/tracker.py", line 453, in log_table
table = wandb.Table(columns=data.keys(), data=data.values())
File "/home/atsushi/miniconda3/envs/wisp/lib/python3.9/site-packages/wandb/data_types.py", line 290, in __init__
self._init_from_list(data, columns, optional, dtype)
File "/home/atsushi/miniconda3/envs/wisp/lib/python3.9/site-packages/wandb/data_types.py", line 309, in _init_from_list
assert isinstance(data, list), "data argument expects a `list` object"
AssertionError: data argument expects a `list` object
What:
- Configure a wandb dashboard first if it is enabled because
wandb.init()should be called earlier than instantiatingSummaryWriteaccording to the official description of wandb (link) - Cast
columnsargument into a list anddataargument into a 2D list because thecolumnsargument should be iterable and, both thedataargument and its element should be iterable as well based on the source (link)
P.S. PR #183 was my fault. Sorry about that. Please ignore it