Error training on custom data

[briancantwe]

Hello! I've managed to get my test data to a point where it trains most of the way, but I'm getting this error.

2024-01-27 09:43:10.564616 easyvolcap.utils.net_utils -> save_npz: Saved model data/trained_model/l3mhet_test/latest.npz at epoch 59 net_utils.py:449 l3mhet_test 0:00:02 59 29959 0.029935 13.920692 0.020401 0.050336 0.0011 0.0598 0.002006 3361 0:00:01 59 29969 0.029938 13.879657 0.020609 0.050547 0.0010 0.0600 0.002005 3361 0:00:01 59 29979 0.029935 13.817179 0.020912 0.050847 0.0008 0.0614 0.002003 3361 0:00:00 59 29989 0.029906 13.771573 0.021099 0.051005 0.0010 0.0579 0.002002 3361 0:00:00 59 29999 0.029902 13.829483 0.020795 0.050696 0.0011 0.0494 0.002000 3361 eta epoch iter prop_loss psnr img_loss loss data batch lr max_mem 2024-01-27 09:43:12.650359 easyvolcap.runners.evaluators.volumetric_video_evaluator -> evaluate: camera: 0 frame: 0 volumetric_video_evaluator.py:46 {'psnr': 10.659576416015625, 'ssim': 0.08238379, 'lpips': 0.6307356953620911} 2024-01-27 09:43:13.742178 easyvolcap.runners.volumetric_video_runner -> 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 0:00:03 < 0:00:00 ? it/s v… test_generator: 2024-01-27 09:43:13.744583 easyvolcap.runners.evaluators.volumetric_video_evaluator -> summarize: volumetric_video_evaluator.py:72 { 'psnr_mean': 10.659576416015625, 'psnr_std': 0.0, 'ssim_mean': 0.08238379657268524, 'ssim_std': 7.450580596923828e-09, 'lpips_mean': 0.6307356953620911, 'lpips_std': 0.0 } 2024-01-27 09:43:13.748727 easyvolcap.runners.volumetric_video_runner -> train: Error in validation pass, ignored and volumetric_video_runner.py:308 continuing ╭─────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮ │ /mnt/c/Users/User/Documents/Github/EasyVolcap/easyvolcap/runners/volumetric_video_runner.py:306 in train │ │ │ │ ❱ 306 │ │ │ │ │ self.test_epoch(epoch + 1) # will this provoke a live display? │ │ │ │ /mnt/c/Users/Use/Documents/Github/EasyVolcap/easyvolcap/runners/volumetric_video_runner.py:405 in test_epoch │ │ │ │ ❱ 405 │ │ for _ in test_generator: pass # the actual calling │ │ │ │ /mnt/c/Users/User/Documents/Github/EasyVolcap/easyvolcap/runners/volumetric_video_runner.py:432 in test_generator │ │ │ │ ❱ 432 │ │ scalar_stats = self.evaluator.summarize() │ │ │ │ /mnt/c/Users/User/Documents/Github/EasyVolcap/easyvolcap/runners/evaluators/volumetric_video_evaluator.py:80 in summarize │ │ │ │ ❱ 80 │ │ │ │ json.dump(metric, f, indent=4) │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ TypeError: 0.08238379 is not JSON serializable (easyvolcap) User@home:/mnt/c/Users/User/Documents/Github/EasyVolcap$

Any ideas?

Thanks!