benjaminrwilson
ValueError: Found zero norm quaternions in 'quat'.
The error occurs in the end of an training epoch. The traceback is shown below, When I look into the highlighted line, it seem that the input args dts and gts are zeros and none. Therefore, I wonder if there is some error for my configuration and path, the src_dir I used is ".../argoverse-sensor/" which contains all train and validation data. May I ask for more suggestions on how to fix the error or some details on configuration file? Thanks a lot!
Traceback:
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py",line 811,in_fit_impl results = self. run(model, ckpt path=self.ckpt path)
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py",line 1236,in_run results=self. run stage()
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer-py",line 1323,in_run_stage return self. run train()
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer-py",line 1353,in_run_train self.fit loop.run()
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/loops/base.py",line 204,in run self.advance(*args,**kwargs)
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop-py", line 268, in advance self.outputs=self.epoch_loop.run(self. data_fetcher)
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch 1ightning/loops/base.py",line 205,in run self.on advance end()
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py",line 255, in on advance end self. run validation()
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop-py",line 309,in_run_validation self.val loop.run()
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/loops/base.py",line 211,in run output=self.on run end()
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop-py",line 188, in on_run_end self._evaluation_epoch_end(self._outputs)
File"/GPFS/rhone/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop-py",line 310,in_evaluation_epoch_end self.trainer. call_lightning_module_hook("validation_epoch_end", output_or_outputs)
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py",line 1595,in_call_lightning_module_hook output=fn(*args,**kwargs)
File"/GPFS/rhome/xiaomingwang/torchbox3d/src/torchbox3d/nn/arch/centerpoint.py",line 278,in validation_epoch_end eval dts, eval gts, metrics = evaluate(
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/site-packages/av2/evaluation/detection/eval.py",line 165,in evaluate outputs: Optional[List[Tuple[NDArrayFloat,NDArrayFloat]]] = p.starmap(accumulate,args_list)
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/1ib/python3.9/multiprocessing/pool.py",line 372,in starmap return self. map_async(func,iterable,starmapstar,chunksize)-get()
File"/GPFS/rhome/xiaomingwang/miniconda3/envs/torchbox3d/lib/python3.9/multiprocessing/pool.py",line 771,in get raise self.value
ValueError:Found zero norm quaternions in quat.
