TypeError: Object of type ndarray is not JSON serializable

[ZaiwangLi]

Met this problem when run the train command: train with single GPU python ./pytorch/train.py train --config_path=./configs/car.fhd.config --model_dir=/path/to/model_dir

Traceback (most recent call last): File "./pytorch/train.py", line 306, in train ret_dict = net_parallel(example_torch) File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/root/second.pytorch/second/pytorch/models/voxelnet.py", line 363, in forward preds_dict = self.network_forward(voxels, num_points, coors, batch_size_dev) File "/root/second.pytorch/second/pytorch/models/voxelnet.py", line 332, in network_forward voxel_features, coors, batch_size) File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/root/second.pytorch/second/pytorch/models/middle.py", line 203, in forward ret = self.middle_conv(ret) File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/root/anaconda3/lib/python3.7/site-packages/spconv/modules.py", line 130, in forward input = module(input) File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/root/anaconda3/lib/python3.7/site-packages/spconv/conv.py", line 177, in forward use_hash=self.use_hash) File "/root/anaconda3/lib/python3.7/site-packages/spconv/ops.py", line 93, in get_indice_pairs stride, padding, dilation, out_padding, int(subm), int(transpose), int(use_hash)) RuntimeError: /root/second.pytorch/second/spconv/src/spconv/indice.cu 165 cuda execution failed with error 8 invalid device function prepareSubMHashKernel failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./pytorch/train.py", line 663, in fire.Fire() File "/root/anaconda3/lib/python3.7/site-packages/fire/core.py", line 138, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/root/anaconda3/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire target=component.name) File "/root/anaconda3/lib/python3.7/site-packages/fire/core.py", line 675, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "./pytorch/train.py", line 421, in train print(json.dumps(example["metadata"], indent=2)) File "/root/anaconda3/lib/python3.7/json/init.py", line 238, in dumps **kw).encode(obj) File "/root/anaconda3/lib/python3.7/json/encoder.py", line 201, in encode chunks = list(chunks) File "/root/anaconda3/lib/python3.7/json/encoder.py", line 429, in _iterencode yield from _iterencode_list(o, _current_indent_level) File "/root/anaconda3/lib/python3.7/json/encoder.py", line 325, in _iterencode_list yield from chunks File "/root/anaconda3/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict yield from chunks File "/root/anaconda3/lib/python3.7/json/encoder.py", line 438, in _iterencode o = _default(o) File "/root/anaconda3/lib/python3.7/json/encoder.py", line 179, in default raise TypeError(f'Object of type {o.class.name} ' TypeError: Object of type ndarray is not JSON serializable

[451578928]

Did you slove this problem？

[simaker]

I had the same problem, but in my case the Memory of the GPU was completely full. Training on a RTX 2070 solved the problem

[deepincold]

@simaker have you changed the batch size or anything else? I met the same error using RTX 2070. and by using nvidia-smi I saw the memory is used up. Could you share your runtime version of cuda, pytorch etc? Thank you~