Data damage?

[YangSiri]

Hi, @xingyul , thanks for your sharing codes and it is great. But i got a little trouble while testing it, could you or anyone help me? I downloaded the processed data of flyingtings3d which is around 11GB, i extracted it and put the folder under data_processing, and then i run 'sh command_train.sh" as you suggested.There is the error :

2019-09-30 13:38:13.146499: W tensorflow/core/framework/allocator.cc:108] Allocation of 33554432 exceeds 10% of system memory. -- 010 / 032 -- mean loss: 1.026778 Traceback (most recent call last): File "train.py", line 283, in train() File "train.py", line 155, in train train_one_epoch(sess, ops, train_writer) File "train.py", line 201, in train_one_epoch batch_data, batch_label, batch_mask = get_batch(TRAIN_DATASET, train_idxs, start_idx, end_idx) File "train.py", line 173, in get_batch pc1, pc2, color1, color2, flow, mask1 = dataset[idxs[i+start_idx]] File "/home/py001/cyz/flownet3d/flying_things_dataset.py", line 36, in getitem data = np.load(fp) File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/site-packages/numpy/lib/npyio.py", line 426, in load pickle_kwargs=pickle_kwargs) File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/site-packages/numpy/lib/npyio.py", line 182, in init _zip = zipfile_factory(fid) File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/site-packages/numpy/lib/npyio.py", line 112, in zipfile_factory return zipfile.ZipFile(file, *args, **kwargs) File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/zipfile.py", line 1026, in init self._RealGetContents() File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/zipfile.py", line 1093, in _RealGetContents raise BadZipFile("File is not a zip file") zipfile.BadZipFile: File is not a zip file Exception ignored in: <bound method NpzFile.del of <numpy.lib.npyio.NpzFile object at 0x7f3f12e80630>> Traceback (most recent call last): File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/site-packages/numpy/lib/npyio.py", line 219, in del self.close() File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/site-packages/numpy/lib/npyio.py", line 210, in close if self.zip is not None: AttributeError: 'NpzFile' object has no attribute 'zip

Is there anything wrong with the data i downloaded ?

[xingyul]

Can you print out the filename of the .npz file that cause the problem?

[YangSiri]

Hi @xingyul , Thanks for answering me. But I think none file is in processing... Doesn't this problem occur as long as the first data file is read?

[xingyul]

Hi, if you take a look at the dataset file, the code read each npz file one by one. From the error message you showed, it seems like there is an error during the reading of the npz file. It would be helpful to check if the npz filename is correct.

[YangSiri]

Hi @xingyul thanks for your attention. This is the processed data folder, i dont spot any mistake?