test dataset no work

[windcatcher]

here is my command python3 predict.py --ckpt log/semantic/best_model_epoch_405.ckpt --set=test --num_samples=500

here is the error `Dataset split: test Loading file_prefixes: ['MarketplaceFeldkirch_Station4_rgb_intensity-reduced'] pl_points shape Tensor("Shape:0", shape=(3,), dtype=int32, device=/device:GPU:0)

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

• https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
• https://github.com/tensorflow/addons If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /home/lms/pointNet2/Open3D-PointNet2-Semantic3D/util/tf_util.py:662: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use rate instead of keep_prob. Rate should be set to rate = 1 - keep_prob. 2019-11-06 10:44:50.087783: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2019-11-06 10:44:50.384107: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2019-11-06 10:44:50.384597: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x657ef10 executing computations on platform CUDA. Devices: 2019-11-06 10:44:50.384613: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce GTX 1050, Compute Capability 6.1 2019-11-06 10:44:50.403814: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2808000000 Hz 2019-11-06 10:44:50.404274: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x65e7580 executing computations on platform Host. Devices: 2019-11-06 10:44:50.404346: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): , 2019-11-06 10:44:50.404616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: name: GeForce GTX 1050 major: 6 minor: 1 memoryClockRate(GHz): 1.455 pciBusID: 0000:01:00.0 totalMemory: 1.95GiB freeMemory: 1.90GiB 2019-11-06 10:44:50.404698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 2019-11-06 10:44:50.406574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-11-06 10:44:50.406587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 2019-11-06 10:44:50.406593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N 2019-11-06 10:44:50.406662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1724 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1) WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. Model restored Processing <dataset.semantic_dataset.SemanticFileData object at 0x7f485ff25400> 2019-11-06 10:44:53.848523: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.17GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2019-11-06 10:44:53.875321: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.02GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2019-11-06 10:44:54.036662: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.16GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2019-11-06 10:44:54.108900: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.13GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2019-11-06 10:44:54.127858: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.32GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. Batch size: 32, time: 1.9697668552398682 Batch size: 32, time: 0.5003781318664551 Batch size: 32, time: 0.49585819244384766 Batch size: 32, time: 0.5022487640380859 Batch size: 32, time: 0.4994063377380371 Batch size: 32, time: 0.4927208423614502 Batch size: 32, time: 0.49471569061279297 Batch size: 32, time: 0.498868465423584 Batch size: 32, time: 0.49785780906677246 Batch size: 32, time: 0.4957921504974365 Batch size: 32, time: 0.49452805519104004 Batch size: 32, time: 0.49374890327453613 Batch size: 32, time: 0.49533581733703613 Batch size: 32, time: 0.49709129333496094 Batch size: 32, time: 0.4983334541320801 2019-11-06 10:45:31.690487: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.77GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2019-11-06 10:45:31.705401: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.65GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2019-11-06 10:45:31.814868: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.11GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2019-11-06 10:45:31.874898: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.10GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2019-11-06 10:45:31.888066: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.22GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. Batch size: 20, time: 0.6125662326812744 Exported sparse pcd to result/sparse/MarketplaceFeldkirch_Station4_rgb_intensity-reduced.pcd Exported sparse labels to result/sparse/MarketplaceFeldkirch_Station4_rgb_intensity-reduced.labels Confusion matrix: 0 1 2 3 4 5 6 7 8 0 0 730814 821 29381 125951 3018863 114555 68655 6960 1 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 IoU per class: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] mIoU (ignoring label 0): 0.0 Overall accuracy /home/lms/pointNet2/Open3D-PointNet2-Semantic3D/util/metric.py:83: RuntimeWarning: invalid value encountered in long_scalars return np.trace(valid_confusion_matrix) / np.sum(valid_confusion_matrix) nan`