FPN_TensorFlow
FPN_TensorFlow copied to clipboard
It seems stop when I train my data
Output is as follows :
+ set -e
+ export PYTHONUNBUFFERED=True
+ PYTHONUNBUFFERED=True
+ GPU_ID=0
+ DATASET=cow
+ array=($@)
+ len=2
+ EXTRA_ARGS=
+ EXTRA_ARGS_SLUG=
++ date +%Y_%m_%d_%H_%M_%S
+ LOG=logs/FPN_cow.txt.2018_08_31_14_51_58
+ exec
++ tee -a logs/FPN_cow.txt.2018_08_31_14_51_58
tee: logs/FPN_cow.txt.2018_08_31_14_51_58: No such file or directory
+ echo Logging output to logs/FPN_cow.txt.2018_08_31_14_51_58
Logging output to logs/FPN_cow.txt.2018_08_31_14_51_58
+ CUDA_VISIBLE_DEVICES=0
+ python ./tools/train.py
tfrecord path is --> /home/dongpeijie/FPN_TensorFlow/data/tfrecords/cow_train*
/home/dongpeijie/miniconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
model restore from pretrained mode, path is: data/pretrained_weights/resnet_v1_101.ckpt
2018-08-31 14:52:26.532747: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-08-31 14:52:26.532790: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-08-31 14:52:26.532798: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-08-31 14:52:26.532803: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-08-31 14:52:26.532808: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2018-08-31 14:52:28.340651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties:
name: Tesla P100-PCIE-16GB
major: 6 minor: 0 memoryClockRate (GHz) 1.3285
pciBusID 0000:06:00.0
Total memory: 15.89GiB
Free memory: 15.60GiB
2018-08-31 14:52:28.340709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0
2018-08-31 14:52:28.340716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: Y
2018-08-31 14:52:28.340727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0)
restore model
I have waited for a long time without other output information.
Thank you for helping me solving the problem.
You need to check your file path is correct or not.