KittiBox
KittiBox copied to clipboard
MarvinTeichmann
ran out of memory
Hi, I have the problem of running out of memory, as I am testing it on a laptop (GPU memory 2G). Does it run with CPU only tensorflow? Or how can I reduce the computing load?
(tensorflow) shawn@governor-ThinkPad-P50:~/Programs/KittiBox$ python demo.py --input_image data/demo.png
2018-02-08 20:01:29,354 INFO No environment variable 'TV_PLUGIN_DIR' found. Set to '/home/nwx8/tv-plugins'.
2018-02-08 20:01:29,354 INFO No environment variable 'TV_STEP_SHOW' found. Set to '50'.
2018-02-08 20:01:29,355 INFO No environment variable 'TV_STEP_EVAL' found. Set to '250'.
2018-02-08 20:01:29,355 INFO No environment variable 'TV_STEP_WRITE' found. Set to '1000'.
2018-02-08 20:01:29,355 INFO No environment variable 'TV_MAX_KEEP' found. Set to '10'.
2018-02-08 20:01:29,355 INFO No environment variable 'TV_STEP_STR' found. Set to 'Step {step}/{total_steps}: loss = {loss_value:.2f}; lr = {lr_value:.2e}; {sec_per_batch:.3f} sec (per Batch); {examples_per_sec:.1f} imgs/sec'.
2018-02-08 20:01:29,355 INFO f: <_io.TextIOWrapper name='RUNS/KittiBox_pretrained/model_files/hypes.json' mode='r' encoding='UTF-8'>
2018-02-08 20:01:29,356 INFO Hypes loaded successfully.
2018-02-08 20:01:29,357 INFO Modules loaded successfully. Starting to build tf graph.
npy file loaded
Layer name: conv1_1
Layer shape: (3, 3, 3, 64)
2018-02-08 20:01:31,717 INFO Creating Summary for: conv1_1/filter
2018-02-08 20:01:31,732 INFO Creating Summary for: conv1_1/biases
Layer name: conv1_2
Layer shape: (3, 3, 64, 64)
2018-02-08 20:01:31,753 INFO Creating Summary for: conv1_2/filter
2018-02-08 20:01:31,765 INFO Creating Summary for: conv1_2/biases
Layer name: conv2_1
Layer shape: (3, 3, 64, 128)
2018-02-08 20:01:31,785 INFO Creating Summary for: conv2_1/filter
2018-02-08 20:01:31,798 INFO Creating Summary for: conv2_1/biases
Layer name: conv2_2
Layer shape: (3, 3, 128, 128)
2018-02-08 20:01:31,821 INFO Creating Summary for: conv2_2/filter
2018-02-08 20:01:31,833 INFO Creating Summary for: conv2_2/biases
Layer name: conv3_1
Layer shape: (3, 3, 128, 256)
2018-02-08 20:01:31,868 INFO Creating Summary for: conv3_1/filter
2018-02-08 20:01:31,881 INFO Creating Summary for: conv3_1/biases
Layer name: conv3_2
Layer shape: (3, 3, 256, 256)
2018-02-08 20:01:31,937 INFO Creating Summary for: conv3_2/filter
2018-02-08 20:01:31,951 INFO Creating Summary for: conv3_2/biases
Layer name: conv3_3
Layer shape: (3, 3, 256, 256)
2018-02-08 20:01:31,991 INFO Creating Summary for: conv3_3/filter
2018-02-08 20:01:32,004 INFO Creating Summary for: conv3_3/biases
Layer name: conv4_1
Layer shape: (3, 3, 256, 512)
2018-02-08 20:01:32,107 INFO Creating Summary for: conv4_1/filter
2018-02-08 20:01:32,122 INFO Creating Summary for: conv4_1/biases
Layer name: conv4_2
Layer shape: (3, 3, 512, 512)
2018-02-08 20:01:32,401 INFO Creating Summary for: conv4_2/filter
2018-02-08 20:01:32,417 INFO Creating Summary for: conv4_2/biases
Layer name: conv4_3
Layer shape: (3, 3, 512, 512)
2018-02-08 20:01:32,720 INFO Creating Summary for: conv4_3/filter
2018-02-08 20:01:32,737 INFO Creating Summary for: conv4_3/biases
Layer name: conv5_1
Layer shape: (3, 3, 512, 512)
2018-02-08 20:01:32,970 INFO Creating Summary for: conv5_1/filter
2018-02-08 20:01:32,985 INFO Creating Summary for: conv5_1/biases
Layer name: conv5_2
Layer shape: (3, 3, 512, 512)
2018-02-08 20:01:33,148 INFO Creating Summary for: conv5_2/filter
2018-02-08 20:01:33,163 INFO Creating Summary for: conv5_2/biases
Layer name: conv5_3
Layer shape: (3, 3, 512, 512)
2018-02-08 20:01:33,363 INFO Creating Summary for: conv5_3/filter
2018-02-08 20:01:33,379 INFO Creating Summary for: conv5_3/biases
Layer name: fc6
Layer shape: [7, 7, 512, 4096]
2018-02-08 20:01:51,219 INFO Creating Summary for: fc6/weights
2018-02-08 20:01:51,233 INFO Creating Summary for: fc6/biases
Layer name: fc7
Layer shape: [1, 1, 4096, 4096]
2018-02-08 20:01:53,957 INFO Creating Summary for: fc7/weights
2018-02-08 20:01:53,973 INFO Creating Summary for: fc7/biases
2018-02-08 20:01:53,994 INFO Creating Summary for: score_fr/weights
2018-02-08 20:01:54,006 INFO Creating Summary for: score_fr/biases
2018-02-08 20:01:54,077 WARNING From /home/nwx8/Programs/KittiBox/incl/tensorflow_fcn/fcn8_vgg.py:114: calling argmax (from tensorflow.python.ops.math_ops) with dimension is deprecated and will be removed in a future version.
Instructions for updating:
Use the axis argument instead
2018-02-08 20:01:54,086 INFO Creating Summary for: upscore2/up_filter
2018-02-08 20:01:54,105 INFO Creating Summary for: score_pool4/weights
2018-02-08 20:01:54,117 INFO Creating Summary for: score_pool4/biases
2018-02-08 20:01:54,140 INFO Creating Summary for: upscore4/up_filter
2018-02-08 20:01:54,160 INFO Creating Summary for: score_pool3/weights
2018-02-08 20:01:54,173 INFO Creating Summary for: score_pool3/biases
2018-02-08 20:01:54,194 INFO Creating Summary for: upscore32/up_filter
2018-02-08 20:01:54,541 INFO Graph build successfully.
2018-02-08 20:01:54.541597: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-02-08 20:01:54.619046: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:895] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-02-08 20:01:54.619518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties:
name: Quadro M1000M major: 5 minor: 0 memoryClockRate(GHz): 1.0715
pciBusID: 0000:01:00.0
totalMemory: 1.96GiB freeMemory: 1.59GiB
2018-02-08 20:01:54.619534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Quadro M1000M, pci bus id: 0000:01:00.0, compute capability: 5.0)
2018-02-08 20:01:55,209 INFO /home/mifs/mttt2/local_disk/RUNS/TensorDetect2/paper_bench/tau5_zoom_0_kitti_2016_11_09_05.57/model.ckpt-179999
2018-02-08 20:01:55,209 INFO Restoring parameters from RUNS/KittiBox_pretrained/model.ckpt-179999
2018-02-08 20:01:59,947 INFO Weights loaded successfully.
2018-02-08 20:01:59,947 INFO Starting inference using data/demo.png as input
2018-02-08 20:02:00.910205: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.03GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2018-02-08 20:02:01.099306: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.09GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2018-02-08 20:02:01.221222: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 526.50MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2018-02-08 20:02:01.377088: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.55GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2018-02-08 20:02:01.573297: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2018-02-08 20:02:01.731673: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 807.75MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2018-02-08 20:02:01.920731: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.06GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
Traceback (most recent call last):
File "demo.py", line 219, in
Thank you. Shawn
