kerlomz
i got this error when training
INFO:tensorflow:Step: 400 Time: 0.846 sec/batch, Cost = 1.53476572, BatchSize: 64, Shape[1]: 18 INFO:tensorflow:Step: 500 Time: 0.792 sec/batch, Cost = 1.08882499, BatchSize: 64, Shape[1]: 18 2019-12-23 15:55:03.028956: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 615.23MiB (rounded to 645120000). Current allocation summary follows. 2019-12-23 15:55:03.035027: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): Total Chunks: 128, Chunks in use: 127. 32.0KiB allocated for chunks. 31.8KiB in use in bin. 15.6KiB client-requested in use in bin. 2019-12-23 15:55:03.042178: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): Total Chunks: 43, Chunks in use: 43. 21.5KiB allocated for chunks. 21.5KiB in use in bin. 17.8KiB client-requested in use in bin. 2019-12-23 15:55:03.047903: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.3KiB allocated for chunks. 1.3KiB in use in bin. 1.0KiB client-requested in use in bin. 2019-12-23 15:55:03.053329: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): Total Chunks: 4, Chunks in use: 4. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 13.5KiB client-requested in use in bin. 2019-12-23 15:55:03.060581: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): Total Chunks: 5, Chunks in use: 4. 31.8KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin. 2019-12-23 15:55:03.068945: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.076905: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): Total Chunks: 9, Chunks in use: 8. 175.0KiB allocated for chunks. 157.0KiB in use in bin. 145.5KiB client-requested in use in bin. 2019-12-23 15:55:03.084378: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): Total Chunks: 5, Chunks in use: 4. 160.0KiB allocated for chunks. 128.0KiB in use in bin. 128.0KiB client-requested in use in bin. 2019-12-23 15:55:03.091372: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.098174: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): Total Chunks: 17, Chunks in use: 16. 3.11MiB allocated for chunks. 2.90MiB in use in bin. 2.77MiB client-requested in use in bin. 2019-12-23 15:55:03.105264: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.112143: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.118394: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.125309: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.131695: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): Total Chunks: 1, Chunks in use: 0. 7.61MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.139277: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): Total Chunks: 1, Chunks in use: 0. 8.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.145077: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.151896: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.159025: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.165895: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): Total Chunks: 1, Chunks in use: 0. 128.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2019-12-23 15:55:03.172894: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): Total Chunks: 4, Chunks in use: 1. 1.96GiB allocated for chunks. 615.23MiB in use in bin. 615.23MiB client-requested in use in bin. 2019-12-23 15:55:03.179905: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 615.23MiB was 256.00MiB, Chunk State: 2019-12-23 15:55:03.183511: I tensorflow/core/common_runtime/bfc_allocator.cc:786] Size: 317.34MiB | Requested Size: 247.61MiB | in_use: 0 | bin_num: 20, prev: Size: 615.23MiB | Requested Size: 615.23MiB | in_use: 1 | bin_num: -1, next: Size: 144.0KiB | Requested Size: 144.0KiB | in_use: 1 | bin_num: -1 2019-12-23 15:55:03.193406: I tensorflow/core/common_runtime/bfc_allocator.cc:786] Size: 512.00MiB | Requested Size: 307.62MiB | in_use: 0 | bin_num: 20 2019-12-23 15:55:03.198517: I tensorflow/core/common_runtime/bfc_allocator.cc:786] Size: 558.68MiB | Requested Size: 307.62MiB | in_use: 0 | bin_num: 20, prev: Size: 144.0KiB | Requested Size: 144.0KiB | in_use: 1 | bin_num: -1 2019-12-23 15:55:03.205800: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1048576 2019-12-23 15:55:03.208822: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703600000 next 1 of size 256 2019-12-23 15:55:03.213355: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703600100 next 2 of size 256 2019-12-23 15:55:03.216876: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703600200 next 3 of size 227328 2019-12-23 15:55:03.220694: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637A00 next 4 of size 256 2019-12-23 15:55:03.224816: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637B00 next 5 of size 256 2019-12-23 15:55:03.228228: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637C00 next 6 of size 256 2019-12-23 15:55:03.232721: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637D00 next 7 of size 256 2019-12-23 15:55:03.236215: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637E00 next 8 of size 256 2019-12-23 15:55:03.240207: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637F00 next 9 of size 256 2019-12-23 15:55:03.243924: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703638000 next 10 of size 18432 2019-12-23 15:55:03.247537: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363C800 next 11 of size 512 2019-12-23 15:55:03.251887: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363CA00 next 12 of size 512 2019-12-23 15:55:03.255408: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363CC00 next 13 of size 512 2019-12-23 15:55:03.259702: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363CE00 next 14 of size 512 2019-12-23 15:55:03.263116: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363D000 next 15 of size 512 2019-12-23 15:55:03.267501: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363D200 next 16 of size 256 2019-12-23 15:55:03.270991: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363D300 next 17 of size 256 2019-12-23 15:55:03.274735: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363D400 next 18 of size 3584 2019-12-23 15:55:03.278855: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363E200 next 19 of size 512 2019-12-23 15:55:03.282935: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363E400 next 20 of size 256 2019-12-23 15:55:03.287390: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363E500 next 21 of size 147456 2019-12-23 15:55:03.291455: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703662500 next 22 of size 256 2019-12-23 15:55:03.295633: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703662600 next 23 of size 256 2019-12-23 15:55:03.299245: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703662700 next 24 of size 256 2019-12-23 15:55:03.303536: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703662800 next 25 of size 6144 2019-12-23 15:55:03.307109: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703664000 next 26 of size 256 2019-12-23 15:55:03.310887: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703664100 next 27 of size 256 2019-12-23 15:55:03.315193: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703664200 next 28 of size 204800 2019-12-23 15:55:03.318711: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703696200 next 29 of size 256 2019-12-23 15:55:03.323267: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703696300 next 30 of size 256 2019-12-23 15:55:03.326718: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703696400 next 31 of size 18944 2019-12-23 15:55:03.331132: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070369AE00 next 32 of size 256 2019-12-23 15:55:03.334594: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070369AF00 next 33 of size 256 2019-12-23 15:55:03.338385: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070369B000 next 34 of size 256 2019-12-23 15:55:03.342615: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070369B100 next 35 of size 147456 2019-12-23 15:55:03.346250: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036BF100 next 36 of size 256 2019-12-23 15:55:03.350587: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036BF200 next 37 of size 256 2019-12-23 15:55:03.354726: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036BF300 next 38 of size 32768 2019-12-23 15:55:03.358855: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7300 next 39 of size 512 2019-12-23 15:55:03.362387: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7500 next 40 of size 512 2019-12-23 15:55:03.365897: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7700 next 41 of size 512 2019-12-23 15:55:03.370651: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7900 next 42 of size 512 2019-12-23 15:55:03.374030: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7B00 next 43 of size 512 2019-12-23 15:55:03.378189: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7D00 next 44 of size 256 2019-12-23 15:55:03.381647: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7E00 next 45 of size 256 2019-12-23 15:55:03.384917: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7F00 next 46 of size 256 2019-12-23 15:55:03.389817: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8000 next 47 of size 256 2019-12-23 15:55:03.393262: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8100 next 48 of size 256 2019-12-23 15:55:03.397372: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8200 next 49 of size 256 2019-12-23 15:55:03.400720: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8300 next 50 of size 256 2019-12-23 15:55:03.404964: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8400 next 51 of size 256 2019-12-23 15:55:03.409168: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8500 next 52 of size 256 2019-12-23 15:55:03.412576: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8600 next 53 of size 256 2019-12-23 15:55:03.417357: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8700 next 54 of size 256 2019-12-23 15:55:03.420692: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8800 next 55 of size 256 2019-12-23 15:55:03.426379: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8900 next 56 of size 256 2019-12-23 15:55:03.429848: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8A00 next 57 of size 256 2019-12-23 15:55:03.435081: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8B00 next 58 of size 256 2019-12-23 15:55:03.438870: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8C00 next 59 of size 512 2019-12-23 15:55:03.443106: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8E00 next 60 of size 512 2019-12-23 15:55:03.446523: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9000 next 61 of size 512 2019-12-23 15:55:03.451070: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9200 next 62 of size 512 2019-12-23 15:55:03.454620: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9400 next 63 of size 512 2019-12-23 15:55:03.458847: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9600 next 64 of size 256 2019-12-23 15:55:03.462621: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9700 next 65 of size 256 2019-12-23 15:55:03.466184: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9800 next 66 of size 256 2019-12-23 15:55:03.470599: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9900 next 67 of size 256 2019-12-23 15:55:03.474181: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9A00 next 68 of size 256 2019-12-23 15:55:03.478221: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9B00 next 69 of size 256 2019-12-23 15:55:03.482325: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9C00 next 70 of size 256 2019-12-23 15:55:03.486454: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9D00 next 71 of size 256 2019-12-23 15:55:03.490066: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9E00 next 72 of size 256 2019-12-23 15:55:03.493746: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9F00 next 73 of size 256 2019-12-23 15:55:03.498162: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036CA000 next 74 of size 1280 2019-12-23 15:55:03.501579: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036CA500 next 75 of size 256 2019-12-23 15:55:03.505785: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036CA600 next 18446744073709551615 of size 219648 2019-12-23 15:55:03.510010: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 2097152 2019-12-23 15:55:03.513823: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703800000 next 77 of size 204800 2019-12-23 15:55:03.517277: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703832000 next 78 of size 227328 2019-12-23 15:55:03.521818: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703869800 next 79 of size 512 2019-12-23 15:55:03.525268: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703869A00 next 80 of size 147456 2019-12-23 15:55:03.529328: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070388DA00 next 81 of size 256 2019-12-23 15:55:03.533531: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070388DB00 next 82 of size 256 2019-12-23 15:55:03.537254: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070388DC00 next 83 of size 512 2019-12-23 15:55:03.541323: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070388DE00 next 84 of size 6144 2019-12-23 15:55:03.544864: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070388F600 next 85 of size 18944 2019-12-23 15:55:03.549147: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703894000 next 86 of size 512 2019-12-23 15:55:03.552947: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703894200 next 87 of size 3584 2019-12-23 15:55:03.556378: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703895000 next 88 of size 512 2019-12-23 15:55:03.561158: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703895200 next 89 of size 256 2019-12-23 15:55:03.564551: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703895300 next 90 of size 18432 2019-12-23 15:55:03.568872: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703899B00 next 91 of size 512 2019-12-23 15:55:03.572483: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703899D00 next 92 of size 512 2019-12-23 15:55:03.576986: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703899F00 next 93 of size 512 2019-12-23 15:55:03.580459: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070389A100 next 94 of size 512 2019-12-23 15:55:03.583930: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070389A300 next 95 of size 32768 2019-12-23 15:55:03.588643: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2300 next 96 of size 512 2019-12-23 15:55:03.592075: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2500 next 97 of size 256 2019-12-23 15:55:03.596119: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2600 next 98 of size 512 2019-12-23 15:55:03.599701: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2800 next 99 of size 512 2019-12-23 15:55:03.602993: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2A00 next 100 of size 256 2019-12-23 15:55:03.607754: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2B00 next 101 of size 512 2019-12-23 15:55:03.611233: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2D00 next 102 of size 256 2019-12-23 15:55:03.615535: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2E00 next 103 of size 256 2019-12-23 15:55:03.619511: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2F00 next 104 of size 256 2019-12-23 15:55:03.623603: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3000 next 105 of size 256 2019-12-23 15:55:03.627092: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3100 next 106 of size 512 2019-12-23 15:55:03.630555: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3300 next 107 of size 256 2019-12-23 15:55:03.634828: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3400 next 108 of size 256 2019-12-23 15:55:03.638848: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3500 next 109 of size 512 2019-12-23 15:55:03.642937: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3700 next 110 of size 512 2019-12-23 15:55:03.646380: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3900 next 111 of size 256 2019-12-23 15:55:03.650475: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3A00 next 112 of size 256 2019-12-23 15:55:03.654036: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3B00 next 113 of size 512 2019-12-23 15:55:03.657806: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3D00 next 114 of size 256 2019-12-23 15:55:03.662041: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3E00 next 115 of size 256 2019-12-23 15:55:03.665886: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3F00 next 116 of size 256 2019-12-23 15:55:03.670073: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4000 next 117 of size 512 2019-12-23 15:55:03.673576: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4200 next 118 of size 256 2019-12-23 15:55:03.676944: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4300 next 119 of size 256 2019-12-23 15:55:03.681189: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4400 next 120 of size 512 2019-12-23 15:55:03.684591: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4600 next 121 of size 512 2019-12-23 15:55:03.688935: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4800 next 122 of size 256 2019-12-23 15:55:03.692329: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4900 next 123 of size 256 2019-12-23 15:55:03.695737: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4A00 next 124 of size 512 2019-12-23 15:55:03.700272: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4C00 next 125 of size 512 2019-12-23 15:55:03.703679: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4E00 next 126 of size 256 2019-12-23 15:55:03.707751: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4F00 next 127 of size 256 2019-12-23 15:55:03.711064: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5000 next 128 of size 256 2019-12-23 15:55:03.715780: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5100 next 129 of size 256 2019-12-23 15:55:03.719225: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5200 next 130 of size 512 2019-12-23 15:55:03.722646: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5400 next 131 of size 256 2019-12-23 15:55:03.726832: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5500 next 132 of size 256 2019-12-23 15:55:03.730208: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5600 next 133 of size 256 2019-12-23 15:55:03.734647: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5700 next 134 of size 512 2019-12-23 15:55:03.738104: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5900 next 135 of size 256 2019-12-23 15:55:03.742371: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5A00 next 136 of size 256 2019-12-23 15:55:03.746156: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5B00 next 137 of size 256 2019-12-23 15:55:03.749717: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5C00 next 138 of size 256 2019-12-23 15:55:03.753928: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5D00 next 139 of size 256 2019-12-23 15:55:03.757406: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5E00 next 140 of size 512 2019-12-23 15:55:03.761853: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6000 next 141 of size 256 2019-12-23 15:55:03.765341: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6100 next 142 of size 256 2019-12-23 15:55:03.769651: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6200 next 143 of size 256 2019-12-23 15:55:03.773187: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6300 next 144 of size 256 2019-12-23 15:55:03.776629: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6400 next 145 of size 256 2019-12-23 15:55:03.781382: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6500 next 146 of size 256 2019-12-23 15:55:03.784928: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6600 next 147 of size 256 2019-12-23 15:55:03.789009: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6700 next 148 of size 256 2019-12-23 15:55:03.793127: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6800 next 149 of size 256 2019-12-23 15:55:03.797284: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6900 next 150 of size 256 2019-12-23 15:55:03.800799: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6A00 next 151 of size 256 2019-12-23 15:55:03.804969: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6B00 next 152 of size 256 2019-12-23 15:55:03.808447: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6C00 next 153 of size 256 2019-12-23 15:55:03.812213: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6D00 next 154 of size 256 2019-12-23 15:55:03.816385: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6E00 next 155 of size 256 2019-12-23 15:55:03.819981: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6F00 next 156 of size 256 2019-12-23 15:55:03.824627: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7000 next 157 of size 256 2019-12-23 15:55:03.828110: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7100 next 158 of size 256 2019-12-23 15:55:03.832317: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7200 next 159 of size 256 2019-12-23 15:55:03.835825: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7300 next 166 of size 256 2019-12-23 15:55:03.839712: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7400 next 177 of size 256 2019-12-23 15:55:03.843962: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7500 next 172 of size 256 2019-12-23 15:55:03.847500: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7600 next 185 of size 256 2019-12-23 15:55:03.851658: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7700 next 174 of size 256 2019-12-23 15:55:03.855511: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7800 next 161 of size 256 2019-12-23 15:55:03.859561: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7900 next 216 of size 256 2019-12-23 15:55:03.863041: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7A00 next 210 of size 256 2019-12-23 15:55:03.866516: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7B00 next 228 of size 256 2019-12-23 15:55:03.871179: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7C00 next 232 of size 256 2019-12-23 15:55:03.874636: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7D00 next 233 of size 256 2019-12-23 15:55:03.878729: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7E00 next 226 of size 256 2019-12-23 15:55:03.882381: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7F00 next 229 of size 256 2019-12-23 15:55:03.886829: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8000 next 230 of size 256 2019-12-23 15:55:03.890253: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8100 next 214 of size 512 2019-12-23 15:55:03.893694: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8300 next 163 of size 512 2019-12-23 15:55:03.897987: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8500 next 164 of size 256 2019-12-23 15:55:03.901467: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8600 next 165 of size 256 2019-12-23 15:55:03.905525: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8700 next 181 of size 256 2019-12-23 15:55:03.909268: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8800 next 179 of size 256 2019-12-23 15:55:03.913465: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8900 next 189 of size 256 2019-12-23 15:55:03.917181: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8A00 next 223 of size 256 2019-12-23 15:55:03.920685: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8B00 next 220 of size 256 2019-12-23 15:55:03.925049: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8C00 next 224 of size 256 2019-12-23 15:55:03.928529: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8D00 next 195 of size 256 2019-12-23 15:55:03.932797: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8E00 next 257 of size 256 2019-12-23 15:55:03.936302: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8F00 next 171 of size 256 2019-12-23 15:55:03.940868: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 00000007038A9000 next 266 of size 256 2019-12-23 15:55:03.944663: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A9100 next 175 of size 256 2019-12-23 15:55:03.947922: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A9200 next 264 of size 256 2019-12-23 15:55:03.952551: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 00000007038A9300 next 208 of size 7936 2019-12-23 15:55:03.956016: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038AB200 next 215 of size 6144 2019-12-23 15:55:03.960264: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038ACA00 next 231 of size 6144 2019-12-23 15:55:03.963889: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038AE200 next 209 of size 512 2019-12-23 15:55:03.968287: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038AE400 next 218 of size 3584 2019-12-23 15:55:03.971737: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038AF200 next 217 of size 3584 2019-12-23 15:55:03.975156: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 00000007038B0000 next 212 of size 18432 2019-12-23 15:55:03.979327: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038B4800 next 206 of size 18432 2019-12-23 15:55:03.982811: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038B9000 next 213 of size 18432 2019-12-23 15:55:03.987578: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 00000007038BD800 next 211 of size 32768 2019-12-23 15:55:03.991109: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038C5800 next 198 of size 32768 2019-12-23 15:55:03.995759: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038CD800 next 204 of size 32768 2019-12-23 15:55:03.999416: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038D5800 next 168 of size 18944 2019-12-23 15:55:04.002934: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038DA200 next 205 of size 30208 2019-12-23 15:55:04.007278: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038E1800 next 201 of size 147456 2019-12-23 15:55:04.010826: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703905800 next 199 of size 147456 2019-12-23 15:55:04.015683: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0000000703929800 next 221 of size 213504 2019-12-23 15:55:04.019174: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070395DA00 next 227 of size 196608 2019-12-23 15:55:04.023370: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070398DA00 next 225 of size 227328 2019-12-23 15:55:04.026986: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007039C5200 next 18446744073709551615 of size 241152 2019-12-23 15:55:04.030930: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 8388608 2019-12-23 15:55:04.035140: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703A00000 next 176 of size 204800 2019-12-23 15:55:04.038806: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703A32000 next 178 of size 204800 2019-12-23 15:55:04.043366: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0000000703A64000 next 18446744073709551615 of size 7979008 2019-12-23 15:55:04.047441: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 8388608 2019-12-23 15:55:04.051363: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0000000704200000 next 18446744073709551615 of size 8388608 2019-12-23 15:55:04.055332: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 134217728 2019-12-23 15:55:04.058469: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0000000704A00000 next 18446744073709551615 of size 134217728 2019-12-23 15:55:04.063751: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 536870912 2019-12-23 15:55:04.066880: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 000000070EE00000 next 18446744073709551615 of size 536870912 2019-12-23 15:55:04.072355: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1563846144 2019-12-23 15:55:04.075439: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000072EE00000 next 246 of size 645120000 2019-12-23 15:55:04.079768: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 000000075553C000 next 219 of size 332759040 2019-12-23 15:55:04.083344: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000769294000 next 222 of size 147456 2019-12-23 15:55:04.088033: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 00000007692B8000 next 18446744073709551615 of size 585819648 2019-12-23 15:55:04.092059: I tensorflow/core/common_runtime/bfc_allocator.cc:809] Summary of in-use Chunks by size: 2019-12-23 15:55:04.096021: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 127 Chunks of size 256 totalling 31.8KiB 2019-12-23 15:55:04.099440: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 43 Chunks of size 512 totalling 21.5KiB 2019-12-23 15:55:04.102860: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.3KiB 2019-12-23 15:55:04.107029: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 3584 totalling 14.0KiB 2019-12-23 15:55:04.110885: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 6144 totalling 24.0KiB 2019-12-23 15:55:04.114834: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 18432 totalling 72.0KiB 2019-12-23 15:55:04.118602: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 18944 totalling 55.5KiB 2019-12-23 15:55:04.122083: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 30208 totalling 29.5KiB 2019-12-23 15:55:04.126229: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 32768 totalling 128.0KiB 2019-12-23 15:55:04.129866: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 6 Chunks of size 147456 totalling 864.0KiB 2019-12-23 15:55:04.134333: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 196608 totalling 192.0KiB 2019-12-23 15:55:04.138252: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 204800 totalling 800.0KiB 2019-12-23 15:55:04.142353: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 219648 totalling 214.5KiB 2019-12-23 15:55:04.145692: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 227328 totalling 666.0KiB 2019-12-23 15:55:04.149094: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 241152 totalling 235.5KiB 2019-12-23 15:55:04.153582: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 645120000 totalling 615.23MiB 2019-12-23 15:55:04.157423: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 618.50MiB 2019-12-23 15:55:04.161201: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 2254857728 memory_limit_: 2254857830 available bytes: 102 curr_region_allocation_bytes_: 2147483648 2019-12-23 15:55:04.169402: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats: Limit: 2254857830 InUse: 648549888 MaxInUse: 2040417280 NumAllocs: 133567 MaxAllocSize: 1083703296
2019-12-23 15:55:04.179973: W tensorflow/core/common_runtime/bfc_allocator.cc:319] _____________________________****************************_______________________________________ 2019-12-23 15:55:04.185661: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at transpose_op.cc:199 : Resource exhausted: OOM when allocating tensor with shape[300,140,60,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call return fn(*args) File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found. (0) Resource exhausted: OOM when allocating tensor with shape[300,140,60,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node CNNX/batch_normalization_1/FusedBatchNorm-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[dense_decoded/_281]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[300,140,60,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node CNNX/batch_normalization_1/FusedBatchNorm-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations. 0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "app.py", line 1403, in training_task self.current_task.train_process() File "C:\Users\super\Desktop\captcha_trainer-master\trains.py", line 196, in train_process feed_dict=val_feed File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 950, in run run_metadata_ptr) File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run feed_dict_tensor, options, run_metadata) File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run run_metadata) File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found. (0) Resource exhausted: OOM when allocating tensor with shape[300,140,60,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node CNNX/batch_normalization_1/FusedBatchNorm-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[dense_decoded/_281]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[300,140,60,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node CNNX/batch_normalization_1/FusedBatchNorm-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations. 0 derived errors ignored.
