sleap
sleap copied to clipboard
talmolab
Command Line Error During Inference: `OOM`
I have a trained model and have been running inference on untrained videos. I've been doing this for a couple of days now, but today I ran into an error. I have never encountered this before and have done nothing different in comparison to the other videos I've use the model on.
Hi @ericas12,
Request for more info
Are you able to provide a few more lines from the terminal (ideally scrolling up until you see "Traceback")? Is there anything at all different with these new videos?
In addition to more lines from the terminal, some things that will help us are:
- The size of the video.
- The training_config.json for the model(s) you are having trouble with (there is also a "Copy to Clipboard" button in the GUI).
Initial guess
Going off of what we can see in the terminal - there appears to be a concatenate issue (which usually occurs when there is some size discrepancy, i.e. downsampling resolution then trying to upsample to the same size, but encountering rounding errors along the way).
Thanks, Liezl
Hi,
Yes, There is nothing different with the videos compared to what I have been doing. Each video is about 3 hours long and 600x800 and I have been running inference on 8 videos at a time. Below, I have copied and pasted the training_config.json. I've already run inference on about 3 sets of 8 videos with this model. I will also add another comment with the whole error, as it is pretty long. Thank you!
data details
{
"_pipeline": "single animal",
"tracking.tracker": "none",
"_predict_frames": "all videos (2597488 frames)"
}
{
"data": {
"labels": {
"training_labels": "Z:/Golden_Lab_Users/MacMillen_Lukas/SLEAP/Models/ofsa.c57.slp",
"validation_labels": null,
"validation_fraction": 0.1,
"test_labels": null,
"split_by_inds": false,
"training_inds": [
1296,
839,
492,
1450,
876,
653,
125,
792,
287,
385,
1234,
1059,
173,
1150,
136,
566,
485,
1173,
567,
404,
48,
1254,
643,
205,
1136,
619,
503,
353,
1145,
1318,
709,
1213,
737,
534,
27,
1250,
753,
441,
987,
502,
1418,
69,
724,
1554,
117,
144,
1274,
786,
1065,
1327,
24,
826,
672,
325,
590,
692,
679,
1172,
132,
694,
45,
1512,
1278,
1137,
731,
1325,
1216,
767,
1443,
305,
1080,
59,
347,
665,
690,
585,
838,
554,
609,
223,
411,
780,
842,
505,
540,
1310,
263,
568,
603,
311,
1106,
1181,
256,
424,
973,
226,
249,
168,
1111,
1188,
268,
863,
1162,
156,
1076,
1003,
145,
982,
327,
1236,
1467,
1040,
878,
285,
1559,
545,
1366,
330,
810,
951,
895,
1275,
957,
66,
1449,
275,
185,
1171,
1014,
218,
239,
570,
1456,
1074,
914,
1537,
326,
55,
22,
513,
1303,
1155,
923,
35,
277,
457,
564,
592,
5,
801,
74,
1405,
1170,
1007,
1500,
1100,
667,
1519,
1428,
866,
733,
1516,
549,
270,
1116,
1022,
632,
894,
429,
38,
203,
338,
309,
677,
1132,
129,
966,
776,
1396,
1415,
1046,
1490,
1317,
1005,
1099,
146,
697,
668,
1117,
532,
602,
490,
1475,
201,
607,
220,
686,
257,
161,
544,
1252,
409,
828,
812,
255,
597,
1527,
1050,
699,
999,
726,
18,
645,
232,
514,
1177,
340,
1161,
649,
1419,
137,
817,
64,
1534,
1102,
1224,
1024,
915,
1233,
791,
174,
1306,
113,
639,
1066,
1121,
1112,
292,
896,
1029,
99,
972,
1197,
681,
684,
1346,
375,
191,
288,
1521,
755,
738,
497,
400,
930,
1532,
1051,
297,
994,
574,
1012,
669,
642,
1558,
54,
1149,
407,
1225,
178,
0,
236,
342,
95,
440,
63,
1120,
1472,
760,
997,
251,
1395,
865,
254,
465,
58,
419,
1387,
822,
80,
903,
1198,
888,
280,
1567,
875,
331,
428,
814,
484,
1211,
1130,
1479,
1312,
196,
840,
360,
1093,
1202,
78,
336,
898,
1251,
119,
710,
1355,
6,
112,
867,
37,
1370,
712,
84,
1518,
528,
1322,
571,
339,
1031,
1469,
1052,
380,
849,
1413,
1126,
354,
439,
1082,
527,
472,
1058,
689,
1506,
708,
1439,
851,
53,
1455,
1494,
176,
224,
216,
1340,
1364,
893,
805,
777,
1442,
329,
454,
1503,
1423,
104,
629,
231,
612,
1259,
635,
1124,
744,
1309,
638,
1468,
599,
530,
1463,
100,
655,
906,
167,
493,
427,
221,
368,
1435,
1566,
431,
461,
1077,
897,
227,
714,
1123,
259,
680,
670,
184,
93,
365,
630,
480,
1339,
577,
1097,
961,
1271,
804,
1037,
841,
1481,
1372,
911,
42,
469,
759,
171,
276,
1056,
169,
495,
1187,
1017,
616,
1244,
783,
1010,
542,
1381,
206,
383,
1349,
1430,
611,
1163,
722,
1242,
26,
877,
1360,
1071,
870,
509,
1426,
1207,
1485,
253,
561,
61,
587,
154,
719,
823,
1086,
438,
1204,
1561,
1536,
60,
1182,
720,
991,
262,
912,
864,
1304,
640,
589,
1549,
350,
1184,
696,
1412,
1384,
705,
1510,
933,
1165,
313,
1389,
1272,
94,
552,
193,
1255,
1345,
827,
1035,
229,
1073,
194,
931,
413,
1144,
373,
1115,
170,
225,
908,
621,
1036,
228,
210,
314,
913,
1324,
1293,
872,
34,
160,
1089,
774,
1001,
1464,
797,
651,
471,
910,
1540,
918,
323,
874,
707,
1075,
1021,
716,
334,
711,
905,
133,
1196,
723,
1260,
1295,
1571,
499,
1548,
788,
763,
1488,
1569,
1033,
1201,
516,
605,
1507,
691,
1582,
324,
1218,
1454,
660,
258,
1167,
372,
1446,
1302,
1465,
1133,
1088,
1122,
126,
1331,
272,
924,
1574,
110,
195,
444,
271,
267,
1160,
945,
496,
1575,
1141,
1287,
1368,
390,
900,
1015,
1330,
1006,
675,
1158,
869,
377,
892,
312,
1484,
52,
32,
907,
293,
180,
940,
962,
1497,
1061,
387,
107,
1153,
1000,
920,
1057,
909,
1363,
978,
1248,
1265,
264,
374,
425,
464,
1432,
186,
790,
67,
1544,
157,
364,
563,
83,
758,
578,
1502,
348,
402,
278,
155,
1487,
1042,
806,
1135,
346,
796,
593,
584,
1451,
433,
1385,
445,
565,
1520,
1560,
1557,
1337,
1515,
72,
416,
141,
28,
856,
662,
1276,
970,
443,
1288,
14,
1394,
1263,
902,
1190,
1329,
1192,
904,
436,
122,
341,
8,
1404,
1489,
399,
420,
219,
698,
803,
1299,
1453,
1286,
367,
613,
1539,
398,
1139,
1393,
591,
627,
1016,
362,
1511,
357,
750,
211,
1019,
1378,
988,
541,
198,
1377,
990,
740,
40,
862,
29,
494,
351,
120,
301,
1326,
1072,
772,
953,
426,
644,
1128,
1429,
562,
470,
1148,
15,
537,
553,
1348,
1009,
683,
1307,
1461,
1266,
620,
539,
189,
887,
70,
1383,
1129,
1320,
282,
960,
727,
147,
1578,
926,
281,
881,
885,
768,
1257,
430,
950,
1193,
1277,
476,
1249,
925,
543,
1166,
332,
274,
1356,
286,
179,
246,
942,
322,
929,
269,
1459,
1471,
1222,
884,
891,
162,
417,
721,
252,
1531,
506,
307,
103,
479,
1146,
379,
1113,
761,
62,
1164,
279,
500,
781,
601,
1421,
483,
1118,
520,
1215,
335,
1460,
1433,
204,
917,
883,
641,
1069,
1379,
1032,
382,
240,
967,
868,
98,
345,
1189,
1157,
1025,
1294,
1301,
824,
661,
600,
969,
586,
421,
757,
1498,
1547,
617,
489,
1350,
536,
56,
378,
1168,
778,
852,
702,
151,
106,
728,
802,
361,
510,
779,
648,
766,
234,
981,
1517,
993,
944,
370,
295,
1584,
475,
434,
623,
989,
50,
1499,
1199,
952,
700,
657,
821,
830,
800,
654,
1154,
785,
688,
836,
1282,
1268,
1114,
1176,
328,
560,
1210,
473,
975,
1543,
344,
1424,
289,
415,
124,
248,
355,
1305,
111,
79,
77,
4,
13,
1361,
504,
1143,
90,
992,
1445,
1273,
1491,
396,
1427,
1185,
770,
65,
647,
771,
995,
1380,
848,
1092,
152,
1466,
108,
573,
82,
1529,
1267,
1238,
1228,
572,
1214,
487,
882,
1270,
1425,
1420,
359,
1476,
16,
114,
187,
1493,
1062,
1078,
963,
976,
1365,
1219,
1386,
142,
109,
521,
81,
1344,
304,
535,
1140,
1492,
212,
815,
1556,
1359,
192,
1390,
550,
1505,
1018,
401,
725,
673,
1291,
847,
395,
1109,
452,
782,
1300,
588,
7,
729,
1041,
128,
1343,
658,
704,
519,
734,
829,
1180,
555,
164,
663,
507,
1194,
559,
181,
834,
1256,
1290,
1055,
959,
230,
1573,
1229,
919,
1530,
352,
546,
1321,
736,
580,
1101,
460,
1486,
1119,
526,
1376,
1583,
551,
1053,
202,
1174,
308,
265,
1289,
414,
491,
391,
1374,
1473,
1028,
854,
208,
825,
596,
921,
478,
303,
1105,
656,
49,
1441,
17,
130,
941,
1319,
1580,
310,
1280,
873,
859,
682,
799,
393,
1261,
1341,
149,
1258,
450,
846,
183,
296,
860,
432,
498,
165,
1240,
1285,
182,
1399,
850,
1555,
23,
687,
937,
1474,
1323,
118,
604,
97,
948,
1480,
1043,
369,
437,
349,
474,
637,
703,
1094,
333,
199,
899,
1279,
949,
713,
739,
1096,
318,
131,
86,
1514,
818,
1357,
889,
1417,
177,
458,
1528,
1110,
515,
715,
1156,
241,
143,
606,
798,
1220,
748,
190,
1011,
321,
153,
1147,
298,
1186,
290,
1391,
1104,
886,
1049,
300,
569,
1131,
685,
134,
871,
1203,
463,
795,
1437,
1333,
1060,
1103,
787,
1552,
115,
1087,
283,
749,
36,
1342,
73,
1362,
356,
807,
366,
1347,
968,
769,
833,
809,
116,
316,
837,
751,
1079,
1245,
1440,
10,
1416,
598,
317,
816,
618,
1209,
172,
558,
1535,
1436,
583,
936,
1253,
213,
1572,
1410,
1448,
741,
91,
320,
92,
1237,
955,
916,
789,
20,
1358,
820,
1352,
1281,
449,
775,
68,
695,
1382,
512,
1227,
1283,
575,
386,
46,
1020,
525,
76,
1297,
1091,
1509,
418,
678,
1217,
1458,
956,
608,
1085,
1048,
946,
1562,
1314,
927,
1178,
1013,
853,
1030,
1351,
624,
266,
1504,
636,
2,
462,
315,
693,
150,
832,
518,
743,
1570,
626,
524,
808,
1579,
1241,
1367,
857,
965,
845,
879,
1269,
547,
1235,
1044,
747,
247,
260,
302,
1054,
466,
88,
242,
943,
1083,
706,
358,
1264,
880,
1125,
998,
1027,
245,
1545,
890,
1246,
1513,
1221,
1462,
1546,
650,
244,
938,
1239,
819,
1434,
1142,
442,
222,
730,
622,
980,
166,
71,
628,
123,
811,
1151,
784,
633,
422,
1336,
1332,
1411,
1081,
1284,
1550,
1414,
1354,
25,
1470,
1542,
453,
1090,
522,
1179,
1068,
701,
813,
634,
1,
756,
1577,
51,
1533,
1523,
594,
958,
1495,
831,
1328,
1152,
1477,
652,
666,
1095,
101,
1232,
793,
405,
217,
1067,
1369,
138,
12,
581,
1159,
1127,
1034,
964,
1524,
148,
765,
718,
732,
1483,
1038,
451,
376,
1070,
403,
1023,
284,
1444,
1212,
273,
1026,
1522,
1568,
934,
237,
371,
625,
468,
1226,
977,
319,
676,
486,
158,
1576,
1262,
43,
456,
19,
1452,
1501,
75,
1422,
397,
30,
531,
33,
1045,
576,
235,
855,
735,
1581,
135,
745,
467,
1400,
983,
517,
291,
299,
87,
159,
742,
102,
294,
674,
3,
381,
388,
579,
1039,
89,
9,
447,
631,
1388,
139,
47,
1408,
974,
1508,
448,
384
],
"validation_inds": [
1403,
1375,
501,
337,
363,
971,
1064,
671,
752,
1311,
954,
215,
773,
529,
1565,
175,
764,
392,
121,
582,
214,
261,
844,
1004,
533,
243,
435,
523,
406,
939,
979,
1526,
1447,
1175,
984,
39,
947,
250,
200,
488,
664,
928,
1407,
343,
1205,
459,
1308,
1525,
207,
1138,
717,
1338,
1401,
901,
408,
1230,
614,
659,
1191,
21,
861,
996,
986,
446,
794,
105,
1169,
127,
1392,
96,
1206,
1134,
482,
1107,
511,
754,
1496,
44,
306,
610,
858,
1406,
1398,
57,
163,
1431,
1208,
556,
595,
394,
1223,
11,
1047,
1315,
1563,
1008,
1292,
1564,
1538,
1397,
1478,
843,
835,
1183,
932,
508,
1298,
1231,
1409,
1108,
646,
477,
1200,
548,
615,
922,
410,
1084,
41,
1482,
1371,
1247,
1553,
423,
746,
1402,
1316,
188,
1243,
481,
1551,
1373,
1098,
762,
1334,
985,
238,
1002,
1353,
455,
1541,
1438,
935,
557,
140,
389,
412,
209,
85,
1313,
538,
233,
31,
1457,
1195,
197,
1335,
1063
],
"test_inds": null,
"search_path_hints": [
"",
"",
""
],
"skeletons": [
]
},
"preprocessing": {
"ensure_rgb": false,
"ensure_grayscale": false,
"imagenet_mode": null,
"input_scaling": 1.0,
"pad_to_stride": 16,
"resize_and_pad_to_target": true,
"target_height": 600,
"target_width": 800
},
"instance_cropping": {
"center_on_part": null,
"crop_size": null,
"crop_size_detection_padding": 16
}
},
"model": {
"backbone": {
"leap": null,
"unet": {
"stem_stride": null,
"max_stride": 16,
"output_stride": 2,
"filters": 16,
"filters_rate": 2.0,
"middle_block": true,
"up_interpolate": true,
"stacks": 1
},
"hourglass": null,
"resnet": null,
"pretrained_encoder": null
},
"heads": {
"single_instance": {
"part_names": [
"nose",
"left_ear",
"right_ear",
"neck",
"center",
"left_flank",
"right_flank",
"tail_base",
"tail_center",
"tail_tip"
],
"sigma": 8.0,
"output_stride": 2,
"loss_weight": 1.0,
"offset_refinement": false
},
"centroid": null,
"centered_instance": null,
"multi_instance": null,
"multi_class_bottomup": null,
"multi_class_topdown": null
},
"base_checkpoint": null
},
"optimization": {
"preload_data": true,
"augmentation_config": {
"rotate": true,
"rotation_min_angle": -15.0,
"rotation_max_angle": 15.0,
"translate": false,
"translate_min": -5,
"translate_max": 5,
"scale": false,
"scale_min": 0.9,
"scale_max": 1.1,
"uniform_noise": false,
"uniform_noise_min_val": 0.0,
"uniform_noise_max_val": 10.0,
"gaussian_noise": false,
"gaussian_noise_mean": 5.0,
"gaussian_noise_stddev": 1.0,
"contrast": false,
"contrast_min_gamma": 0.5,
"contrast_max_gamma": 2.0,
"brightness": false,
"brightness_min_val": 0.0,
"brightness_max_val": 10.0,
"random_crop": false,
"random_crop_height": 256,
"random_crop_width": 256,
"random_flip": false,
"flip_horizontal": false
},
"online_shuffling": true,
"shuffle_buffer_size": 128,
"prefetch": true,
"batch_size": 4,
"batches_per_epoch": 352,
"min_batches_per_epoch": 200,
"val_batches_per_epoch": 39,
"min_val_batches_per_epoch": 10,
"epochs": 200,
"optimizer": "adam",
"initial_learning_rate": 0.0001,
"learning_rate_schedule": {
"reduce_on_plateau": true,
"reduction_factor": 0.5,
"plateau_min_delta": 1e-06,
"plateau_patience": 5,
"plateau_cooldown": 3,
"min_learning_rate": 1e-08
},
"hard_keypoint_mining": {
"online_mining": false,
"hard_to_easy_ratio": 2.0,
"min_hard_keypoints": 2,
"max_hard_keypoints": null,
"loss_scale": 5.0
},
"early_stopping": {
"stop_training_on_plateau": true,
"plateau_min_delta": 1e-08,
"plateau_patience": 10
}
},
"outputs": {
"save_outputs": true,
"run_name": "250310_135911.single_instance.n=1585",
"run_name_prefix": "ofsa_c57_v3.1_s8_",
"run_name_suffix": "",
"runs_folder": "Z:/Golden_Lab_Users/MacMillen_Lukas/SLEAP/Models\\models",
"tags": [
""
],
"save_visualizations": true,
"delete_viz_images": true,
"zip_outputs": false,
"log_to_csv": true,
"checkpointing": {
"initial_model": false,
"best_model": true,
"every_epoch": false,
"latest_model": false,
"final_model": false
},
"tensorboard": {
"write_logs": false,
"loss_frequency": "epoch",
"architecture_graph": false,
"profile_graph": false,
"visualizations": true
},
"zmq": {
"subscribe_to_controller": true,
"controller_address": "tcp://127.0.0.1:9000",
"controller_polling_timeout": 10,
"publish_updates": true,
"publish_address": "tcp://127.0.0.1:9001"
}
},
"name": "",
"description": "",
"sleap_version": "1.3.3",
"filename": "Z:/Golden_Lab_Users/MacMillen_Lukas/SLEAP/Models\\models\\ofsa_c57_v3.1_s8_250310_135911.single_instance.n=1585\\training_config.json"
}
Command Line Full Error
Prior sleap-label (multiple skeletons?)
Happy SLEAPing! :)
Restoring GUI state...
Saving config: C:\Users\Nape_Computer_2/.sleap/1.3.3/preferences.yaml
Traceback (most recent call last):
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\gui\app.py", line 886, in <lambda>
lambda: self._show_learning_dialog("inference"),
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\gui\app.py", line 1490, in _show_learning_dialog
self._child_windows[mode].skeleton = self.labels.skeleton
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\io\dataset.py", line 564, in skeleton
"Labels.skeleton can only be used when there is only a single skeleton "
ValueError: Labels.skeleton can only be used when there is only a single skeleton saved in the labels. Use Labels.skeletons instead.
Saving config: C:\Users\Nape_Computer_2/.sleap/1.3.3/preferences.yaml
Run inference printout
(sleap) C:\Users\Nape_Computer_2>sleap-label
Saving config: C:\Users\Nape_Computer_2/.sleap/1.3.3/preferences.yaml
Restoring GUI state...
Software versions:
SLEAP: 1.3.3
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Windows-10-10.0.19041-SP0
Happy SLEAPing! :)
Using already trained model for single_instance: Z:/Golden_Lab_Users/MacMillen_Lukas/SLEAP/Models\models\ofsa_c57_v3.1_s8_250310_135911.single_instance.n=1585\training_config.json
Command line call:
sleap-track Z:/Golden_Lab_Users/MacMillen_Lukas/SLEAP/Models/v3_C57_07_04_inference_data.v001.slp --video.index 0 --frames 0,-324716 -m Z:/Golden_Lab_Users/MacMillen_Lukas/SLEAP/Models\models\ofsa_c57_v3.1_s8_250310_135911.single_instance.n=1585\training_config.json --tracking.tracker none -o Z:/Golden_Lab_Users/MacMillen_Lukas/SLEAP/Models\predictions\v3_C57_07_04_inference_data.v001.slp.250317_174756.predictions.slp --verbosity json --no-empty-frames
Started inference at: 2025-03-17 17:48:00.930449
Args:
{
'data_path': 'Z:/Golden_Lab_Users/MacMillen_Lukas/SLEAP/Models/v3_C57_07_04_inference_data.v001.slp',
'models': [
'Z:/Golden_Lab_Users/MacMillen_Lukas/SLEAP/Models\\models\\ofsa_c57_v3.1_s8_250310_135911.single_instance.n=1585\\training_config.json'
],
'frames': '0,-324716',
'only_labeled_frames': False,
'only_suggested_frames': False,
'output': 'Z:/Golden_Lab_Users/MacMillen_Lukas/SLEAP/Models\\predictions\\v3_C57_07_04_inference_data.v001.slp.250317_174756.predictions.slp',
'no_empty_frames': True,
'verbosity': 'json',
'video.dataset': None,
'video.input_format': 'channels_last',
'video.index': '0',
'cpu': False,
'first_gpu': False,
'last_gpu': False,
'gpu': 'auto',
'max_edge_length_ratio': 0.25,
'dist_penalty_weight': 1.0,
'batch_size': 4,
'open_in_gui': False,
'peak_threshold': 0.2,
'max_instances': None,
'tracking.tracker': 'none',
'tracking.max_tracking': None,
'tracking.max_tracks': None,
'tracking.target_instance_count': None,
'tracking.pre_cull_to_target': None,
'tracking.pre_cull_iou_threshold': None,
'tracking.post_connect_single_breaks': None,
2025-03-17 17:48:02.678537: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
'tracking.clean_instance_count': None,
'tracking.clean_iou_threshold': None,
'tracking.similarity': None,
'tracking.match': None,
'tracking.robust': None,
'tracking.track_window': None,
'tracking.min_new_track_points': None,
'tracking.min_match_points': None,
'tracking.img_scale': None,
2025-03-17 17:48:03.154850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8962 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:02:00.0, compute capability: 7.5
'tracking.of_window_size': None,
'tracking.of_max_levels': None,
'tracking.save_shifted_instances': None,
'tracking.kf_node_indices': None,
'tracking.kf_init_frame_count': None
}
CUDA_ERROR_OUT_OF_MEMORY
INFO:sleap.nn.inference:Auto-selected GPU 0 with 7646 MiB of free memory.
2025-03-17 17:48:08.459043: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
2025-03-17 17:48:09.817328: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.824270: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 3.60G (3865470464 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.830345: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 3.24G (3478923264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.836352: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 2.92G (3131030784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.842345: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 2.62G (2817927680 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.848336: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 2.36G (2536134912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.854267: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 2.12G (2282521344 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.854399: W tensorflow/core/common_runtime/bfc_allocator.cc:343] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
2025-03-17 17:48:09.935580: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.935747: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.10GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-03-17 17:48:09.987532: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.987695: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.10GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-03-17 17:48:10.050382: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.050546: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 283.19MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-03-17 17:48:10.103461: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.103636: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 283.19MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-03-17 17:48:10.169747: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.169911: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 817.63MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-03-17 17:48:10.221803: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.221964: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 817.63MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-03-17 17:48:10.288597: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.288769: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 550.38MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-03-17 17:48:10.340730: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.340895: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 550.38MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-03-17 17:48:10.409433: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.409595: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-03-17 17:48:10.461451: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.461640: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2025-03-17 17:48:10.535375: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.586566: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.646931: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.698324: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.759694: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.810776: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.862537: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.913208: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:10.977120: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.028204: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.100538: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.152149: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.212404: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.263787: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.352662: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.407110: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.473669: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.527415: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.598891: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.650401: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.747654: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.798977: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.853408: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.904444: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:11.978279: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:12.028309: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:12.090134: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:12.144583: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:12.196171: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:12.248021: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:12.300703: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:12.351966: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:22.406915: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:22.463227: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:22.463403: W tensorflow/core/common_runtime/bfc_allocator.cc:462] Allocator (GPU_0_bfc) ran out of memory trying to allocate 89.06MiB (rounded to 93388800)requested by op single_instance_inference_model/single_instance_inference_layer/model/stack0_dec1_s8_to_s4_skip_concat/concat
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Allocation summary
Current allocation summary follows.
Current allocation summary follows.
2025-03-17 17:48:22.463992: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] BFCAllocator dump for GPU_0_bfc
2025-03-17 17:48:22.464339: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (256): Total Chunks: 53, Chunks in use: 52. 13.2KiB allocated for chunks. 13.0KiB in use in bin. 1.9KiB client-requested in use in bin.
2025-03-17 17:48:22.464629: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (512): Total Chunks: 5, Chunks in use: 5. 2.8KiB allocated for chunks. 2.8KiB in use in bin. 2.6KiB client-requested in use in bin.
2025-03-17 17:48:22.464958: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1024): Total Chunks: 4, Chunks in use: 4. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 4.3KiB client-requested in use in bin.
2025-03-17 17:48:22.465268: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2048): Total Chunks: 2, Chunks in use: 0. 5.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-03-17 17:48:22.465584: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-03-17 17:48:22.465882: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8192): Total Chunks: 1, Chunks in use: 1. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 9.0KiB client-requested in use in bin.
2025-03-17 17:48:22.466191: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16384): Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin.
2025-03-17 17:48:22.466514: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (32768): Total Chunks: 3, Chunks in use: 2. 108.0KiB allocated for chunks. 72.0KiB in use in bin. 72.0KiB client-requested in use in bin.
2025-03-17 17:48:22.466812: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (65536): Total Chunks: 3, Chunks in use: 2. 316.2KiB allocated for chunks. 234.0KiB in use in bin. 180.0KiB client-requested in use in bin.
2025-03-17 17:48:22.467099: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (131072): Total Chunks: 5, Chunks in use: 2. 756.2KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2025-03-17 17:48:22.467366: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (262144): Total Chunks: 4, Chunks in use: 3. 1.44MiB allocated for chunks. 1.16MiB in use in bin. 1.16MiB client-requested in use in bin.
2025-03-17 17:48:22.467666: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (524288): Total Chunks: 3, Chunks in use: 2. 2.11MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2025-03-17 17:48:22.467927: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1048576): Total Chunks: 3, Chunks in use: 1. 3.71MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2025-03-17 17:48:22.468209: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2097152): Total Chunks: 2, Chunks in use: 2. 4.28MiB allocated for chunks. 4.28MiB in use in bin. 3.94MiB client-requested in use in bin.
2025-03-17 17:48:22.468482: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4194304): Total Chunks: 3, Chunks in use: 0. 17.25MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-03-17 17:48:22.468768: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-03-17 17:48:22.469047: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16777216): Total Chunks: 1, Chunks in use: 1. 29.69MiB allocated for chunks. 29.69MiB in use in bin. 29.69MiB client-requested in use in bin.
2025-03-17 17:48:22.469321: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (33554432): Total Chunks: 5, Chunks in use: 2. 276.44MiB allocated for chunks. 118.75MiB in use in bin. 118.75MiB client-requested in use in bin.
2025-03-17 17:48:22.469604: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (67108864): Total Chunks: 1, Chunks in use: 0. 77.88MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-03-17 17:48:22.469879: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-03-17 17:48:22.470159: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2025-03-17 17:48:22.470435: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Bin for 89.06MiB was 64.00MiB, Chunk State:
2025-03-17 17:48:22.470721: I tensorflow/core/common_runtime/bfc_allocator.cc:1039] Size: 77.88MiB | Requested Size: 14.84MiB | in_use: 0 | bin_num: 18, prev: Size: 59.38MiB | Requested Size: 59.38MiB | in_use: 1 | bin_num: -1
2025-03-17 17:48:22.470997: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 2097152
2025-03-17 17:48:22.471257: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000000 of size 1280 next 1
2025-03-17 17:48:22.471471: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000500 of size 256 next 2
2025-03-17 17:48:22.471502: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000600 of size 256 next 3
2025-03-17 17:48:22.471716: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000700 of size 256 next 4
2025-03-17 17:48:22.471744: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000800 of size 256 next 5
2025-03-17 17:48:22.471974: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000900 of size 256 next 8
2025-03-17 17:48:22.472005: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000a00 of size 256 next 9
2025-03-17 17:48:22.472222: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000b00 of size 256 next 10
2025-03-17 17:48:22.472249: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000c00 of size 256 next 13
2025-03-17 17:48:22.472477: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000d00 of size 256 next 6
2025-03-17 17:48:22.472507: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000e00 of size 256 next 61
2025-03-17 17:48:22.472723: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311000f00 of size 256 next 12
2025-03-17 17:48:22.472752: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001000 of size 256 next 55
2025-03-17 17:48:22.472972: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001100 of size 256 next 14
2025-03-17 17:48:22.473000: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001200 of size 256 next 17
2025-03-17 17:48:22.473234: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001300 of size 256 next 18
2025-03-17 17:48:22.473263: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001400 of size 256 next 19
2025-03-17 17:48:22.473474: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001500 of size 256 next 22
2025-03-17 17:48:22.473502: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001600 of size 256 next 23
2025-03-17 17:48:22.473733: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001700 of size 256 next 26
2025-03-17 17:48:22.473764: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001800 of size 256 next 27
2025-03-17 17:48:22.473972: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001900 of size 256 next 28
2025-03-17 17:48:22.474001: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001a00 of size 256 next 29
2025-03-17 17:48:22.474235: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001b00 of size 256 next 32
2025-03-17 17:48:22.474265: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001c00 of size 256 next 33
2025-03-17 17:48:22.474474: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001d00 of size 256 next 68
2025-03-17 17:48:22.474501: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001e00 of size 256 next 36
2025-03-17 17:48:22.474741: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311001f00 of size 256 next 37
2025-03-17 17:48:22.474770: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311002000 of size 256 next 38
2025-03-17 17:48:22.474978: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311002100 of size 256 next 57
2025-03-17 17:48:22.475007: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311002200 of size 256 next 42
2025-03-17 17:48:22.475231: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311002300 of size 256 next 43
2025-03-17 17:48:22.475260: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311002400 of size 256 next 44
2025-03-17 17:48:22.475473: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311002500 of size 256 next 30
2025-03-17 17:48:22.475501: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311002600 of size 256 next 59
2025-03-17 17:48:22.475738: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311002700 of size 256 next 47
2025-03-17 17:48:22.475767: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1311002800 of size 256 next 45
2025-03-17 17:48:22.475982: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311002900 of size 256 next 48
2025-03-17 17:48:22.476010: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311002a00 of size 256 next 49
2025-03-17 17:48:22.476239: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1311002b00 of size 3072 next 62
2025-03-17 17:48:22.476270: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311003700 of size 256 next 64
2025-03-17 17:48:22.476489: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311003800 of size 256 next 65
2025-03-17 17:48:22.476518: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311003900 of size 256 next 66
2025-03-17 17:48:22.476745: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311003a00 of size 256 next 67
2025-03-17 17:48:22.476775: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1311003b00 of size 2560 next 69
2025-03-17 17:48:22.476994: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311004500 of size 256 next 34
2025-03-17 17:48:22.477024: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311004600 of size 256 next 16
2025-03-17 17:48:22.477242: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311004700 of size 256 next 73
2025-03-17 17:48:22.477270: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311004800 of size 256 next 76
2025-03-17 17:48:22.477507: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311004900 of size 256 next 78
2025-03-17 17:48:22.477537: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311004a00 of size 256 next 54
2025-03-17 17:48:22.477768: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311004b00 of size 12800 next 11
2025-03-17 17:48:22.477831: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1311007d00 of size 147712 next 20
2025-03-17 17:48:22.478470: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102be00 of size 512 next 82
2025-03-17 17:48:22.478505: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102c000 of size 1536 next 88
2025-03-17 17:48:22.478725: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102c600 of size 512 next 92
2025-03-17 17:48:22.478753: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102c800 of size 512 next 87
2025-03-17 17:48:22.478973: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102ca00 of size 256 next 94
2025-03-17 17:48:22.479001: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102cb00 of size 256 next 95
2025-03-17 17:48:22.479230: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102cc00 of size 1280 next 101
2025-03-17 17:48:22.479262: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102d100 of size 256 next 103
2025-03-17 17:48:22.479472: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102d200 of size 768 next 104
2025-03-17 17:48:22.479502: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102d500 of size 256 next 39
2025-03-17 17:48:22.479741: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102d600 of size 512 next 24
2025-03-17 17:48:22.479771: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102d800 of size 1024 next 105
2025-03-17 17:48:22.479799: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131102dc00 of size 256 next 41
2025-03-17 17:48:22.480018: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 131102dd00 of size 84224 next 58
2025-03-17 17:48:22.480048: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311042600 of size 18432 next 15
2025-03-17 17:48:22.480277: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311046e00 of size 129024 next 74
2025-03-17 17:48:22.480310: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1311066600 of size 36864 next 102
2025-03-17 17:48:22.480530: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131106f600 of size 36864 next 77
2025-03-17 17:48:22.480560: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311078600 of size 36864 next 96
2025-03-17 17:48:22.482795: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1311081600 of size 1567232 next 18446744073709551615
2025-03-17 17:48:22.483024: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 4194304
2025-03-17 17:48:22.483238: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1311200000 of size 294912 next 25
2025-03-17 17:48:22.483460: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311248000 of size 589824 next 60
2025-03-17 17:48:22.483693: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 13112d8000 of size 1179648 next 84
2025-03-17 17:48:22.483912: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 13113f8000 of size 2129920 next 18446744073709551615
2025-03-17 17:48:22.484135: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 8388608
2025-03-17 17:48:22.484352: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1311600000 of size 1032192 next 53
2025-03-17 17:48:22.484382: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 13116fc000 of size 110592 next 100
2025-03-17 17:48:22.484638: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1311717000 of size 184320 next 80
2025-03-17 17:48:22.484949: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311744000 of size 147456 next 79
2025-03-17 17:48:22.485253: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1311768000 of size 147456 next 98
2025-03-17 17:48:22.485561: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131178c000 of size 147456 next 81
2025-03-17 17:48:22.485859: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 13117b0000 of size 442368 next 56
2025-03-17 17:48:22.486136: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 131181c000 of size 6176768 next 18446744073709551615
2025-03-17 17:48:22.486426: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 16777216
2025-03-17 17:48:22.486700: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311e00000 of size 589824 next 89
2025-03-17 17:48:22.486975: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1311e90000 of size 480000 next 99
2025-03-17 17:48:22.487216: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1311f05300 of size 1142016 next 97
2025-03-17 17:48:22.487243: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 131201c000 of size 294912 next 75
2025-03-17 17:48:22.487476: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1312064000 of size 5750784 next 91
2025-03-17 17:48:22.487509: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 13125e0000 of size 2359296 next 83
2025-03-17 17:48:22.487732: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1312820000 of size 6160384 next 18446744073709551615
2025-03-17 17:48:22.487765: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 134217728
2025-03-17 17:48:22.487798: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1314e00000 of size 46694400 next 7
2025-03-17 17:48:22.488036: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1317a88000 of size 31129600 next 21
2025-03-17 17:48:22.488066: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1319838000 of size 56393728 next 18446744073709551615
2025-03-17 17:48:22.488281: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 268435456
2025-03-17 17:48:22.488308: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 1331a00000 of size 62259200 next 51
2025-03-17 17:48:22.488529: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 1335560000 of size 62259200 next 52
2025-03-17 17:48:22.488557: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at 13390c0000 of size 62259200 next 31
2025-03-17 17:48:22.488788: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at 133cc20000 of size 81657856 next 18446744073709551615
2025-03-17 17:48:22.488822: I tensorflow/core/common_runtime/bfc_allocator.cc:1071] Summary of in-use Chunks by size:
2025-03-17 17:48:22.489048: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 52 Chunks of size 256 totalling 13.0KiB
2025-03-17 17:48:22.489079: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 4 Chunks of size 512 totalling 2.0KiB
2025-03-17 17:48:22.489295: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 768 totalling 768B
2025-03-17 17:48:22.489324: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1024 totalling 1.0KiB
2025-03-17 17:48:22.489554: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 1280 totalling 2.5KiB
2025-03-17 17:48:22.489587: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1536 totalling 1.5KiB
2025-03-17 17:48:22.489822: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 12800 totalling 12.5KiB
2025-03-17 17:48:22.489851: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 18432 totalling 18.0KiB
2025-03-17 17:48:22.490069: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 36864 totalling 72.0KiB
2025-03-17 17:48:22.490103: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 110592 totalling 108.0KiB
2025-03-17 17:48:22.490327: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 129024 totalling 126.0KiB
2025-03-17 17:48:22.490583: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 147456 totalling 288.0KiB
2025-03-17 17:48:22.490619: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 294912 totalling 288.0KiB
2025-03-17 17:48:22.490837: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 442368 totalling 432.0KiB
2025-03-17 17:48:22.490869: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 480000 totalling 468.8KiB
2025-03-17 17:48:22.491092: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 589824 totalling 1.12MiB
2025-03-17 17:48:22.491121: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1179648 totalling 1.12MiB
2025-03-17 17:48:22.491345: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 2129920 totalling 2.03MiB
2025-03-17 17:48:22.491376: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 2359296 totalling 2.25MiB
2025-03-17 17:48:22.491590: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 31129600 totalling 29.69MiB
2025-03-17 17:48:22.491622: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 62259200 totalling 118.75MiB
2025-03-17 17:48:22.491863: I tensorflow/core/common_runtime/bfc_allocator.cc:1078] Sum Total of in-use chunks: 156.76MiB
2025-03-17 17:48:22.491901: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] total_region_allocated_bytes_: 434110464 memory_limit_: 9397665792 available bytes: 8963555328 curr_region_allocation_bytes_: 4294967296
2025-03-17 17:48:22.492127: I tensorflow/core/common_runtime/bfc_allocator.cc:1086] Stats:
Limit: 9397665792
InUse: 164374528
MaxInUse: 1626649600
NumAllocs: 509
MaxAllocSize: 1207486720
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
Traceback
2025-03-17 17:48:22.492183: W tensorflow/core/common_runtime/bfc_allocator.cc:474] **_*_*____________********____________***************_____________****************__________________
2025-03-17 17:48:22.492241: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at concat_op.cc:158 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[4,192,152,200] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\Scripts\sleap-track-script.py", line 33, in <module>
Versions:
sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')())
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 5424, in main
labels_pr = predictor.predict(provider)
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 526, in predict
self._make_labeled_frames_from_generator(generator, data)
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1583, in _make_labeled_frames_from_generator
for ex in generator:
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 457, in _predict_generator
ex = process_batch(ex)
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 399, in process_batch
preds = self.inference_model.predict_on_batch(ex, numpy=True)
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1069, in predict_on_batch
outs = super().predict_on_batch(data, **kwargs)
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1986, in predict_on_batch
outputs = self.predict_function(iterator)
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\tensorflow\python\util\traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[4,192,152,200] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node single_instance_inference_model/single_instance_inference_layer/model/stack0_dec1_s8_to_s4_skip_concat/concat
(defined at C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\backend.py:3224)
]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[[single_instance_inference_model/single_instance_inference_layer/PartitionedCall_1/Cast/_22]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
(1) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[4,192,152,200] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node single_instance_inference_model/single_instance_inference_layer/model/stack0_dec1_s8_to_s4_skip_concat/concat
(defined at C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\backend.py:3224)
]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
0 successful operations.
0 derived errors ignored. [Op:__inference_predict_function_2313]
Errors may have originated from an input operation.
Input Source operations connected to node single_instance_inference_model/single_instance_inference_layer/model/stack0_dec1_s8_to_s4_skip_concat/concat:
In[0] single_instance_inference_model/single_instance_inference_layer/model/stack0_enc2_act1_relu/Relu (defined at C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\backend.py:4867)
In[1] single_instance_inference_model/single_instance_inference_layer/model/stack0_dec1_s8_to_s4_interp_bilinear/resize/ResizeBilinear (defined at C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\backend.py:3334)
In[2] single_instance_inference_model/single_instance_inference_layer/model/stack0_dec1_s8_to_s4_skip_concat/concat/axis:
Operation defined at: (most recent call last)
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\Scripts\sleap-track-script.py", line 33, in <module>
>>> sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')())
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 5424, in main
>>> labels_pr = predictor.predict(provider)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 526, in predict
>>> self._make_labeled_frames_from_generator(generator, data)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1583, in _make_labeled_frames_from_generator
>>> for ex in generator:
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 457, in _predict_generator
>>> ex = process_batch(ex)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 399, in process_batch
>>> preds = self.inference_model.predict_on_batch(ex, numpy=True)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1069, in predict_on_batch
>>> outs = super().predict_on_batch(data, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1986, in predict_on_batch
>>> outputs = self.predict_function(iterator)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1621, in predict_function
>>> return step_function(self, iterator)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1611, in step_function
>>> outputs = model.distribute_strategy.run(run_step, args=(data,))
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1604, in run_step
>>> outputs = model.predict_step(data)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1572, in predict_step
>>> return self(x, training=False)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1400, in call
>>> return self.single_instance_layer(example)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1329, in call
>>> preds = self.keras_model(imgs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\functional.py", line 452, in call
>>> inputs, training=training, mask=mask)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\functional.py", line 589, in _run_internal_graph
>>> outputs = node.layer(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\layers\merge.py", line 183, in call
>>> return self._merge_function(inputs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\layers\merge.py", line 528, in _merge_function
>>> return backend.concatenate(inputs, axis=self.axis)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\backend.py", line 3224, in concatenate
>>> return tf.concat([to_dense(x) for x in tensors], axis)
>>>
Input Source operations connected to node single_instance_inference_model/single_instance_inference_layer/model/stack0_dec1_s8_to_s4_skip_concat/concat:
In[0] single_instance_inference_model/single_instance_inference_layer/model/stack0_enc2_act1_relu/Relu (defined at C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\backend.py:4867)
In[1] single_instance_inference_model/single_instance_inference_layer/model/stack0_dec1_s8_to_s4_interp_bilinear/resize/ResizeBilinear (defined at C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\backend.py:3334)
In[2] single_instance_inference_model/single_instance_inference_layer/model/stack0_dec1_s8_to_s4_skip_concat/concat/axis:
Operation defined at: (most recent call last)
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\Scripts\sleap-track-script.py", line 33, in <module>
>>> sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')())
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 5424, in main
>>> labels_pr = predictor.predict(provider)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 526, in predict
>>> self._make_labeled_frames_from_generator(generator, data)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1583, in _make_labeled_frames_from_generator
>>> for ex in generator:
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 457, in _predict_generator
>>> ex = process_batch(ex)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 399, in process_batch
>>> preds = self.inference_model.predict_on_batch(ex, numpy=True)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1069, in predict_on_batch
>>> outs = super().predict_on_batch(data, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1986, in predict_on_batch
>>> outputs = self.predict_function(iterator)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1621, in predict_function
>>> return step_function(self, iterator)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1611, in step_function
>>> outputs = model.distribute_strategy.run(run_step, args=(data,))
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1604, in run_step
>>> outputs = model.predict_step(data)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\training.py", line 1572, in predict_step
>>> return self(x, training=False)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1400, in call
>>> return self.single_instance_layer(example)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1329, in call
>>> preds = self.keras_model(imgs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\functional.py", line 452, in call
>>> inputs, training=training, mask=mask)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\functional.py", line 589, in _run_internal_graph
>>> outputs = node.layer(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\engine\base_layer.py", line 1083, in __call__
>>> outputs = call_fn(inputs, *args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\utils\traceback_utils.py", line 92, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\layers\merge.py", line 183, in call
>>> return self._merge_function(inputs)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\layers\merge.py", line 528, in _merge_function
>>> return backend.concatenate(inputs, axis=self.axis)
>>>
>>> File "C:\Users\Nape_Computer_2\anaconda3\envs\sleap\lib\site-packages\keras\backend.py", line 3224, in concatenate
>>> return tf.concat([to_dense(x) for x in tensors], axis)
>>>
Function call stack:
predict_function -> predict_function
SLEAP: 1.3.3
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Windows-10-10.0.19041-SP0
System:
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True
Process return code: 1
Hello @ericas12,
Thanks for the additional info.
Analysis
The error is an "OOM: Out of Memory" issue which means your GPU did not have enough contiguous memory when trying to run inference. Your GPU does in fact have enough (fragmented) memory available:
INFO:sleap.nn.inference:Auto-selected GPU 0 with 7646 MiB of free memory.
, but was unsuccessful when attempting to allocate smaller chunks of contiguous memory:
2025-03-17 17:48:08.459043: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
2025-03-17 17:48:09.817328: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.824270: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 3.60G (3865470464 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.830345: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 3.24G (3478923264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.836352: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 2.92G (3131030784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.842345: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 2.62G (2817927680 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.848336: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 2.36G (2536134912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.854267: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 2.12G (2282521344 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2025-03-17 17:48:09.854399: W tensorflow/core/common_runtime/bfc_allocator.cc:343] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
2025-03-17 17:48:22.463403: W tensorflow/core/common_runtime/bfc_allocator.cc:462] Allocator (GPU_0_bfc) ran out of memory trying to allocate 89.06MiB (rounded to 93388800)requested by op single_instance_inference_model/single_instance_inference_layer/model/stack0_dec1_s8_to_s4_skip_concat/concat
.
Action Items to Try
It's possible that memory is being used by other programs running on your computer. You can run the command:
nvidia-smi
to see a description of any processes that might be consuming memory. I would try running this command, closing any programs that might be using GPU memory, and rerunning inference.
Let us know if that helps.
Thanks, Liezl
I ran the command and it did not look like any other programs were using memory, however, I did close out of some things to see if it would help. I tried running inference again and instead got this new error.
I haven't done anything new to the project since reaching out. The window in the GUI to run inference doesn't even open when I try to run it, the error just appears in the command terminal.
Hi @ericas12,
Thanks for the follow-up.
Yes, the slp file should never have multiple skeletons in it. We have closed entry points for this bug in the past, but we are suspicious that there may still be an open entry point on merging labels that should be fixed by this (when merged):
- #2075
.
Do you mind uploading your slp file (no need for models or videos, just the slp) to this form for debugging/correction? I'd like to see which skeletons there are - and which skeletons are unused (if any) or used less. Please include your email as a contact in the form as I'll send you back a corrected slp.
Alternatively, there are some instructions in this comment on removing unused skeletons yourself:
- https://github.com/talmolab/sleap/issues/713#issuecomment-1170560224
.
Apologies for the troubles, Liezl
Hi @roomrys,
Thank you, I submitted our slp file! However, I tried running inference on just one other video in the meanwhile, and I am still getting the error I was getting before with the memory. No other programs are running on the computer that are using GPU. Do you have any other suggestions on what it might be?
Thank you for your help!
Hi @ericas12,
Thanks again for the follow-up.
The labels you sent
Strange, but when opening the labels I did not see multiple Skeletons - in fact, I saw 0 Skeletons and just 8 Videos:
In [1]: ds = "inference_data.v001.slp" # Renamed by me
In [2]: import sleap
In [3]: labels = sleap.load_file(ds)
In [4]: labels
Out[4]: Labels(labeled_frames=0, videos=8, skeletons=0, tracks=0)
. Let's focus on the other issue unless this comes up again.
OOM issue
Memory-efficient models
Changing the number of videos will not help with the OOM issue. Things that would help with the OOM would require re-training a model and:
- decreasing batch size
- decreasing input scaling
- switching to top-down model (most memory efficient)
(see this thread:
- https://github.com/talmolab/sleap/issues/1570#issuecomment-1779750493
). However, your hardware seems perfectly capable of handling the requested workload, so I would not advice this (yet).
More ideas to try
I've been researching the issue a bit more here:
- https://github.com/tensorflow/models/issues/1993
. Some of the easier things to try that have helped others are:
- Restart your computer (comment)
- Run inference on CPU... while doable, for the number of frames you are running on, not ideal. (comment)
Thanks for your patience (and follow-ups), Liezl
Hi @ericas12,
Any updates from your end? I also wonder if this might be a shared/remote GPU that you are using?
Thanks, Liezl
Hi @roomrys,
I apologize for the delayed response; I haven't had the opportunity to try it again until now. I don't think we use a shared/remote GPU but I did restart the computer and it seems to be up and running again!
Thank you for all your help!