demo_run program crashing?

[keyansaravanan]

Program crashing when beginning the epoch1? like this, Epoch 1/300 tcmalloc: large alloc 3623936000 bytes == 0x55d811920000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f tcmalloc: large alloc 3623936000 bytes == 0x55d8f24ae000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f tcmalloc: large alloc 4831920128 bytes == 0x55d8f24ae000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f tcmalloc: large alloc 4831920128 bytes == 0x55d9324ae000 @ 0x7fe4a6f961e7 0x7fe473f84902 0x7fe476b62110 0x7fe476bc6bd1 0x7fe476bc86d8 0x7fe476bcac83 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f 2021-03-07 11:35:48.388501: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 7247757312 exceeds 10% of system memory. 2021-03-07 11:35:48.388501: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 7247757312 exceeds 10% of system memory. tcmalloc: large alloc 7247757312 bytes == 0x55d9324ae000 @ 0x7fe4a6f78b6b 0x7fe4a6f98379 0x7fe470171087 0x7fe46ff6d59f 0x7fe46fe37f1b 0x7fe46fdfd8f6 0x7fe46fe005f1 0x7fe476aa1c00 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f tcmalloc: large alloc 7247757312 bytes == 0x55daeacd0000 @ 0x7fe4a6f78b6b 0x7fe4a6f98379 0x7fe470171087 0x7fe46ff6d59f 0x7fe46fe37f1b 0x7fe46fdfd8f6 0x7fe46fe005f1 0x7fe476aa2c3d 0x7fe4700a000c 0x7fe470092775 0x7fe4701531c1 0x7fe4701508b8 0x7fe4a58786df 0x7fe4a6d4b6db 0x7fe4a5e8071f ^C

[woodywff]

That's an out of memory error. $ nvidia-smi to check you GPU status.