CUDA out of memory

[svanschalkwyk]

Two M40 24GB CUDA devices. Nothing else running on them. Tried to lower "bsize" as well.

2020-12-01 14:10:17.531996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21359 MB memory) -> physical GPU (device: 0, name: Tesla M40 24GB, pci bus id: 0000:04:00.0, compute capability: 5.2)
2020-12-01 14:10:17.533698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 21359 MB memory) -> physical GPU (device: 1, name: Tesla M40 24GB, pci bus id: 0000:83:00.0, compute capability: 5.2)
Vocab size : 100000
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
INFO:werkzeug: * Running on http://0.0.0.0:8001/ (Press CTRL+C to quit)
2020-12-01 14:11:05.707333: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 22.41 GiB total capacity; 764.76 MiB already allocated; 12.81 MiB free; 812.00 MiB reserved in total by PyTorch)
INFO:werkzeug:192.168.5.100 - - [01/Dec/2020 14:11:21] "POST /get_predictions HTTP/1.1" 500 -