Clearing GPU memory in Keras

[SphrGhfri]

80% my GPU memory get's full after loading pre-trained Xception model. but after deleting my model , memory doesn't get empty or flush. I've also used codes like : K.clear_session() , gc.collect() , tf.reset_default_graph() , del model but none of them worked. Gpu properties say's 85% of memory is full.

Nothing flush gpu memory except numba.cuda.close() but won't allow me to use my gpu again. The only way to clear it is restarting kernel and rerun my code.

I'm looking for any script code to add my code allow me to use my code in for loop and clear gpu in every loop.

Part of my code :

image_input = Input(shape=(224, 224, 3)) base_model = Xception(input_tensor=image_input, include_top=False,weights='imagenet') base_model.compile(loss='categorical_crossentropy',optimizer='adadelta',metrics=['accuracy']) hist = base_model.fit(X,Y,epochs=2)

System information

Have I written custom code :
Windows 10 64-bit
TensorFlow installed from conda install tensorflow-gpu
TensorFlow version: 1.3
Python version: 3.6
CUDA/cuDNN version: 9.2
GPU model and memory: Asus GTX 1060 6gb

[nateraw]

This function looks promising, stolen from fastai forums

from keras.backend.tensorflow_backend import set_session
from keras.backend.tensorflow_backend import clear_session
from keras.backend.tensorflow_backend import get_session
import tensorflow

# Reset Keras Session
def reset_keras():
    sess = get_session()
    clear_session()
    sess.close()
    sess = get_session()

    try:
        del classifier # this is from global space - change this as you need
    except:
        pass

    print(gc.collect()) # if it's done something you should see a number being outputted

    # use the same config as you used to create the session
    config = tensorflow.ConfigProto()
    config.gpu_options.per_process_gpu_memory_fraction = 1
    config.gpu_options.visible_device_list = "0"
    set_session(tensorflow.Session(config=config))

[SphrGhfri]

@nateraw Thank you very much !!! The only script that works. now i can use the for loop to train multiple times.

[Moondra]

This function looks promising, stolen from fastai forums

from keras.backend.tensorflow_backend import set_session
from keras.backend.tensorflow_backend import clear_session
from keras.backend.tensorflow_backend import get_session
import tensorflow

# Reset Keras Session
def reset_keras():
    sess = get_session()
    clear_session()
    sess.close()
    sess = get_session()

    try:
        del classifier # this is from global space - change this as you need
    except:
        pass

    print(gc.collect()) # if it's done something you should see a number being outputted

    # use the same config as you used to create the session
    config = tensorflow.ConfigProto()
    config.gpu_options.per_process_gpu_memory_fraction = 1
    config.gpu_options.visible_device_list = "0"
    set_session(tensorflow.Session(config=config))

I keep getting "CUDA_ERROR_OUT_OF_MEMORY" when running the above function. The only thing that clears up my memory is restarting my computer.

[ambigus9]

This function looks promising, stolen from fastai forums

from keras.backend.tensorflow_backend import set_session
from keras.backend.tensorflow_backend import clear_session
from keras.backend.tensorflow_backend import get_session
import tensorflow

# Reset Keras Session
def reset_keras():
    sess = get_session()
    clear_session()
    sess.close()
    sess = get_session()

    try:
        del classifier # this is from global space - change this as you need
    except:
        pass

    print(gc.collect()) # if it's done something you should see a number being outputted

    # use the same config as you used to create the session
    config = tensorflow.ConfigProto()
    config.gpu_options.per_process_gpu_memory_fraction = 1
    config.gpu_options.visible_device_list = "0"
    set_session(tensorflow.Session(config=config))

Thanks! Works perfect!!

[cpoptic]

running the above reset_keras() function still throws an OOM error on Ubuntu with 16GB of GPU memory.

InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: out of memory

TensorFlow installed from conda install tensorflow-gpu TensorFlow version: 1.14 Python version: 3.6 CUDA/cuDNN version: 10.0.168 GPU model and memory: Tesla V100-PCIE-16GB 16gb

Same when I try running:

#from keras import backend as K
import tensorflow as tf
from tensorflow.keras import backend as K
curr_session = tf.get_default_session()
# close current session
if curr_session is not None:
    curr_session.close()
# reset graph
K.clear_session()
# create new session
s = tf.InteractiveSession()
K.set_session(s)

I find it fascinating that the TensorFlow team has not made a very straightforward way to clear GPU memory from a session. So much is broken with TF. Little annoyances like this; a user reasonably expects TF to handle clearing CUDA memory or have memory leaks, yet there appears no explicit way to handle this. Even K.clear_session() doesn't work. This is not unreasonable. Maybe the blame should be directed towards nvidia, as even the following code doesn't clear the memory:

from numba import cuda
cuda.select_device(0)
cuda.close()

After several hours of scouring StackOverflow and the Github issues, and trying the above approaches (none of which worked for some reason), I'm left with the decidedly inelegant approach of restarting the entire kernel. Frustrating.

[ilyarudyak]

Why do they close issues that are not solved like at all?

[mattalhonte]

Having this issue too. Restarting doesn't always fix it either. Even shutting down Jupyter doesn't necessarily fix it. Using Windows, no idea how to fix this when it pops up. Any time I restart the kernel with TF on it there's a random chance it'll happen after restarting.

[akshaybabloo]

Is this still happening if you use fit_generator(), looking at OP's issue with full memory I guess its because of the dataset being too big?

[elmahyai]

Any solution, yet ?

[azelk]

After several hours of scouring StackOverflow and the Github issues, and trying the above approaches (none of which worked for some reason), I'm left with the decidedly inelegant approach of restarting the entire kernel. Frustrating.

I have to do the same :frowning_face:

[omertortumlu]

Tensorflow version: 2.1.0 Keras version: 2.3.1 When use reset_keras() function; I get this error get_session is not available when using TensorFlow 2.0. Do you have a working function for this version :))

[devdimit93]

Tensorflow version: 2.1.0 Keras version: 2.3.1 When use reset_keras() function; I get this error get_session is not available when using TensorFlow 2.0. Do you have a working function for this version :))

It works for me. Google colab with TF 2.3.0

#reset Keras Session
def reset_keras():
    sess = tf.compat.v1.keras.backend.get_session()
    tf.compat.v1.keras.backend.clear_session()
    sess.close()
    sess = tf.compat.v1.keras.backend.get_session()

    try:
        del classifier # this is from global space - change this as you need
    except:
        pass

    # use the same config as you used to create the session
    config = tf.compat.v1.ConfigProto()
    config.gpu_options.per_process_gpu_memory_fraction = 1
    config.gpu_options.visible_device_list = "0"
    tf.compat.v1.keras.backend.set_session(tf.compat.v1.Session(config=config))

[SUFEHeisenberg]

Tensorflow version: 2.1.0 Keras version: 2.3.1 When use reset_keras() function; I get this error get_session is not available when using TensorFlow 2.0. Do you have a working function for this version :))

It works for me. Google colab with TF 2.3.0

#reset Keras Session
def reset_keras():
    sess = tf.compat.v1.keras.backend.get_session()
    tf.compat.v1.keras.backend.clear_session()
    sess.close()
    sess = tf.compat.v1.keras.backend.get_session()

    try:
        del classifier # this is from global space - change this as you need
    except:
        pass

    # use the same config as you used to create the session
    config = tf.compat.v1.ConfigProto()
    config.gpu_options.per_process_gpu_memory_fraction = 1
    config.gpu_options.visible_device_list = "0"
    tf.compat.v1.keras.backend.set_session(tf.compat.v1.Session(config=config))

Thanks a lot for sharing your code in TF 2.0, but I still cannot fix it up with TF==2.2.0& keras ==2.3.1. And after I run your code, the console shows following messages BUT GPU still almost full which confuses me a lot :

2021-05-09 23:12:03.111033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:21:00.0 name: Quadro P2000 computeCapability: 6.1 coreClock: 1.4805GHz coreCount: 8 deviceMemorySize: 5.00GiB deviceMemoryBandwidth: 130.53GiB/s 2021-05-09 23:12:03.111607: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll 2021-05-09 23:12:03.111866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll 2021-05-09 23:12:03.112118: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll 2021-05-09 23:12:03.112363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll 2021-05-09 23:12:03.112637: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll 2021-05-09 23:12:03.112897: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll 2021-05-09 23:12:03.113170: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll 2021-05-09 23:12:03.113497: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2021-05-09 23:12:03.113796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-05-09 23:12:03.114042: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2021-05-09 23:12:03.114196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2021-05-09 23:12:03.114435: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3841 MB memory) -> physical GPU (device: 0, name: Quadro P2000, pci bus id: 0000:21:00.0, compute capability: 6.1)

May I ask you have you ever met this issue? BTW, could you please me what the classifier (from the global space) is in your own code? should I just del model?

[BatmanofZuhandArrgh]

Tensorflow version: 2.1.0 Keras version: 2.3.1 When use reset_keras() function; I get this error get_session is not available when using TensorFlow 2.0. Do you have a working function for this version :))

It works for me. Google colab with TF 2.3.0

#reset Keras Session
def reset_keras():
    sess = tf.compat.v1.keras.backend.get_session()
    tf.compat.v1.keras.backend.clear_session()
    sess.close()
    sess = tf.compat.v1.keras.backend.get_session()

    try:
        del classifier # this is from global space - change this as you need
    except:
        pass

    # use the same config as you used to create the session
    config = tf.compat.v1.ConfigProto()
    config.gpu_options.per_process_gpu_memory_fraction = 1
    config.gpu_options.visible_device_list = "0"
    tf.compat.v1.keras.backend.set_session(tf.compat.v1.Session(config=config))

Didn't work for me

[maitry63]

Hey @SphrGhfri, I tested the reset_keras() function, and it works fine with get_session() in the latest version of Keras. I've attached the gist here: gist

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed because it has been inactive for 28 days. Please reopen if you'd like to work on this further.