GPU memory leak.

[epbsb]

Hello,

I'm using pycave for a project where the data is unidimensional of size 8e9. The GPU options works well, and I'm splitting the data in "for loops" to do the predictions. However, as the loops goes on, it takes more and more of the GPU memory, and eventually runs out of memory. To contour this issue, I'm using torch clean cache at each interaction in addition to the garbage collector function in python, as shown in the code below, however this process is slow.

import gc
def clear_gpu_memory():
    torch.cuda.empty_cache()
    gc.collect()
    torch.cuda.empty_cache()

I've tried to use the pycave built-in function of batches as well, but it also runs in the memory issue.

Is there anything I could do to fix this?