MinkowskiEngine CUDA OOM error leads to GPU memory leak

CUDA OOM error leads to GPU memory leak

Open evelkey opened this issue 4 years ago • 11 comments

trafficstars

Describe the bug

I migrated from CUDA 10.2 PyTorch 1.7.1 to CUDA 11.0.221 PyTorch 1.7.1 (current ME master) to solve an issue with uncatchable exceptions. We encountered a memory leak when there is an exception in C++ and we handle it from python. As we cannot "reset" the GPU it's not possible to recover in the same process after this OOM issue occurs. Found a similar issue here: https://github.com/pytorch/pytorch/issues/38966.

To Reproduce

I'm currently working on a minimalistic code snippet but the problem generically occurs when dealing with large number of points and an exception occurs in C++ code.

After each iteration I do:

del x,y,z
gc.collect()
torch.cuda.empty_cache()

Example 6598MB -> 6134MB:

2021-06-02 16:00:40 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6598.5
2021-06-02 16:00:43 dev-ssh-geza root[14429] DEBUG std::bad_alloc: cudaErrorMemoryAllocation: out of memory
2021-06-02 16:00:43 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
  File "/.../gh3d_train.py", line 466, in train
    out = model(input_st)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File ".../net/minkunet.py", line 213, in forward
    out = self.block8(out)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/modules/resnet_block.py", line 55, in forward
    out = self.conv1(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 314, in forward
    outfeat = self.conv.apply(
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 72, in forward
    return fw_fn(
MemoryError: std::bad_alloc: cudaErrorMemoryAllocation: out of memory

2021-06-02 16:00:43 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:43 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6134.5

Other example:

2021-06-02 15:59:24 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:59:25 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7872.5
2021-06-02 15:59:30 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 202.00 MiB (GPU 0; 15.78 GiB total capacity; 13.47 GiB already allocated; 148.75 MiB free; 14.00 GiB reserved in total by PyTorch)
2021-06-02 15:59:30 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
  File "...gh3d_train.py", line 466, in train
    out = model(input_st)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File ".../net/minkunet.py", line 197, in forward
    out = self.block7(out)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/modules/resnet_block.py", line 55, in forward
    out = self.conv1(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 314, in forward
    outfeat = self.conv.apply(
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 72, in forward
    return fw_fn(
RuntimeError: CUDA out of memory. Tried to allocate 202.00 MiB (GPU 0; 15.78 GiB total capacity; 13.47 GiB already allocated; 148.75 MiB free; 14.00 GiB reserved in total by PyTorch)

2021-06-02 15:59:30 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:59:30 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7670.5
2021-06-02 15:59:34 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0013539012288674712
2021-06-02 15:59:37 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7670.5

Sometimes even happens without an exception:

2021-06-02 17:50:09 dev-ssh-geza root[26709] DEBUG Free GPU ram: 13722.5
2021-06-02 17:50:12 dev-ssh-geza root[26709] DEBUG Batch loss: 0.09159479290246964
2021-06-02 17:50:15 dev-ssh-geza root[26709] DEBUG Free GPU ram: 13722.5
2021-06-02 17:50:18 dev-ssh-geza root[26709] DEBUG Batch loss: 0.05059795454144478
2021-06-02 17:50:22 dev-ssh-geza root[26709] DEBUG Free GPU ram: 13722.5
2021-06-02 17:50:26 dev-ssh-geza root[26709] DEBUG Batch loss: 0.03134541213512421
2021-06-02 17:50:29 dev-ssh-geza root[26709] DEBUG Free GPU ram: 12800.5
2021-06-02 17:50:32 dev-ssh-geza root[26709] DEBUG Batch loss: 0.02406206540763378
2021-06-02 17:50:36 dev-ssh-geza root[26709] DEBUG Free GPU ram: 12800.5
2021-06-02 17:50:39 dev-ssh-geza root[26709] DEBUG Batch loss: 0.021245351061224937
2021-06-02 17:50:43 dev-ssh-geza root[26709] DEBUG Free GPU ram: 12800.5
2021-06-02 17:50:47 dev-ssh-geza root[26709] DEBUG Batch loss: 0.01783326268196106
2021-06-02 17:50:50 dev-ssh-geza root[26709] DEBUG Free GPU ram: 12800.5
2021-06-02 17:50:53 dev-ssh-geza root[26709] DEBUG Batch loss: 0.01228021178394556
2021-06-02 17:50:56 dev-ssh-geza root[26709] DEBUG Free GPU ram: 12800.5

Free VRAM is measured with:

import torch

def available_cuda_memory_in_mbytes(safety_margin_for_gpu_memory=512):

    cuda_total_memory = torch.cuda.get_device_properties(torch.cuda.current_device).total_memory
    cuda_cached_memory = torch.cuda.memory_reserved(torch.cuda.current_device)

    available_cuda_memory_for_dbscan_in_mbytes = (
        cuda_total_memory - cuda_cached_memory
    ) / 1024 ** 2 - safety_margin_for_gpu_memory

    return available_cuda_memory_for_dbscan_in_mbytes

logging.debug(f"Free GPU ram: {available_cuda_memory_in_mbytes(safety_margin_for_gpu_memory=0)}")

Logs of the constantly vanishing available memory

2021-06-02 15:48:18 dev-ssh-geza bif_train[14429] INFO Training started
2021-06-02 15:49:03 dev-ssh-geza root[14429] DEBUG Free GPU ram: 15970.5
2021-06-02 15:49:03 dev-ssh-geza bif_train[14429] INFO batch info: coords: torch.Size([1511440, 4]), features:torch.Size([1511440, 3]), 
2021-06-02 15:49:12 dev-ssh-geza root[14429] DEBUG Batch loss: 1.7967263460159302
2021-06-02 15:49:16 dev-ssh-geza root[14429] INFO Iteration: 0/1184, epoch 0, Bif Loss: 1.7967263460159302,  lr: 0.10000
2021-06-02 15:49:18 dev-ssh-geza root[14429] DEBUG Free GPU ram: 12116.5
2021-06-02 15:49:25 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 224.00 MiB (GPU 0; 15.78 GiB total capacity; 13.61 GiB already allocated; 156.75 MiB free; 13.99 GiB reserved in total by PyTorch)
2021-06-02 15:49:25 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:49:25 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:49:26 dev-ssh-geza root[14429] DEBUG Free GPU ram: 12116.5
2021-06-02 15:49:37 dev-ssh-geza root[14429] DEBUG Batch loss: 0.48397472500801086
2021-06-02 15:49:41 dev-ssh-geza root[14429] DEBUG Free GPU ram: 12140.5
2021-06-02 15:49:45 dev-ssh-geza root[14429] DEBUG Batch loss: 0.19845232367515564
2021-06-02 15:49:49 dev-ssh-geza root[14429] DEBUG Free GPU ram: 12140.5
2021-06-02 15:49:53 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0688539370894432
2021-06-02 15:49:55 dev-ssh-geza root[14429] DEBUG Free GPU ram: 12140.5
2021-06-02 15:49:58 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 220.00 MiB (GPU 0; 15.78 GiB total capacity; 13.68 GiB already allocated; 38.75 MiB free; 14.11 GiB reserved in total by PyTorch)
2021-06-02 15:49:58 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:49:58 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:49:59 dev-ssh-geza root[14429] DEBUG Free GPU ram: 12140.5
2021-06-02 15:50:03 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 228.00 MiB (GPU 0; 15.78 GiB total capacity; 13.57 GiB already allocated; 192.75 MiB free; 13.96 GiB reserved in total by PyTorch)
2021-06-02 15:50:03 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:50:03 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:50:03 dev-ssh-geza root[14429] DEBUG Free GPU ram: 12140.5
2021-06-02 15:50:07 dev-ssh-geza root[14429] DEBUG Batch loss: 0.04011991620063782
2021-06-02 15:50:10 dev-ssh-geza root[14429] DEBUG Free GPU ram: 12140.5
2021-06-02 15:50:13 dev-ssh-geza root[14429] DEBUG Batch loss: 0.025560127571225166
2021-06-02 15:50:16 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:50:19 dev-ssh-geza root[14429] DEBUG Batch loss: 0.018149442970752716
2021-06-02 15:50:22 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11458.5
2021-06-02 15:50:26 dev-ssh-geza root[14429] DEBUG Batch loss: 0.01395785715430975
2021-06-02 15:50:30 dev-ssh-geza root[14429] INFO Iteration: 10/1184, epoch 0, Bif Loss: 0.01395785715430975,  lr: 0.10000
2021-06-02 15:50:30 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11458.5
2021-06-02 15:50:33 dev-ssh-geza root[14429] DEBUG Batch loss: 0.008124448359012604
2021-06-02 15:50:35 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11458.5
2021-06-02 15:50:39 dev-ssh-geza root[14429] DEBUG Batch loss: 0.028406884521245956
2021-06-02 15:50:43 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11458.5
2021-06-02 15:50:46 dev-ssh-geza root[14429] DEBUG Batch loss: 0.008276463486254215
2021-06-02 15:50:49 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:50:53 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 222.00 MiB (GPU 0; 15.78 GiB total capacity; 13.47 GiB already allocated; 148.75 MiB free; 14.00 GiB reserved in total by PyTorch)
2021-06-02 15:50:53 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:50:53 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:50:53 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:50:57 dev-ssh-geza root[14429] DEBUG Batch loss: 0.00563023891299963
2021-06-02 15:50:59 dev-ssh-geza root[14429] INFO Iteration: 15/1184, epoch 0, Bif Loss: 0.00563023891299963,  lr: 0.10000
2021-06-02 15:50:59 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:51:03 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 422.00 MiB (GPU 0; 15.78 GiB total capacity; 13.10 GiB already allocated; 372.75 MiB free; 13.78 GiB reserved in total by PyTorch)
2021-06-02 15:51:03 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:51:03 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:51:03 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:51:06 dev-ssh-geza root[14429] DEBUG Batch loss: 0.010648244060575962
2021-06-02 15:51:10 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:51:13 dev-ssh-geza root[14429] DEBUG Batch loss: 0.007375478744506836
2021-06-02 15:51:15 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:51:19 dev-ssh-geza root[14429] DEBUG Batch loss: 0.004747511353343725
2021-06-02 15:51:22 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:51:26 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 208.00 MiB (GPU 0; 15.78 GiB total capacity; 13.63 GiB already allocated; 6.75 MiB free; 14.14 GiB reserved in total by PyTorch)
2021-06-02 15:51:26 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:51:26 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:51:26 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:51:29 dev-ssh-geza root[14429] DEBUG Batch loss: 0.00971425510942936
2021-06-02 15:51:33 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:51:36 dev-ssh-geza root[14429] DEBUG Batch loss: 0.008378207683563232
2021-06-02 15:51:39 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:51:42 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0039612650871276855
2021-06-02 15:51:45 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:51:48 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0076295011676847935
2021-06-02 15:51:50 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:51:53 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 250.00 MiB (GPU 0; 15.78 GiB total capacity; 13.43 GiB already allocated; 102.75 MiB free; 14.04 GiB reserved in total by PyTorch)
2021-06-02 15:51:53 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:51:53 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:51:54 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:51:58 dev-ssh-geza root[14429] DEBUG Batch loss: 0.003734805854037404
2021-06-02 15:51:58 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 210.00 MiB (GPU 0; 15.78 GiB total capacity; 13.49 GiB already allocated; 134.75 MiB free; 14.01 GiB reserved in total by PyTorch)
2021-06-02 15:51:58 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:51:58 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:51:58 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:52:02 dev-ssh-geza root[14429] DEBUG Batch loss: 0.005232085939496756
2021-06-02 15:52:05 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:52:08 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0038282519672065973
2021-06-02 15:52:11 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:52:14 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0035880189388990402
2021-06-02 15:52:17 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:52:20 dev-ssh-geza root[14429] DEBUG Batch loss: 0.003988857381045818
2021-06-02 15:52:20 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 190.00 MiB (GPU 0; 15.78 GiB total capacity; 13.54 GiB already allocated; 110.75 MiB free; 14.04 GiB reserved in total by PyTorch)
2021-06-02 15:52:20 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:52:20 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:52:21 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:52:24 dev-ssh-geza root[14429] DEBUG Batch loss: 0.002539991168305278
2021-06-02 15:52:27 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:52:30 dev-ssh-geza root[14429] DEBUG Batch loss: 0.002289108233526349
2021-06-02 15:52:33 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:52:37 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 500.00 MiB (GPU 0; 15.78 GiB total capacity; 13.13 GiB already allocated; 438.75 MiB free; 13.71 GiB reserved in total by PyTorch)
2021-06-02 15:52:37 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:52:37 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:52:37 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11460.5
2021-06-02 15:52:41 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 218.00 MiB (GPU 0; 15.78 GiB total capacity; 13.62 GiB already allocated; 46.75 MiB free; 14.10 GiB reserved in total by PyTorch)
2021-06-02 15:52:41 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:52:41 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:52:42 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11242.5
2021-06-02 15:52:46 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 576.00 MiB (GPU 0; 15.78 GiB total capacity; 13.45 GiB already allocated; 206.75 MiB free; 13.94 GiB reserved in total by PyTorch)
2021-06-02 15:52:46 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:52:46 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:52:47 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11242.5
2021-06-02 15:52:50 dev-ssh-geza root[14429] DEBUG Batch loss: 0.002650761278346181
2021-06-02 15:52:53 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11242.5
2021-06-02 15:52:56 dev-ssh-geza root[14429] DEBUG Batch loss: 0.003258180571720004
2021-06-02 15:52:59 dev-ssh-geza root[14429] DEBUG Free GPU ram: 11240.5
2021-06-02 15:53:03 dev-ssh-geza root[14429] DEBUG std::bad_alloc: cudaErrorMemoryAllocation: out of memory
2021-06-02 15:53:03 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:53:03 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:53:03 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10676.5
2021-06-02 15:53:07 dev-ssh-geza root[14429] DEBUG Batch loss: 0.002907110843807459
2021-06-02 15:53:11 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10676.5
2021-06-02 15:53:14 dev-ssh-geza root[14429] DEBUG Batch loss: 0.002752267988398671
2021-06-02 15:53:18 dev-ssh-geza root[14429] INFO Iteration: 40/1184, epoch 0, Bif Loss: 0.002752267988398671,  lr: 0.10000
2021-06-02 15:53:18 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10676.5
2021-06-02 15:53:22 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 194.00 MiB (GPU 0; 15.78 GiB total capacity; 13.60 GiB already allocated; 14.75 MiB free; 14.13 GiB reserved in total by PyTorch)
2021-06-02 15:53:22 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:53:22 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:53:23 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10678.5
2021-06-02 15:53:26 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 242.00 MiB (GPU 0; 15.78 GiB total capacity; 13.36 GiB already allocated; 228.75 MiB free; 13.92 GiB reserved in total by PyTorch)
2021-06-02 15:53:26 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:53:26 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:53:26 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10678.5
2021-06-02 15:53:30 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 542.00 MiB (GPU 0; 15.78 GiB total capacity; 13.34 GiB already allocated; 244.75 MiB free; 13.90 GiB reserved in total by PyTorch)
2021-06-02 15:53:30 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:53:30 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:53:31 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10678.5
2021-06-02 15:53:35 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0014050310710445046
2021-06-02 15:53:38 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10678.5
2021-06-02 15:53:41 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 510.00 MiB (GPU 0; 15.78 GiB total capacity; 13.49 GiB already allocated; 104.75 MiB free; 14.04 GiB reserved in total by PyTorch)
2021-06-02 15:53:41 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:53:41 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:53:41 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10678.5
2021-06-02 15:53:45 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0021043377928435802
2021-06-02 15:53:49 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10676.5
2021-06-02 15:53:54 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0022998256608843803
2021-06-02 15:53:58 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10676.5
2021-06-02 15:54:02 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0016059863846749067
2021-06-02 15:54:05 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10676.5
2021-06-02 15:54:09 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0015030876966193318
2021-06-02 15:54:12 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10676.5
2021-06-02 15:54:16 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 398.00 MiB (GPU 0; 15.78 GiB total capacity; 13.51 GiB already allocated; 28.75 MiB free; 14.12 GiB reserved in total by PyTorch)
2021-06-02 15:54:16 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:54:16 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:54:16 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10676.5
2021-06-02 15:54:20 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0015725989360362291
2021-06-02 15:54:23 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10676.5
2021-06-02 15:54:26 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 234.00 MiB (GPU 0; 15.78 GiB total capacity; 13.54 GiB already allocated; 58.75 MiB free; 14.09 GiB reserved in total by PyTorch)
2021-06-02 15:54:26 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:54:26 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:54:26 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10442.5
2021-06-02 15:54:30 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 282.00 MiB (GPU 0; 15.78 GiB total capacity; 13.49 GiB already allocated; 106.75 MiB free; 14.04 GiB reserved in total by PyTorch)
2021-06-02 15:54:30 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:54:30 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:54:30 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10442.5
2021-06-02 15:54:34 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0016234779031947255
2021-06-02 15:54:37 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10442.5
2021-06-02 15:54:41 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 232.00 MiB (GPU 0; 15.78 GiB total capacity; 13.31 GiB already allocated; 222.75 MiB free; 13.93 GiB reserved in total by PyTorch)
2021-06-02 15:54:41 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:54:41 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:54:42 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10442.5
2021-06-02 15:54:47 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0012412808137014508
2021-06-02 15:54:48 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10444.5
2021-06-02 15:54:54 dev-ssh-geza root[14429] DEBUG Batch loss: 0.002102112164720893
2021-06-02 15:54:57 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10444.5
2021-06-02 15:55:00 dev-ssh-geza root[14429] DEBUG Batch loss: 0.002046464942395687
2021-06-02 15:55:03 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10444.5
2021-06-02 15:55:06 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 222.00 MiB (GPU 0; 15.78 GiB total capacity; 13.35 GiB already allocated; 104.75 MiB free; 14.04 GiB reserved in total by PyTorch)
2021-06-02 15:55:06 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:55:06 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:55:06 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10444.5
2021-06-02 15:55:10 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0013181521790102124
2021-06-02 15:55:12 dev-ssh-geza root[14429] INFO Iteration: 60/1184, epoch 0, Bif Loss: 0.0013181521790102124,  lr: 0.10000
2021-06-02 15:55:12 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10444.5
2021-06-02 15:55:15 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0012966286158189178
2021-06-02 15:55:18 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10444.5
2021-06-02 15:55:21 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 208.00 MiB (GPU 0; 15.78 GiB total capacity; 13.48 GiB already allocated; 34.75 MiB free; 14.11 GiB reserved in total by PyTorch)
2021-06-02 15:55:21 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:55:21 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:55:21 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10236.5
2021-06-02 15:55:25 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 202.00 MiB (GPU 0; 15.78 GiB total capacity; 13.71 GiB already allocated; 36.75 MiB free; 14.11 GiB reserved in total by PyTorch)
2021-06-02 15:55:25 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:55:25 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:55:26 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10034.5
2021-06-02 15:55:29 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 536.00 MiB (GPU 0; 15.78 GiB total capacity; 13.15 GiB already allocated; 430.75 MiB free; 13.72 GiB reserved in total by PyTorch)
2021-06-02 15:55:29 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:55:29 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:55:30 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10034.5
2021-06-02 15:55:34 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0008652745746076107
2021-06-02 15:55:36 dev-ssh-geza root[14429] INFO Iteration: 65/1184, epoch 0, Bif Loss: 0.0008652745746076107,  lr: 0.10000
2021-06-02 15:55:36 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10034.5
2021-06-02 15:55:39 dev-ssh-geza root[14429] DEBUG Batch loss: 0.002144662430509925
2021-06-02 15:55:42 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10034.5
2021-06-02 15:55:45 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 198.00 MiB (GPU 0; 15.78 GiB total capacity; 13.60 GiB already allocated; 106.75 MiB free; 14.04 GiB reserved in total by PyTorch)
2021-06-02 15:55:45 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:55:45 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:55:45 dev-ssh-geza root[14429] DEBUG Free GPU ram: 10034.5
2021-06-02 15:55:49 dev-ssh-geza root[14429] DEBUG std::bad_alloc: cudaErrorMemoryAllocation: out of memory
2021-06-02 15:55:49 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:55:49 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:55:49 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9574.5
2021-06-02 15:55:54 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0015719735529273748
2021-06-02 15:55:57 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9574.5
2021-06-02 15:56:00 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 220.00 MiB (GPU 0; 15.78 GiB total capacity; 13.53 GiB already allocated; 104.75 MiB free; 14.04 GiB reserved in total by PyTorch)
2021-06-02 15:56:00 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:56:00 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:56:00 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9574.5
2021-06-02 15:56:04 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 198.00 MiB (GPU 0; 15.78 GiB total capacity; 13.77 GiB already allocated; 8.75 MiB free; 14.13 GiB reserved in total by PyTorch)
2021-06-02 15:56:04 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:56:04 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:56:04 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9376.5
2021-06-02 15:56:08 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0013531267177313566
2021-06-02 15:56:11 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9376.5
2021-06-02 15:56:14 dev-ssh-geza root[14429] DEBUG Batch loss: 0.002273419639095664
2021-06-02 15:56:16 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9376.5
2021-06-02 15:56:19 dev-ssh-geza root[14429] DEBUG Batch loss: 0.001437308732420206
2021-06-02 15:56:22 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9376.5
2021-06-02 15:56:25 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0005609480431303382
2021-06-02 15:56:27 dev-ssh-geza root[14429] INFO Iteration: 75/1184, epoch 0, Bif Loss: 0.0005609480431303382,  lr: 0.10000
2021-06-02 15:56:28 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9376.5
2021-06-02 15:56:31 dev-ssh-geza root[14429] DEBUG Batch loss: 0.002295363461598754
2021-06-02 15:56:34 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9374.5
2021-06-02 15:56:38 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0012436049291864038
2021-06-02 15:56:41 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9374.5
2021-06-02 15:56:44 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0011464578565210104
2021-06-02 15:56:46 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9374.5
2021-06-02 15:56:50 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 188.00 MiB (GPU 0; 15.78 GiB total capacity; 13.64 GiB already allocated; 120.75 MiB free; 14.03 GiB reserved in total by PyTorch)
2021-06-02 15:56:50 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:56:50 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:56:50 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9374.5
2021-06-02 15:56:53 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 234.00 MiB (GPU 0; 15.78 GiB total capacity; 13.35 GiB already allocated; 180.75 MiB free; 13.97 GiB reserved in total by PyTorch)
2021-06-02 15:56:53 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:56:53 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:56:53 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9374.5
2021-06-02 15:56:57 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 226.00 MiB (GPU 0; 15.78 GiB total capacity; 13.46 GiB already allocated; 128.75 MiB free; 14.02 GiB reserved in total by PyTorch)
2021-06-02 15:56:57 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:56:57 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:56:57 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9374.5
2021-06-02 15:57:03 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0005807738052681088
2021-06-02 15:57:06 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9374.5
2021-06-02 15:57:11 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 228.00 MiB (GPU 0; 15.78 GiB total capacity; 13.62 GiB already allocated; 20.75 MiB free; 14.12 GiB reserved in total by PyTorch)
2021-06-02 15:57:11 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:57:11 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:57:12 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9374.5
2021-06-02 15:57:16 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 206.00 MiB (GPU 0; 15.78 GiB total capacity; 13.66 GiB already allocated; 48.75 MiB free; 14.10 GiB reserved in total by PyTorch)
2021-06-02 15:57:16 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:57:16 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:57:16 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9374.5
2021-06-02 15:57:22 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0013863586355000734
2021-06-02 15:57:25 dev-ssh-geza root[14429] INFO Iteration: 85/1184, epoch 0, Bif Loss: 0.0013863586355000734,  lr: 0.10000
2021-06-02 15:57:25 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9374.5
2021-06-02 15:57:29 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 242.00 MiB (GPU 0; 15.78 GiB total capacity; 13.46 GiB already allocated; 220.75 MiB free; 13.93 GiB reserved in total by PyTorch)
2021-06-02 15:57:29 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:57:29 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:57:29 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9012.5
2021-06-02 15:57:34 dev-ssh-geza root[14429] DEBUG Batch loss: 0.000902367930393666
2021-06-02 15:57:38 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9012.5
2021-06-02 15:57:45 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0015246984548866749
2021-06-02 15:57:49 dev-ssh-geza root[14429] DEBUG Free GPU ram: 9012.5
2021-06-02 15:57:54 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0018272273009642959
2021-06-02 15:57:54 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 160.00 MiB (GPU 0; 15.78 GiB total capacity; 13.64 GiB already allocated; 90.75 MiB free; 14.05 GiB reserved in total by PyTorch)
2021-06-02 15:57:54 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:57:54 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:57:54 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8852.5
2021-06-02 15:58:00 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 478.00 MiB (GPU 0; 15.78 GiB total capacity; 13.47 GiB already allocated; 178.75 MiB free; 13.97 GiB reserved in total by PyTorch)
2021-06-02 15:58:00 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:58:00 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:58:01 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8852.5
2021-06-02 15:58:04 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0014570835046470165
2021-06-02 15:58:07 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8852.5
2021-06-02 15:58:10 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 212.00 MiB (GPU 0; 15.78 GiB total capacity; 13.41 GiB already allocated; 142.75 MiB free; 14.00 GiB reserved in total by PyTorch)
2021-06-02 15:58:10 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:58:10 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:58:10 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8852.5
2021-06-02 15:58:13 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0013765012845396996
2021-06-02 15:58:16 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8852.5
2021-06-02 15:58:19 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 172.00 MiB (GPU 0; 15.78 GiB total capacity; 13.75 GiB already allocated; 26.75 MiB free; 14.12 GiB reserved in total by PyTorch)
2021-06-02 15:58:19 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:58:19 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:58:20 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8680.5
2021-06-02 15:58:23 dev-ssh-geza root[14429] DEBUG Batch loss: 0.002021253574639559
2021-06-02 15:58:26 dev-ssh-geza root[14429] INFO Iteration: 95/1184, epoch 0, Bif Loss: 0.002021253574639559,  lr: 0.10000
2021-06-02 15:58:26 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8680.5
2021-06-02 15:58:30 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0011810617288574576
2021-06-02 15:58:33 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8680.5
2021-06-02 15:58:37 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0011345669627189636
2021-06-02 15:58:40 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8680.5
2021-06-02 15:58:43 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 258.00 MiB (GPU 0; 15.78 GiB total capacity; 13.42 GiB already allocated; 206.75 MiB free; 13.94 GiB reserved in total by PyTorch)
2021-06-02 15:58:43 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:58:43 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:58:43 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8680.5
2021-06-02 15:58:46 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 194.00 MiB (GPU 0; 15.78 GiB total capacity; 13.65 GiB already allocated; 110.75 MiB free; 14.04 GiB reserved in total by PyTorch)
2021-06-02 15:58:46 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:58:46 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:58:47 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8294.5
2021-06-02 15:58:50 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0008859835215844214
2021-06-02 15:58:52 dev-ssh-geza root[14429] INFO Iteration: 100/1184, epoch 0, Bif Loss: 0.0008859835215844214,  lr: 0.10000
2021-06-02 15:58:52 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8294.5
2021-06-02 15:58:55 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 352.00 MiB (GPU 0; 15.78 GiB total capacity; 13.43 GiB already allocated; 264.75 MiB free; 13.88 GiB reserved in total by PyTorch)
2021-06-02 15:58:55 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:58:55 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:58:55 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8294.5
2021-06-02 15:58:58 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 206.00 MiB (GPU 0; 15.78 GiB total capacity; 13.55 GiB already allocated; 102.75 MiB free; 14.04 GiB reserved in total by PyTorch)
2021-06-02 15:58:58 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:58:58 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:58:59 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8088.5
2021-06-02 15:59:02 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 212.00 MiB (GPU 0; 15.78 GiB total capacity; 13.43 GiB already allocated; 204.75 MiB free; 13.94 GiB reserved in total by PyTorch)
2021-06-02 15:59:02 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:59:02 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:59:02 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8088.5
2021-06-02 15:59:06 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0010611028410494328
2021-06-02 15:59:09 dev-ssh-geza root[14429] DEBUG Free GPU ram: 8088.5
2021-06-02 15:59:13 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 216.00 MiB (GPU 0; 15.78 GiB total capacity; 13.51 GiB already allocated; 12.75 MiB free; 14.13 GiB reserved in total by PyTorch)
2021-06-02 15:59:13 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:59:13 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:59:13 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7872.5
2021-06-02 15:59:18 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 202.00 MiB (GPU 0; 15.78 GiB total capacity; 13.72 GiB already allocated; 58.75 MiB free; 14.09 GiB reserved in total by PyTorch)
2021-06-02 15:59:18 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:59:18 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:59:19 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7872.5
2021-06-02 15:59:24 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 198.00 MiB (GPU 0; 15.78 GiB total capacity; 13.51 GiB already allocated; 142.75 MiB free; 14.00 GiB reserved in total by PyTorch)
2021-06-02 15:59:24 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:59:24 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:59:25 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7872.5
2021-06-02 15:59:30 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 202.00 MiB (GPU 0; 15.78 GiB total capacity; 13.47 GiB already allocated; 148.75 MiB free; 14.00 GiB reserved in total by PyTorch)
2021-06-02 15:59:30 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:59:30 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:59:30 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7670.5
2021-06-02 15:59:34 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0013539012288674712
2021-06-02 15:59:37 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7670.5
2021-06-02 15:59:40 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0007145553245209157
2021-06-02 15:59:42 dev-ssh-geza root[14429] INFO Iteration: 110/1184, epoch 0, Bif Loss: 0.0007145553245209157,  lr: 0.10000
2021-06-02 15:59:43 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7670.5
2021-06-02 15:59:48 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 358.00 MiB (GPU 0; 15.78 GiB total capacity; 13.51 GiB already allocated; 190.75 MiB free; 13.96 GiB reserved in total by PyTorch)
2021-06-02 15:59:48 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:59:48 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:59:49 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7670.5
2021-06-02 15:59:55 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 174.00 MiB (GPU 0; 15.78 GiB total capacity; 13.70 GiB already allocated; 116.75 MiB free; 14.03 GiB reserved in total by PyTorch)
2021-06-02 15:59:55 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:59:55 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:59:56 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7670.5
2021-06-02 15:59:58 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 212.00 MiB (GPU 0; 15.78 GiB total capacity; 13.60 GiB already allocated; 42.75 MiB free; 14.10 GiB reserved in total by PyTorch)
2021-06-02 15:59:58 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 15:59:58 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 15:59:59 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7670.5
2021-06-02 16:00:01 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 214.00 MiB (GPU 0; 15.78 GiB total capacity; 13.63 GiB already allocated; 56.75 MiB free; 14.09 GiB reserved in total by PyTorch)
2021-06-02 16:00:01 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:01 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:02 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7670.5
2021-06-02 16:00:04 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 224.00 MiB (GPU 0; 15.78 GiB total capacity; 13.60 GiB already allocated; 32.75 MiB free; 14.11 GiB reserved in total by PyTorch)
2021-06-02 16:00:04 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:04 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:05 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7334.5
2021-06-02 16:00:08 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 192.00 MiB (GPU 0; 15.78 GiB total capacity; 13.57 GiB already allocated; 112.75 MiB free; 14.03 GiB reserved in total by PyTorch)
2021-06-02 16:00:08 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:08 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:08 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7142.5
2021-06-02 16:00:11 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 184.00 MiB (GPU 0; 15.78 GiB total capacity; 13.55 GiB already allocated; 170.75 MiB free; 13.98 GiB reserved in total by PyTorch)
2021-06-02 16:00:11 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:11 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:11 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7142.5
2021-06-02 16:00:14 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 304.00 MiB (GPU 0; 15.78 GiB total capacity; 13.63 GiB already allocated; 96.75 MiB free; 14.05 GiB reserved in total by PyTorch)
2021-06-02 16:00:14 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:14 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:15 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7142.5
2021-06-02 16:00:18 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0011402334785088897
2021-06-02 16:00:20 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7142.5
2021-06-02 16:00:23 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 196.00 MiB (GPU 0; 15.78 GiB total capacity; 13.57 GiB already allocated; 46.75 MiB free; 14.10 GiB reserved in total by PyTorch)
2021-06-02 16:00:23 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:23 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:23 dev-ssh-geza root[14429] DEBUG Free GPU ram: 7142.5
2021-06-02 16:00:26 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 178.00 MiB (GPU 0; 15.78 GiB total capacity; 13.73 GiB already allocated; 54.75 MiB free; 14.09 GiB reserved in total by PyTorch)
2021-06-02 16:00:26 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:26 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:27 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6788.5
2021-06-02 16:00:30 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 344.00 MiB (GPU 0; 15.78 GiB total capacity; 13.28 GiB already allocated; 320.75 MiB free; 13.83 GiB reserved in total by PyTorch)
2021-06-02 16:00:30 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:30 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:30 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6788.5
2021-06-02 16:00:33 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 180.00 MiB (GPU 0; 15.78 GiB total capacity; 13.63 GiB already allocated; 56.75 MiB free; 14.09 GiB reserved in total by PyTorch)
2021-06-02 16:00:33 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:33 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:33 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6788.5
2021-06-02 16:00:36 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 190.00 MiB (GPU 0; 15.78 GiB total capacity; 13.56 GiB already allocated; 58.75 MiB free; 14.09 GiB reserved in total by PyTorch)
2021-06-02 16:00:36 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:36 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:37 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6598.5
2021-06-02 16:00:40 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 318.00 MiB (GPU 0; 15.78 GiB total capacity; 13.72 GiB already allocated; 26.75 MiB free; 14.12 GiB reserved in total by PyTorch)
2021-06-02 16:00:40 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:40 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:40 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6598.5
2021-06-02 16:00:43 dev-ssh-geza root[14429] DEBUG std::bad_alloc: cudaErrorMemoryAllocation: out of memory
2021-06-02 16:00:43 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:43 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:43 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6134.5
2021-06-02 16:00:46 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 182.00 MiB (GPU 0; 15.78 GiB total capacity; 13.70 GiB already allocated; 50.75 MiB free; 14.09 GiB reserved in total by PyTorch)
2021-06-02 16:00:46 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:46 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:47 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6134.5
2021-06-02 16:00:49 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 276.00 MiB (GPU 0; 15.78 GiB total capacity; 13.65 GiB already allocated; 78.75 MiB free; 14.07 GiB reserved in total by PyTorch)
2021-06-02 16:00:49 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:49 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:50 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6134.5
2021-06-02 16:00:53 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0016590757295489311
2021-06-02 16:00:55 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6136.5
2021-06-02 16:00:58 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 168.00 MiB (GPU 0; 15.78 GiB total capacity; 13.58 GiB already allocated; 132.75 MiB free; 14.01 GiB reserved in total by PyTorch)
2021-06-02 16:00:58 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:00:58 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:00:59 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6136.5
2021-06-02 16:01:02 dev-ssh-geza root[14429] DEBUG Batch loss: 0.0032221886795014143
2021-06-02 16:01:04 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6136.5
2021-06-02 16:01:07 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 312.00 MiB (GPU 0; 15.78 GiB total capacity; 13.56 GiB already allocated; 192.75 MiB free; 13.96 GiB reserved in total by PyTorch)
2021-06-02 16:01:07 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:01:07 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:01:07 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6136.5
2021-06-02 16:01:10 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 188.00 MiB (GPU 0; 15.78 GiB total capacity; 13.66 GiB already allocated; 8.75 MiB free; 14.13 GiB reserved in total by PyTorch)
2021-06-02 16:01:10 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:01:10 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:01:11 dev-ssh-geza root[14429] DEBUG Free GPU ram: 6136.5
2021-06-02 16:01:14 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 184.00 MiB (GPU 0; 15.78 GiB total capacity; 13.58 GiB already allocated; 102.75 MiB free; 14.04 GiB reserved in total by PyTorch)
2021-06-02 16:01:14 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:01:14 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:01:14 dev-ssh-geza root[14429] DEBUG Free GPU ram: 5860.5
2021-06-02 16:01:18 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 178.00 MiB (GPU 0; 15.78 GiB total capacity; 13.66 GiB already allocated; 60.75 MiB free; 14.08 GiB reserved in total by PyTorch)
2021-06-02 16:01:18 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:01:18 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:01:18 dev-ssh-geza root[14429] DEBUG Free GPU ram: 5682.5
2021-06-02 16:01:23 dev-ssh-geza root[14429] DEBUG CUDA out of memory. Tried to allocate 142.00 MiB (GPU 0; 15.78 GiB total capacity; 13.78 GiB already allocated; 8.75 MiB free; 14.13 GiB reserved in total by PyTorch)
2021-06-02 16:01:23 dev-ssh-geza root[14429] DEBUG Traceback (most recent call last):
2021-06-02 16:01:23 dev-ssh-geza root[14429] WARNING CUDA OOM Error during batch execution, skipping batch.
2021-06-02 16:01:23 dev-ssh-geza root[14429] DEBUG Free GPU ram: 5682.5

Expected behavior

Free VRAM should remain stable after first few warm-up iterations.

==========System==========
Linux-5.4.0-1047-aws-x86_64-with-glibc2.10
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
3.8.5 (default, Sep  4 2020, 07:30:14) 
[GCC 7.3.0]
==========Pytorch==========
1.7.0a0+57bffc3
torch.cuda.is_available(): True
==========NVIDIA-SMI==========
/usr/bin/nvidia-smi
Driver Version 460.73.01
CUDA Version 11.2
VBIOS Version 88.00.4F.00.09
Image Version G503.0201.00.03
==========NVCC==========
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0
==========CC==========
/usr/bin/c++
c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

==========MinkowskiEngine==========
0.5.4
MinkowskiEngine compiled with CUDA Support: True
NVCC version MinkowskiEngine is compiled: 11000
CUDART version MinkowskiEngine is compiled: 11000

Jun 02 '21 16:06 evelkey

Hi @chrischoy I've managed to put together a minimal repro code (works on both RTX2070 8GB and Tesla V100 16GB).

import gc
import traceback

import MinkowskiEngine as ME
import torch
import torch.nn as nn
from MinkowskiEngine import SparseTensor

# from pytorch_memlab import MemReporter


class TestNet(ME.MinkowskiNetwork):
    def __init__(self, in_feat, out_feat, D, layers=80):
        super(TestNet, self).__init__(D)
        convs = [out_feat for _ in range(layers)]
        self.convs = []
        prev = in_feat
        for outchannels in convs:
            layer = nn.Sequential(
                ME.MinkowskiConvolution(
                    in_channels=prev,
                    out_channels=outchannels,
                    kernel_size=3,
                    stride=2,
                    dilation=1,
                    bias=True,
                    dimension=D,
                ),
                ME.MinkowskiReLU(),
            )
            self.convs.append(layer)
            prev = outchannels

    def forward(self, x):
        temp = x
        for convlayer in self.convs:
            temp = convlayer(temp)
            torch.cuda.synchronize()
        return temp

    def cuda(self):
        super(TestNet, self).cuda()
        self.convs = [c.cuda() for c in self.convs]
        return self


point_count = 6000000
in_channels, out_channels, D = 2, 3, 3


testnetwork = TestNet(in_channels, 50, 3).cuda()

# reporter = MemReporter()

for i in range(100):
    points = point_count + i * 100000
    print(f"starting {i}")
    print(f"Points: {points}")
    oom = False

    try:
        coords, feats = (
            torch.randint(low=-1200, high=1200, size=(points, D + 1)).int(),
            torch.rand(size=(points, in_channels)),
        )
        coords[:, 0] = 0
        xt = SparseTensor(feats.cuda(), coordinates=coords.cuda(), device="cuda")
        torch.cuda.synchronize()
        print("run forward")
        res = testnetwork(xt)

        torch.cuda.synchronize()
        print("run backward")
        loss = res.F.sum()
        loss.backward()

    except Exception as e:
        print(f"OOM occurred: {e} -> {traceback.format_exc()}")
        oom = True

    try:
        del xt
        del res
        del loss
    except NameError:
        pass
    gc.collect()
    torch.cuda.empty_cache()

    print(torch.cuda.list_gpu_processes())
    print(torch.cuda.memory_stats())

    cuda_total_memory = torch.cuda.get_device_properties(torch.cuda.current_device).total_memory
    cuda_cached_memory = torch.cuda.memory_reserved(torch.cuda.current_device)
    available_cuda_memory_in_mbytes = (cuda_total_memory - cuda_cached_memory) / 1024 ** 2
    print(f"\nFree GPU vram: {available_cuda_memory_in_mbytes} MB\n{'*'*80}\n")

It simply grows the pointcloud to a limit where it starts to OOM. In some cases (see the logs below) a tensor escapes GC and remains in the CUDA memory and it cannot be removed from there. It is now reproducible in all cases.

Logs (long)

starting 0
Points: 6000000
run forward
run backward
GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 1141), ('active.all.current', 320), ('active.all.freed', 821), ('active.all.peak', 674), ('active.large_pool.allocated', 150), ('active.large_pool.current', 0), ('active.large_pool.freed', 150), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 991), ('active.small_pool.current', 320), ('active.small_pool.freed', 671), ('active.small_pool.peak', 614), ('active_bytes.all.allocated', 30046140928), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 30003323392), ('active_bytes.all.peak', 13442540032), ('active_bytes.large_pool.allocated', 29993401856), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 29993401856), ('active_bytes.large_pool.peak', 13395151360), ('active_bytes.small_pool.allocated', 52739072), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 9921536), ('active_bytes.small_pool.peak', 47942144), ('allocated_bytes.all.allocated', 30046140928), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 30003323392), ('allocated_bytes.all.peak', 13442540032), ('allocated_bytes.large_pool.allocated', 29993401856), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 29993401856), ('allocated_bytes.large_pool.peak', 13395151360), ('allocated_bytes.small_pool.allocated', 52739072), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 9921536), ('allocated_bytes.small_pool.peak', 47942144), ('allocation.all.allocated', 1141), ('allocation.all.current', 320), ('allocation.all.freed', 821), ('allocation.all.peak', 674), ('allocation.large_pool.allocated', 150), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 150), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 991), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 671), ('allocation.small_pool.peak', 614), ('inactive_split.all.allocated', 449), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 369), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 67), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 67), ('inactive_split.large_pool.peak', 19), ('inactive_split.small_pool.allocated', 382), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 302), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 12512323072), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 12504808960), ('inactive_split_bytes.all.peak', 5412354560), ('inactive_split_bytes.large_pool.allocated', 12455651840), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 12455651840), ('inactive_split_bytes.large_pool.peak', 5407806976), ('inactive_split_bytes.small_pool.allocated', 56671232), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 49157120), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 2), ('num_ooms', 0), ('reserved_bytes.all.allocated', 20298334208), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 20248002560), ('reserved_bytes.all.peak', 15657336832), ('reserved_bytes.large_pool.allocated', 20245905408), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 20245905408), ('reserved_bytes.large_pool.peak', 15604908032), ('reserved_bytes.small_pool.allocated', 52428800), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 2097152), ('reserved_bytes.small_pool.peak', 52428800), ('segment.all.allocated', 61), ('segment.all.current', 24), ('segment.all.freed', 37), ('segment.all.peak', 57), ('segment.large_pool.allocated', 36), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 36), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 25), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 1), ('segment.small_pool.peak', 25)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 1
Points: 6100000
run forward
run backward
GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 2073), ('active.all.current', 320), ('active.all.freed', 1753), ('active.all.peak', 753), ('active.large_pool.allocated', 300), ('active.large_pool.current', 0), ('active.large_pool.freed', 300), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 1773), ('active.small_pool.current', 320), ('active.small_pool.freed', 1453), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 60511686656), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 60468869120), ('active_bytes.all.peak', 13646366720), ('active_bytes.large_pool.allocated', 60427642368), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 60427642368), ('active_bytes.large_pool.peak', 13598424576), ('active_bytes.small_pool.allocated', 84044288), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 41226752), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 60511686656), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 60468869120), ('allocated_bytes.all.peak', 13646366720), ('allocated_bytes.large_pool.allocated', 60427642368), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 60427642368), ('allocated_bytes.large_pool.peak', 13598424576), ('allocated_bytes.small_pool.allocated', 84044288), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 41226752), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 2073), ('allocation.all.current', 320), ('allocation.all.freed', 1753), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 300), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 300), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 1773), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 1453), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 928), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 848), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 119), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 119), ('inactive_split.large_pool.peak', 19), ('inactive_split.small_pool.allocated', 809), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 729), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 24619601920), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 24612087808), ('inactive_split_bytes.all.peak', 5412354560), ('inactive_split_bytes.large_pool.allocated', 24530165760), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 24530165760), ('inactive_split_bytes.large_pool.peak', 5407806976), ('inactive_split_bytes.small_pool.allocated', 89436160), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 81922048), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 4), ('num_ooms', 0), ('reserved_bytes.all.allocated', 40837840896), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 40787509248), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 40781217792), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 40781217792), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 56623104), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 6291456), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 99), ('segment.all.current', 24), ('segment.all.freed', 75), ('segment.all.peak', 58), ('segment.large_pool.allocated', 72), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 72), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 27), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 3), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 2
Points: 6200000
run forward
run backward
GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 3005), ('active.all.current', 320), ('active.all.freed', 2685), ('active.all.peak', 753), ('active.large_pool.allocated', 450), ('active.large_pool.current', 0), ('active.large_pool.freed', 450), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 2555), ('active.small_pool.current', 320), ('active.small_pool.freed', 2235), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 91400537088), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 91357719552), ('active_bytes.all.peak', 13847045632), ('active_bytes.large_pool.allocated', 91285187584), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 91285187584), ('active_bytes.large_pool.peak', 13799103488), ('active_bytes.small_pool.allocated', 115349504), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 72531968), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 91400537088), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 91357719552), ('allocated_bytes.all.peak', 13847045632), ('allocated_bytes.large_pool.allocated', 91285187584), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 91285187584), ('allocated_bytes.large_pool.peak', 13799103488), ('allocated_bytes.small_pool.allocated', 115349504), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 72531968), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 3005), ('allocation.all.current', 320), ('allocation.all.freed', 2685), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 450), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 450), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 2555), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 2235), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 1408), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 1328), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 172), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 172), ('inactive_split.large_pool.peak', 19), ('inactive_split.small_pool.allocated', 1236), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 1156), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 36290313216), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 36282799104), ('inactive_split_bytes.all.peak', 5412354560), ('inactive_split_bytes.large_pool.allocated', 36168112128), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 36168112128), ('inactive_split_bytes.large_pool.peak', 5407806976), ('inactive_split_bytes.small_pool.allocated', 122201088), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 114686976), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 8), ('num_ooms', 0), ('reserved_bytes.all.allocated', 61668851712), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 61618520064), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 61608034304), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 61608034304), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 60817408), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 10485760), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 137), ('segment.all.current', 24), ('segment.all.freed', 113), ('segment.all.peak', 58), ('segment.large_pool.allocated', 108), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 108), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 29), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 5), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 3
Points: 6300000
run forward
run backward
GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 3937), ('active.all.current', 320), ('active.all.freed', 3617), ('active.all.peak', 753), ('active.large_pool.allocated', 600), ('active.large_pool.current', 0), ('active.large_pool.freed', 600), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 3337), ('active.small_pool.current', 320), ('active.small_pool.freed', 3017), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 122708960768), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 122666143232), ('active_bytes.all.peak', 14041238016), ('active_bytes.large_pool.allocated', 122562306048), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 122562306048), ('active_bytes.large_pool.peak', 13993295872), ('active_bytes.small_pool.allocated', 146654720), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 103837184), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 122708960768), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 122666143232), ('allocated_bytes.all.peak', 14041238016), ('allocated_bytes.large_pool.allocated', 122562306048), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 122562306048), ('allocated_bytes.large_pool.peak', 13993295872), ('allocated_bytes.small_pool.allocated', 146654720), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 103837184), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 3937), ('allocation.all.current', 320), ('allocation.all.freed', 3617), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 600), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 600), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 3337), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 3017), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 1910), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 1830), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 247), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 247), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 1663), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 1583), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 49443032576), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 49435518464), ('inactive_split_bytes.all.peak', 6114472960), ('inactive_split_bytes.large_pool.allocated', 49288066560), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 49288066560), ('inactive_split_bytes.large_pool.peak', 6109643776), ('inactive_split_bytes.small_pool.allocated', 154966016), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 147451904), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 13), ('num_ooms', 0), ('reserved_bytes.all.allocated', 82816532480), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 82766200832), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 82751520768), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 82751520768), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 65011712), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 14680064), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 175), ('segment.all.current', 24), ('segment.all.freed', 151), ('segment.all.peak', 58), ('segment.large_pool.allocated', 144), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 144), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 31), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 7), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 4
Points: 6400000
run forward
run backward
GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 4869), ('active.all.current', 320), ('active.all.freed', 4549), ('active.all.peak', 753), ('active.large_pool.allocated', 750), ('active.large_pool.current', 0), ('active.large_pool.freed', 750), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 4119), ('active.small_pool.current', 320), ('active.small_pool.freed', 3799), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 154449377280), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 154406559744), ('active_bytes.all.peak', 14244352000), ('active_bytes.large_pool.allocated', 154271417344), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 154271417344), ('active_bytes.large_pool.peak', 14196409856), ('active_bytes.small_pool.allocated', 177959936), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 135142400), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 154449377280), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 154406559744), ('allocated_bytes.all.peak', 14244352000), ('allocated_bytes.large_pool.allocated', 154271417344), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 154271417344), ('allocated_bytes.large_pool.peak', 14196409856), ('allocated_bytes.small_pool.allocated', 177959936), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 135142400), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 4869), ('allocation.all.current', 320), ('allocation.all.freed', 4549), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 750), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 750), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 4119), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 3799), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 2417), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 2337), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 327), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 327), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 2090), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 2010), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 61566077440), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 61558563328), ('inactive_split_bytes.all.peak', 6114472960), ('inactive_split_bytes.large_pool.allocated', 61378346496), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 61378346496), ('inactive_split_bytes.large_pool.peak', 6109643776), ('inactive_split_bytes.small_pool.allocated', 187730944), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 180216832), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 18), ('num_ooms', 0), ('reserved_bytes.all.allocated', 104159248384), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 104108916736), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 104090042368), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 104090042368), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 69206016), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 18874368), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 212), ('segment.all.current', 24), ('segment.all.freed', 188), ('segment.all.peak', 58), ('segment.large_pool.allocated', 179), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 179), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 33), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 9), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 5
Points: 6500000
run forward
run backward
GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 5801), ('active.all.current', 320), ('active.all.freed', 5481), ('active.all.peak', 753), ('active.large_pool.allocated', 900), ('active.large_pool.current', 0), ('active.large_pool.freed', 900), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 4901), ('active.small_pool.current', 320), ('active.small_pool.freed', 4581), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 186631837184), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 186589019648), ('active_bytes.all.peak', 14451425792), ('active_bytes.large_pool.allocated', 186422572032), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 186422572032), ('active_bytes.large_pool.peak', 14403483648), ('active_bytes.small_pool.allocated', 209265152), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 166447616), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 186631837184), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 186589019648), ('allocated_bytes.all.peak', 14451425792), ('allocated_bytes.large_pool.allocated', 186422572032), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 186422572032), ('allocated_bytes.large_pool.peak', 14403483648), ('allocated_bytes.small_pool.allocated', 209265152), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 166447616), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 5801), ('allocation.all.current', 320), ('allocation.all.freed', 5481), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 900), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 900), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 4901), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 4581), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 2903), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 2823), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 386), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 386), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 2517), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 2437), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 74396853248), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 74389339136), ('inactive_split_bytes.all.peak', 6114472960), ('inactive_split_bytes.large_pool.allocated', 74176357376), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 74176357376), ('inactive_split_bytes.large_pool.peak', 6109643776), ('inactive_split_bytes.small_pool.allocated', 220495872), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 212981760), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 23), ('num_ooms', 0), ('reserved_bytes.all.allocated', 125785079808), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 125734748160), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 125711679488), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 125711679488), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 73400320), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 23068672), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 249), ('segment.all.current', 24), ('segment.all.freed', 225), ('segment.all.peak', 58), ('segment.large_pool.allocated', 214), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 214), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 35), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 11), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 6
Points: 6600000
run forward
run backward
GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 6733), ('active.all.current', 320), ('active.all.freed', 6413), ('active.all.peak', 753), ('active.large_pool.allocated', 1050), ('active.large_pool.current', 0), ('active.large_pool.freed', 1050), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 5683), ('active.small_pool.current', 320), ('active.small_pool.freed', 5363), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 219235617280), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 219192799744), ('active_bytes.all.peak', 14644858880), ('active_bytes.large_pool.allocated', 218995046912), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 218995046912), ('active_bytes.large_pool.peak', 14596916736), ('active_bytes.small_pool.allocated', 240570368), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 197752832), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 219235617280), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 219192799744), ('allocated_bytes.all.peak', 14644858880), ('allocated_bytes.large_pool.allocated', 218995046912), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 218995046912), ('allocated_bytes.large_pool.peak', 14596916736), ('allocated_bytes.small_pool.allocated', 240570368), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 197752832), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 6733), ('allocation.all.current', 320), ('allocation.all.freed', 6413), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 1050), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 1050), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 5683), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 5363), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 3390), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 3310), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 446), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 446), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 2944), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 2864), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 86852259328), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 86844745216), ('inactive_split_bytes.all.peak', 6114472960), ('inactive_split_bytes.large_pool.allocated', 86598998528), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 86598998528), ('inactive_split_bytes.large_pool.peak', 6109643776), ('inactive_split_bytes.small_pool.allocated', 253260800), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 245746688), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 28), ('num_ooms', 0), ('reserved_bytes.all.allocated', 147706609664), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 147656278016), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 147629015040), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 147629015040), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 77594624), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 27262976), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 286), ('segment.all.current', 24), ('segment.all.freed', 262), ('segment.all.peak', 58), ('segment.large_pool.allocated', 249), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 249), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 37), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 13), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 7
Points: 6700000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 1.23 GiB (GPU 0; 15.78 GiB total capacity; 11.74 GiB already allocated; 1.22 GiB free; 13.62 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 1.23 GiB (GPU 0; 15.78 GiB total capacity; 11.74 GiB already allocated; 1.22 GiB free; 13.62 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 7651), ('active.all.current', 320), ('active.all.freed', 7331), ('active.all.peak', 753), ('active.large_pool.allocated', 1192), ('active.large_pool.current', 0), ('active.large_pool.freed', 1192), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 6459), ('active.small_pool.current', 320), ('active.small_pool.freed', 6139), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 247867228672), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 247824411136), ('active_bytes.all.peak', 14850433024), ('active_bytes.large_pool.allocated', 247595636736), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 247595636736), ('active_bytes.large_pool.peak', 14802490880), ('active_bytes.small_pool.allocated', 271591936), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 228774400), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 247867228672), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 247824411136), ('allocated_bytes.all.peak', 14850433024), ('allocated_bytes.large_pool.allocated', 247595636736), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 247595636736), ('allocated_bytes.large_pool.peak', 14802490880), ('allocated_bytes.small_pool.allocated', 271591936), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 228774400), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 7651), ('allocation.all.current', 320), ('allocation.all.freed', 7331), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 1192), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 1192), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 6459), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 6139), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 3864), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 3784), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 497), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 497), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 3367), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 3287), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 99089904640), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 99082390528), ('inactive_split_bytes.all.peak', 6114472960), ('inactive_split_bytes.large_pool.allocated', 98804162560), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 98804162560), ('inactive_split_bytes.large_pool.peak', 6109643776), ('inactive_split_bytes.small_pool.allocated', 285742080), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 278227968), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 31), ('num_ooms', 1), ('reserved_bytes.all.allocated', 165914083328), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 165863751680), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 165832294400), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 165832294400), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 81788928), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 31457280), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 320), ('segment.all.current', 24), ('segment.all.freed', 296), ('segment.all.peak', 58), ('segment.large_pool.allocated', 281), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 281), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 39), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 15), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 8
Points: 6800000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 1.12 GiB (GPU 0; 15.78 GiB total capacity; 12.89 GiB already allocated; 1.12 GiB free; 13.73 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 1.12 GiB (GPU 0; 15.78 GiB total capacity; 12.89 GiB already allocated; 1.12 GiB free; 13.73 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 8561), ('active.all.current', 320), ('active.all.freed', 8241), ('active.all.peak', 753), ('active.large_pool.allocated', 1329), ('active.large_pool.current', 0), ('active.large_pool.freed', 1329), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 7232), ('active.small_pool.current', 320), ('active.small_pool.freed', 6912), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 273959147520), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 273916329984), ('active_bytes.all.peak', 14850433024), ('active_bytes.large_pool.allocated', 273656805376), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 273656805376), ('active_bytes.large_pool.peak', 14802490880), ('active_bytes.small_pool.allocated', 302342144), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 259524608), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 273959147520), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 273916329984), ('allocated_bytes.all.peak', 14850433024), ('allocated_bytes.large_pool.allocated', 273656805376), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 273656805376), ('allocated_bytes.large_pool.peak', 14802490880), ('allocated_bytes.small_pool.allocated', 302342144), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 259524608), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 8561), ('allocation.all.current', 320), ('allocation.all.freed', 8241), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 1329), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 1329), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 7232), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 6912), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 4339), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 4259), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 551), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 551), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 3788), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 3708), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 112078152704), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 112070638592), ('inactive_split_bytes.all.peak', 6155523584), ('inactive_split_bytes.large_pool.allocated', 111760200704), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 111760200704), ('inactive_split_bytes.large_pool.peak', 6150976000), ('inactive_split_bytes.small_pool.allocated', 317952000), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 310437888), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 33), ('num_ooms', 2), ('reserved_bytes.all.allocated', 181865021440), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 181814689792), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 181779038208), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 181779038208), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 85983232), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 35651584), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 354), ('segment.all.current', 24), ('segment.all.freed', 330), ('segment.all.peak', 58), ('segment.large_pool.allocated', 313), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 313), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 41), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 17), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 9
Points: 6900000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 1.13 GiB (GPU 0; 15.78 GiB total capacity; 13.06 GiB already allocated; 1008.75 MiB free; 13.86 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 1.13 GiB (GPU 0; 15.78 GiB total capacity; 13.06 GiB already allocated; 1008.75 MiB free; 13.86 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 9471), ('active.all.current', 320), ('active.all.freed', 9151), ('active.all.peak', 753), ('active.large_pool.allocated', 1466), ('active.large_pool.current', 0), ('active.large_pool.freed', 1466), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 8005), ('active.small_pool.current', 320), ('active.small_pool.freed', 7685), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 300374056960), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 300331239424), ('active_bytes.all.peak', 14920765440), ('active_bytes.large_pool.allocated', 300040964608), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 300040964608), ('active_bytes.large_pool.peak', 14872552960), ('active_bytes.small_pool.allocated', 333092352), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 290274816), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 300374056960), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 300331239424), ('allocated_bytes.all.peak', 14920765440), ('allocated_bytes.large_pool.allocated', 300040964608), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 300040964608), ('allocated_bytes.large_pool.peak', 14872552960), ('allocated_bytes.small_pool.allocated', 333092352), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 290274816), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 9471), ('allocation.all.current', 320), ('allocation.all.freed', 9151), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 1466), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 1466), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 8005), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 7685), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 4827), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 4747), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 618), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 618), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 4209), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 4129), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 123876127232), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 123868613120), ('inactive_split_bytes.all.peak', 6155523584), ('inactive_split_bytes.large_pool.allocated', 123525965312), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 123525965312), ('inactive_split_bytes.large_pool.peak', 6150976000), ('inactive_split_bytes.small_pool.allocated', 350161920), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 342647808), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 35), ('num_ooms', 3), ('reserved_bytes.all.allocated', 197966954496), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 197916622848), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 197876776960), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 197876776960), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 90177536), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 39845888), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 386), ('segment.all.current', 24), ('segment.all.freed', 362), ('segment.all.peak', 58), ('segment.large_pool.allocated', 343), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 343), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 43), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 19), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 10
Points: 7000000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 1.15 GiB (GPU 0; 15.78 GiB total capacity; 13.23 GiB already allocated; 828.75 MiB free; 14.04 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 1.15 GiB (GPU 0; 15.78 GiB total capacity; 13.23 GiB already allocated; 828.75 MiB free; 14.04 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 10381), ('active.all.current', 320), ('active.all.freed', 10061), ('active.all.peak', 753), ('active.large_pool.allocated', 1603), ('active.large_pool.current', 0), ('active.large_pool.freed', 1603), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 8778), ('active.small_pool.current', 320), ('active.small_pool.freed', 8458), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 327102324224), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 327059506688), ('active_bytes.all.peak', 15106243072), ('active_bytes.large_pool.allocated', 326738481664), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 326738481664), ('active_bytes.large_pool.peak', 15058030592), ('active_bytes.small_pool.allocated', 363842560), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 321025024), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 327102324224), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 327059506688), ('allocated_bytes.all.peak', 15106243072), ('allocated_bytes.large_pool.allocated', 326738481664), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 326738481664), ('allocated_bytes.large_pool.peak', 15058030592), ('allocated_bytes.small_pool.allocated', 363842560), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 321025024), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 10381), ('allocation.all.current', 320), ('allocation.all.freed', 10061), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 1603), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 1603), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 8778), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 8458), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 5318), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 5238), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 688), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 688), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 4630), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 4550), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 137041595392), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 137034081280), ('inactive_split_bytes.all.peak', 6199079936), ('inactive_split_bytes.large_pool.allocated', 136659223552), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 136659223552), ('inactive_split_bytes.large_pool.peak', 6194532352), ('inactive_split_bytes.small_pool.allocated', 382371840), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 374857728), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 37), ('num_ooms', 4), ('reserved_bytes.all.allocated', 214261825536), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 214211493888), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 214167453696), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 214167453696), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 94371840), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 44040192), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 418), ('segment.all.current', 24), ('segment.all.freed', 394), ('segment.all.peak', 58), ('segment.large_pool.allocated', 373), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 373), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 45), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 21), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 11
Points: 7100000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 1.16 GiB (GPU 0; 15.78 GiB total capacity; 13.40 GiB already allocated; 646.75 MiB free; 14.21 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 1.16 GiB (GPU 0; 15.78 GiB total capacity; 13.40 GiB already allocated; 646.75 MiB free; 14.21 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 11291), ('active.all.current', 320), ('active.all.freed', 10971), ('active.all.peak', 753), ('active.large_pool.allocated', 1740), ('active.large_pool.current', 0), ('active.large_pool.freed', 1740), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 9551), ('active.small_pool.current', 320), ('active.small_pool.freed', 9231), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 354159954432), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 354117136896), ('active_bytes.all.peak', 15298292736), ('active_bytes.large_pool.allocated', 353765361664), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 353765361664), ('active_bytes.large_pool.peak', 15250080256), ('active_bytes.small_pool.allocated', 394592768), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 351775232), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 354159954432), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 354117136896), ('allocated_bytes.all.peak', 15298292736), ('allocated_bytes.large_pool.allocated', 353765361664), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 353765361664), ('allocated_bytes.large_pool.peak', 15250080256), ('allocated_bytes.small_pool.allocated', 394592768), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 351775232), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 11291), ('allocation.all.current', 320), ('allocation.all.freed', 10971), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 1740), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 1740), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 9551), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 9231), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 5792), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 5712), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 741), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 741), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 5051), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 4971), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 149207909888), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 149200395776), ('inactive_split_bytes.all.peak', 6199079936), ('inactive_split_bytes.large_pool.allocated', 148793328128), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 148793328128), ('inactive_split_bytes.large_pool.peak', 6194532352), ('inactive_split_bytes.small_pool.allocated', 414581760), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 407067648), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 39), ('num_ooms', 5), ('reserved_bytes.all.allocated', 230751731712), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 230701400064), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 230653165568), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 230653165568), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 98566144), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 48234496), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 450), ('segment.all.current', 24), ('segment.all.freed', 426), ('segment.all.peak', 58), ('segment.large_pool.allocated', 403), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 403), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 47), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 23), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 12
Points: 7200000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 1.18 GiB (GPU 0; 15.78 GiB total capacity; 12.95 GiB already allocated; 1.09 GiB free; 13.76 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/function.py", line 89, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 99, in backward
    grad_in_feat, grad_kernel = bw_fn(
RuntimeError: CUDA out of memory. Tried to allocate 1.18 GiB (GPU 0; 15.78 GiB total capacity; 12.95 GiB already allocated; 1.09 GiB free; 13.76 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 12197), ('active.all.current', 320), ('active.all.freed', 11877), ('active.all.peak', 753), ('active.large_pool.allocated', 1874), ('active.large_pool.current', 0), ('active.large_pool.freed', 1874), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 10323), ('active.small_pool.current', 320), ('active.small_pool.freed', 10003), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 379954420224), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 379911602688), ('active_bytes.all.peak', 15298292736), ('active_bytes.large_pool.allocated', 379529347584), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 379529347584), ('active_bytes.large_pool.peak', 15250080256), ('active_bytes.small_pool.allocated', 425072640), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 382255104), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 379954420224), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 379911602688), ('allocated_bytes.all.peak', 15298292736), ('allocated_bytes.large_pool.allocated', 379529347584), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 379529347584), ('allocated_bytes.large_pool.peak', 15250080256), ('allocated_bytes.small_pool.allocated', 425072640), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 382255104), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 12197), ('allocation.all.current', 320), ('allocation.all.freed', 11877), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 1874), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 1874), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 10323), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 10003), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 6255), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 6175), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 783), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 783), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 5472), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 5392), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 161074206720), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 161066692608), ('inactive_split_bytes.all.peak', 6199079936), ('inactive_split_bytes.large_pool.allocated', 160627685376), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 160627685376), ('inactive_split_bytes.large_pool.peak', 6194532352), ('inactive_split_bytes.small_pool.allocated', 446521344), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 439007232), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 40), ('num_ooms', 6), ('reserved_bytes.all.allocated', 246159507456), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 246109175808), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 246056747008), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 246056747008), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 102760448), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 52428800), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 481), ('segment.all.current', 24), ('segment.all.freed', 457), ('segment.all.peak', 58), ('segment.large_pool.allocated', 432), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 432), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 49), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 25), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 13
Points: 7300000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 1.19 GiB (GPU 0; 15.78 GiB total capacity; 13.11 GiB already allocated; 914.75 MiB free; 13.95 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/function.py", line 89, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 99, in backward
    grad_in_feat, grad_kernel = bw_fn(
RuntimeError: CUDA out of memory. Tried to allocate 1.19 GiB (GPU 0; 15.78 GiB total capacity; 13.11 GiB already allocated; 914.75 MiB free; 13.95 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 13103), ('active.all.current', 320), ('active.all.freed', 12783), ('active.all.peak', 753), ('active.large_pool.allocated', 2008), ('active.large_pool.current', 0), ('active.large_pool.freed', 2008), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 11095), ('active.small_pool.current', 320), ('active.small_pool.freed', 10775), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 406038187520), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 405995369984), ('active_bytes.all.peak', 15298292736), ('active_bytes.large_pool.allocated', 405582635008), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 405582635008), ('active_bytes.large_pool.peak', 15250080256), ('active_bytes.small_pool.allocated', 455552512), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 412734976), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 406038187520), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 405995369984), ('allocated_bytes.all.peak', 15298292736), ('allocated_bytes.large_pool.allocated', 405582635008), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 405582635008), ('allocated_bytes.large_pool.peak', 15250080256), ('allocated_bytes.small_pool.allocated', 455552512), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 412734976), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 13103), ('allocation.all.current', 320), ('allocation.all.freed', 12783), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 2008), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 2008), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 11095), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 10775), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 6721), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 6641), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 828), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 828), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 5893), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 5813), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 173126571008), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 173119056896), ('inactive_split_bytes.all.peak', 6199079936), ('inactive_split_bytes.large_pool.allocated', 172648110080), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 172648110080), ('inactive_split_bytes.large_pool.peak', 6194532352), ('inactive_split_bytes.small_pool.allocated', 478460928), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 470946816), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 41), ('num_ooms', 7), ('reserved_bytes.all.allocated', 261779095552), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 261728763904), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 261672140800), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 261672140800), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 106954752), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 56623104), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 513), ('segment.all.current', 24), ('segment.all.freed', 489), ('segment.all.peak', 58), ('segment.large_pool.allocated', 462), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 462), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 51), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 27), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 14
Points: 7400000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 1.21 GiB (GPU 0; 15.78 GiB total capacity; 13.26 GiB already allocated; 766.75 MiB free; 14.10 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/function.py", line 89, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 99, in backward
    grad_in_feat, grad_kernel = bw_fn(
RuntimeError: CUDA out of memory. Tried to allocate 1.21 GiB (GPU 0; 15.78 GiB total capacity; 13.26 GiB already allocated; 766.75 MiB free; 14.10 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 14009), ('active.all.current', 320), ('active.all.freed', 13689), ('active.all.peak', 753), ('active.large_pool.allocated', 2142), ('active.large_pool.current', 0), ('active.large_pool.freed', 2142), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 11867), ('active.small_pool.current', 320), ('active.small_pool.freed', 11547), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 432408520192), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 432365702656), ('active_bytes.all.peak', 15437549568), ('active_bytes.large_pool.allocated', 431922487808), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 431922487808), ('active_bytes.large_pool.peak', 15389607424), ('active_bytes.small_pool.allocated', 486032384), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 443214848), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 432408520192), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 432365702656), ('allocated_bytes.all.peak', 15437549568), ('allocated_bytes.large_pool.allocated', 431922487808), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 431922487808), ('allocated_bytes.large_pool.peak', 15389607424), ('allocated_bytes.small_pool.allocated', 486032384), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 443214848), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 14009), ('allocation.all.current', 320), ('allocation.all.freed', 13689), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 2142), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 2142), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 11867), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 11547), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 7209), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 7129), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 895), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 895), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 6314), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 6234), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 185358830592), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 185351316480), ('inactive_split_bytes.all.peak', 6199079936), ('inactive_split_bytes.large_pool.allocated', 184848430080), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 184848430080), ('inactive_split_bytes.large_pool.peak', 6194532352), ('inactive_split_bytes.small_pool.allocated', 510400512), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 502886400), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 42), ('num_ooms', 8), ('reserved_bytes.all.allocated', 277555970048), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 277505638400), ('reserved_bytes.all.peak', 15890120704), ('reserved_bytes.large_pool.allocated', 277444820992), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 277444820992), ('reserved_bytes.large_pool.peak', 15837691904), ('reserved_bytes.small_pool.allocated', 111149056), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 60817408), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 544), ('segment.all.current', 24), ('segment.all.freed', 520), ('segment.all.peak', 58), ('segment.large_pool.allocated', 491), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 491), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 53), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 29), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 15
Points: 7500000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 1.22 GiB (GPU 0; 15.78 GiB total capacity; 13.42 GiB already allocated; 608.75 MiB free; 14.25 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/function.py", line 89, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 99, in backward
    grad_in_feat, grad_kernel = bw_fn(
RuntimeError: CUDA out of memory. Tried to allocate 1.22 GiB (GPU 0; 15.78 GiB total capacity; 13.42 GiB already allocated; 608.75 MiB free; 14.25 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 14915), ('active.all.current', 320), ('active.all.freed', 14595), ('active.all.peak', 753), ('active.large_pool.allocated', 2276), ('active.large_pool.current', 0), ('active.large_pool.freed', 2276), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 12639), ('active.small_pool.current', 320), ('active.small_pool.freed', 12319), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 459077766144), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 459034948608), ('active_bytes.all.peak', 15613830144), ('active_bytes.large_pool.allocated', 458561253888), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 458561253888), ('active_bytes.large_pool.peak', 15565888000), ('active_bytes.small_pool.allocated', 516512256), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 473694720), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 459077766144), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 459034948608), ('allocated_bytes.all.peak', 15613830144), ('allocated_bytes.large_pool.allocated', 458561253888), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 458561253888), ('allocated_bytes.large_pool.peak', 15565888000), ('allocated_bytes.small_pool.allocated', 516512256), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 473694720), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 14915), ('allocation.all.current', 320), ('allocation.all.freed', 14595), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 2276), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 2276), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 12639), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 12319), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 7693), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 7613), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 958), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 958), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 6735), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 6655), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 197647358976), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 197639844864), ('inactive_split_bytes.all.peak', 6199079936), ('inactive_split_bytes.large_pool.allocated', 197105018880), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 197105018880), ('inactive_split_bytes.large_pool.peak', 6194532352), ('inactive_split_bytes.small_pool.allocated', 542340096), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 534825984), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 44), ('num_ooms', 9), ('reserved_bytes.all.allocated', 293500616704), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 293450285056), ('reserved_bytes.all.peak', 15904800768), ('reserved_bytes.large_pool.allocated', 293385273344), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 293385273344), ('reserved_bytes.large_pool.peak', 15852371968), ('reserved_bytes.small_pool.allocated', 115343360), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 65011712), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 575), ('segment.all.current', 24), ('segment.all.freed', 551), ('segment.all.peak', 58), ('segment.large_pool.allocated', 520), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 520), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 55), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 31), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 16
Points: 7600000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 578.00 MiB (GPU 0; 15.78 GiB total capacity; 14.14 GiB already allocated; 446.75 MiB free; 14.41 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 578.00 MiB (GPU 0; 15.78 GiB total capacity; 14.14 GiB already allocated; 446.75 MiB free; 14.41 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 15817), ('active.all.current', 320), ('active.all.freed', 15497), ('active.all.peak', 753), ('active.large_pool.allocated', 2408), ('active.large_pool.current', 0), ('active.large_pool.freed', 2408), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 13409), ('active.small_pool.current', 320), ('active.small_pool.freed', 13089), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 485434386944), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 485391569408), ('active_bytes.all.peak', 15613830144), ('active_bytes.large_pool.allocated', 484887395840), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 484887395840), ('active_bytes.large_pool.peak', 15565888000), ('active_bytes.small_pool.allocated', 546991104), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 504173568), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 485434386944), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 485391569408), ('allocated_bytes.all.peak', 15613830144), ('allocated_bytes.large_pool.allocated', 484887395840), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 484887395840), ('allocated_bytes.large_pool.peak', 15565888000), ('allocated_bytes.small_pool.allocated', 546991104), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 504173568), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 15817), ('allocation.all.current', 320), ('allocation.all.freed', 15497), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 2408), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 2408), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 13409), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 13089), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 8165), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 8085), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1011), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 1011), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 7154), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 7074), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 212147481600), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 212139967488), ('inactive_split_bytes.all.peak', 7423313408), ('inactive_split_bytes.large_pool.allocated', 211573202944), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 211573202944), ('inactive_split_bytes.large_pool.peak', 7418765824), ('inactive_split_bytes.small_pool.allocated', 574278656), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 566764544), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 45), ('num_ooms', 10), ('reserved_bytes.all.allocated', 309013250048), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 308962918400), ('reserved_bytes.all.peak', 15904800768), ('reserved_bytes.large_pool.allocated', 308893712384), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 308893712384), ('reserved_bytes.large_pool.peak', 15852371968), ('reserved_bytes.small_pool.allocated', 119537664), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 69206016), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 605), ('segment.all.current', 24), ('segment.all.freed', 581), ('segment.all.peak', 58), ('segment.large_pool.allocated', 548), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 548), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 57), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 33), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 17
Points: 7700000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 578.00 MiB (GPU 0; 15.78 GiB total capacity; 14.30 GiB already allocated; 280.75 MiB free; 14.57 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 578.00 MiB (GPU 0; 15.78 GiB total capacity; 14.30 GiB already allocated; 280.75 MiB free; 14.57 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 16719), ('active.all.current', 320), ('active.all.freed', 16399), ('active.all.peak', 753), ('active.large_pool.allocated', 2540), ('active.large_pool.current', 0), ('active.large_pool.freed', 2540), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 14179), ('active.small_pool.current', 320), ('active.small_pool.freed', 13859), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 512087726080), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 512044908544), ('active_bytes.all.peak', 15613830144), ('active_bytes.large_pool.allocated', 511510256128), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 511510256128), ('active_bytes.large_pool.peak', 15565888000), ('active_bytes.small_pool.allocated', 577469952), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 534652416), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 512087726080), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 512044908544), ('allocated_bytes.all.peak', 15613830144), ('allocated_bytes.large_pool.allocated', 511510256128), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 511510256128), ('allocated_bytes.large_pool.peak', 15565888000), ('allocated_bytes.small_pool.allocated', 577469952), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 534652416), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 16719), ('allocation.all.current', 320), ('allocation.all.freed', 16399), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 2540), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 2540), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 14179), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 13859), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 8626), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 8546), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1053), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 1053), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 7573), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 7493), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 226016208384), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 226008694272), ('inactive_split_bytes.all.peak', 7628574720), ('inactive_split_bytes.large_pool.allocated', 225409991168), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 225409991168), ('inactive_split_bytes.large_pool.peak', 7621065728), ('inactive_split_bytes.small_pool.allocated', 606217216), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 598703104), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 46), ('num_ooms', 11), ('reserved_bytes.all.allocated', 324699947008), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 324649615360), ('reserved_bytes.all.peak', 15904800768), ('reserved_bytes.large_pool.allocated', 324576215040), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 324576215040), ('reserved_bytes.large_pool.peak', 15852371968), ('reserved_bytes.small_pool.allocated', 123731968), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 73400320), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 635), ('segment.all.current', 24), ('segment.all.freed', 611), ('segment.all.peak', 58), ('segment.large_pool.allocated', 576), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 576), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 59), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 35), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 18
Points: 7800000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 580.00 MiB (GPU 0; 15.78 GiB total capacity; 14.46 GiB already allocated; 78.75 MiB free; 14.77 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 580.00 MiB (GPU 0; 15.78 GiB total capacity; 14.46 GiB already allocated; 78.75 MiB free; 14.77 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 17621), ('active.all.current', 320), ('active.all.freed', 17301), ('active.all.peak', 753), ('active.large_pool.allocated', 2672), ('active.large_pool.current', 0), ('active.large_pool.freed', 2672), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 14949), ('active.small_pool.current', 320), ('active.small_pool.freed', 14629), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 539033305088), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 538990487552), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 538425356288), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 538425356288), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 607948800), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 565131264), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 539033305088), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 538990487552), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 538425356288), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 538425356288), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 607948800), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 565131264), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 17621), ('allocation.all.current', 320), ('allocation.all.freed', 17301), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 2672), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 2672), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 14949), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 14629), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 9098), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 9018), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1106), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 1106), ('inactive_split.large_pool.peak', 26), ('inactive_split.small_pool.allocated', 7992), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 7912), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 240097674240), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 240090160128), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 239459518464), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 239459518464), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 638155776), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 630641664), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 47), ('num_ooms', 12), ('reserved_bytes.all.allocated', 340510375936), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 340460044288), ('reserved_bytes.all.peak', 15904800768), ('reserved_bytes.large_pool.allocated', 340382449664), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 340382449664), ('reserved_bytes.large_pool.peak', 15852371968), ('reserved_bytes.small_pool.allocated', 127926272), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 77594624), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 665), ('segment.all.current', 24), ('segment.all.freed', 641), ('segment.all.peak', 58), ('segment.large_pool.allocated', 604), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 604), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 61), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 37), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 19
Points: 7900000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 582.00 MiB (GPU 0; 15.78 GiB total capacity; 14.12 GiB already allocated; 428.75 MiB free; 14.43 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/function.py", line 89, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 99, in backward
    grad_in_feat, grad_kernel = bw_fn(
RuntimeError: CUDA out of memory. Tried to allocate 582.00 MiB (GPU 0; 15.78 GiB total capacity; 14.12 GiB already allocated; 428.75 MiB free; 14.43 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 18519), ('active.all.current', 320), ('active.all.freed', 18199), ('active.all.peak', 753), ('active.large_pool.allocated', 2801), ('active.large_pool.current', 0), ('active.large_pool.freed', 2801), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 15718), ('active.small_pool.current', 320), ('active.small_pool.freed', 15398), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 565494688256), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 565451870720), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 564856530944), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 564856530944), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 638157312), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 595339776), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 565494688256), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 565451870720), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 564856530944), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 564856530944), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 638157312), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 595339776), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 18519), ('allocation.all.current', 320), ('allocation.all.freed', 18199), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 2801), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 2801), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 15718), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 15398), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 9586), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 9506), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1175), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 1175), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 8411), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 8331), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 252910905856), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 252903391744), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 252241081856), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 252241081856), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 669824000), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 662309888), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 48), ('num_ooms', 13), ('reserved_bytes.all.allocated', 355953803264), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 355903471616), ('reserved_bytes.all.peak', 15904800768), ('reserved_bytes.large_pool.allocated', 355821682688), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 355821682688), ('reserved_bytes.large_pool.peak', 15852371968), ('reserved_bytes.small_pool.allocated', 132120576), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 81788928), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 694), ('segment.all.current', 24), ('segment.all.freed', 670), ('segment.all.peak', 58), ('segment.large_pool.allocated', 631), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 631), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 63), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 39), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 20
Points: 8000000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 584.00 MiB (GPU 0; 15.78 GiB total capacity; 14.28 GiB already allocated; 270.75 MiB free; 14.58 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/function.py", line 89, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 99, in backward
    grad_in_feat, grad_kernel = bw_fn(
RuntimeError: CUDA out of memory. Tried to allocate 584.00 MiB (GPU 0; 15.78 GiB total capacity; 14.28 GiB already allocated; 270.75 MiB free; 14.58 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 19417), ('active.all.current', 320), ('active.all.freed', 19097), ('active.all.peak', 753), ('active.large_pool.allocated', 2930), ('active.large_pool.current', 0), ('active.large_pool.freed', 2930), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 16487), ('active.small_pool.current', 320), ('active.small_pool.freed', 16167), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 592245444096), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 592202626560), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 591577078272), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 591577078272), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 668365824), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 625548288), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 592245444096), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 592202626560), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 591577078272), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 591577078272), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 668365824), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 625548288), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 19417), ('allocation.all.current', 320), ('allocation.all.freed', 19097), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 2930), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 2930), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 16487), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 16167), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10072), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 9992), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1242), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 1242), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 8830), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 8750), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 267315757568), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 267308243456), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 266614265344), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 266614265344), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 701492224), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 693978112), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 49), ('num_ooms', 14), ('reserved_bytes.all.allocated', 371562905600), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 371512573952), ('reserved_bytes.all.peak', 15904800768), ('reserved_bytes.large_pool.allocated', 371426590720), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 371426590720), ('reserved_bytes.large_pool.peak', 15852371968), ('reserved_bytes.small_pool.allocated', 136314880), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 85983232), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 723), ('segment.all.current', 24), ('segment.all.freed', 699), ('segment.all.peak', 58), ('segment.large_pool.allocated', 658), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 658), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 65), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 41), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 21
Points: 8100000
run forward
run backward
OOM occurred: CUDA out of memory. Tried to allocate 586.00 MiB (GPU 0; 15.78 GiB total capacity; 14.44 GiB already allocated; 116.75 MiB free; 14.73 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 75, in <module>
    loss.backward()
  File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/function.py", line 89, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 99, in backward
    grad_in_feat, grad_kernel = bw_fn(
RuntimeError: CUDA out of memory. Tried to allocate 586.00 MiB (GPU 0; 15.78 GiB total capacity; 14.44 GiB already allocated; 116.75 MiB free; 14.73 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     1003.000 MB GPU memory
OrderedDict([('active.all.allocated', 20315), ('active.all.current', 320), ('active.all.freed', 19995), ('active.all.peak', 753), ('active.large_pool.allocated', 3059), ('active.large_pool.current', 0), ('active.large_pool.freed', 3059), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17256), ('active.small_pool.current', 320), ('active.small_pool.freed', 16936), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 619287548928), ('active_bytes.all.current', 42817536), ('active_bytes.all.freed', 619244731392), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 618588974592), ('active_bytes.large_pool.current', 0), ('active_bytes.large_pool.freed', 618588974592), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698574336), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655756800), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 619287548928), ('allocated_bytes.all.current', 42817536), ('allocated_bytes.all.freed', 619244731392), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 618588974592), ('allocated_bytes.large_pool.current', 0), ('allocated_bytes.large_pool.freed', 618588974592), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698574336), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655756800), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 20315), ('allocation.all.current', 320), ('allocation.all.freed', 19995), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3059), ('allocation.large_pool.current', 0), ('allocation.large_pool.freed', 3059), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17256), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 16936), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10546), ('inactive_split.all.current', 80), ('inactive_split.all.freed', 10466), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1297), ('inactive_split.large_pool.current', 0), ('inactive_split.large_pool.freed', 1297), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9249), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9169), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 281826680832), ('inactive_split_bytes.all.current', 7514112), ('inactive_split_bytes.all.freed', 281819166720), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 281093520384), ('inactive_split_bytes.large_pool.current', 0), ('inactive_split_bytes.large_pool.freed', 281093520384), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733160448), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725646336), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 50), ('num_ooms', 15), ('reserved_bytes.all.allocated', 387333488640), ('reserved_bytes.all.current', 50331648), ('reserved_bytes.all.freed', 387283156992), ('reserved_bytes.all.peak', 15904800768), ('reserved_bytes.large_pool.allocated', 387192979456), ('reserved_bytes.large_pool.current', 0), ('reserved_bytes.large_pool.freed', 387192979456), ('reserved_bytes.large_pool.peak', 15852371968), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 752), ('segment.all.current', 24), ('segment.all.freed', 728), ('segment.all.peak', 58), ('segment.large_pool.allocated', 685), ('segment.large_pool.current', 0), ('segment.large_pool.freed', 685), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 16112.5 MB
********************************************************************************

starting 22
Points: 8200000
run forward
OOM occurred: std::bad_alloc: cudaErrorMemoryAllocation: out of memory -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 314, in forward
    outfeat = self.conv.apply(
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 72, in forward
    return fw_fn(
MemoryError: std::bad_alloc: cudaErrorMemoryAllocation: out of memory

GPU:0
process      18575 uses     1955.000 MB GPU memory
OrderedDict([('active.all.allocated', 20417), ('active.all.current', 321), ('active.all.freed', 20096), ('active.all.peak', 753), ('active.large_pool.allocated', 3150), ('active.large_pool.current', 1), ('active.large_pool.freed', 3149), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17267), ('active.small_pool.current', 320), ('active.small_pool.freed', 16947), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 645763552256), ('active_bytes.all.current', 185043968), ('active_bytes.all.freed', 645578508288), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 645064972288), ('active_bytes.large_pool.current', 142226432), ('active_bytes.large_pool.freed', 644922745856), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698579968), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655762432), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 645763552256), ('allocated_bytes.all.current', 185043968), ('allocated_bytes.all.freed', 645578508288), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 645064972288), ('allocated_bytes.large_pool.current', 142226432), ('allocated_bytes.large_pool.freed', 644922745856), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698579968), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655762432), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 20417), ('allocation.all.current', 321), ('allocation.all.freed', 20096), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3150), ('allocation.large_pool.current', 1), ('allocation.large_pool.freed', 3149), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17267), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 16947), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10586), ('inactive_split.all.current', 82), ('inactive_split.all.freed', 10504), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1326), ('inactive_split.large_pool.current', 2), ('inactive_split.large_pool.freed', 1324), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9260), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9180), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 294001964032), ('inactive_split_bytes.all.current', 863532032), ('inactive_split_bytes.all.freed', 293138432000), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 293268797952), ('inactive_split_bytes.large_pool.current', 856017920), ('inactive_split_bytes.large_pool.freed', 292412780032), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733166080), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725651968), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 50), ('num_ooms', 15), ('reserved_bytes.all.allocated', 403183763456), ('reserved_bytes.all.current', 1048576000), ('reserved_bytes.all.freed', 402135187456), ('reserved_bytes.all.peak', 15904800768), ('reserved_bytes.large_pool.allocated', 403043254272), ('reserved_bytes.large_pool.current', 998244352), ('reserved_bytes.large_pool.freed', 402045009920), ('reserved_bytes.large_pool.peak', 15852371968), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 778), ('segment.all.current', 25), ('segment.all.freed', 753), ('segment.all.peak', 58), ('segment.large_pool.allocated', 711), ('segment.large_pool.current', 1), ('segment.large_pool.freed', 710), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 15160.5 MB
********************************************************************************

starting 23
Points: 8300000
run forward
OOM occurred: std::bad_alloc: cudaErrorMemoryAllocation: out of memory -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 314, in forward
    outfeat = self.conv.apply(
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 72, in forward
    return fw_fn(
MemoryError: std::bad_alloc: cudaErrorMemoryAllocation: out of memory

GPU:0
process      18575 uses     2909.000 MB GPU memory
OrderedDict([('active.all.allocated', 20500), ('active.all.current', 322), ('active.all.freed', 20178), ('active.all.peak', 753), ('active.large_pool.allocated', 3224), ('active.large_pool.current', 2), ('active.large_pool.freed', 3222), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17276), ('active.small_pool.current', 320), ('active.small_pool.freed', 16956), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 670102288896), ('active_bytes.all.current', 1185385472), ('active_bytes.all.freed', 668916903424), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 669403704320), ('active_bytes.large_pool.current', 1142567936), ('active_bytes.large_pool.freed', 668261136384), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698584576), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655767040), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 670102288896), ('allocated_bytes.all.current', 1185385472), ('allocated_bytes.all.freed', 668916903424), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 669403704320), ('allocated_bytes.large_pool.current', 1142567936), ('allocated_bytes.large_pool.freed', 668261136384), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698584576), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655767040), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 20500), ('allocation.all.current', 322), ('allocation.all.freed', 20178), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3224), ('allocation.large_pool.current', 2), ('allocation.large_pool.freed', 3222), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17276), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 16956), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10630), ('inactive_split.all.current', 82), ('inactive_split.all.freed', 10548), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1361), ('inactive_split.large_pool.current', 2), ('inactive_split.large_pool.freed', 1359), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9269), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9189), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 305426830848), ('inactive_split_bytes.all.current', 863532032), ('inactive_split_bytes.all.freed', 304563298816), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 304693660160), ('inactive_split_bytes.large_pool.current', 856017920), ('inactive_split_bytes.large_pool.freed', 303837642240), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733170688), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725656576), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 50), ('num_ooms', 15), ('reserved_bytes.all.allocated', 417782038528), ('reserved_bytes.all.current', 2048917504), ('reserved_bytes.all.freed', 415733121024), ('reserved_bytes.all.peak', 15904800768), ('reserved_bytes.large_pool.allocated', 417641529344), ('reserved_bytes.large_pool.current', 1998585856), ('reserved_bytes.large_pool.freed', 415642943488), ('reserved_bytes.large_pool.peak', 15852371968), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 792), ('segment.all.current', 26), ('segment.all.freed', 766), ('segment.all.peak', 58), ('segment.large_pool.allocated', 725), ('segment.large_pool.current', 2), ('segment.large_pool.freed', 723), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 14206.5 MB
********************************************************************************

starting 24
Points: 8400000
run forward
OOM occurred: CUDA out of memory. Tried to allocate 958.00 MiB (GPU 0; 15.78 GiB total capacity; 13.97 GiB already allocated; 132.75 MiB free; 14.72 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 314, in forward
    outfeat = self.conv.apply(
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 72, in forward
    return fw_fn(
RuntimeError: CUDA out of memory. Tried to allocate 958.00 MiB (GPU 0; 15.78 GiB total capacity; 13.97 GiB already allocated; 132.75 MiB free; 14.72 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     2909.000 MB GPU memory
OrderedDict([('active.all.allocated', 20579), ('active.all.current', 322), ('active.all.freed', 20257), ('active.all.peak', 753), ('active.large_pool.allocated', 3294), ('active.large_pool.current', 2), ('active.large_pool.freed', 3292), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17285), ('active.small_pool.current', 320), ('active.small_pool.freed', 16965), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 693437778944), ('active_bytes.all.current', 1185385472), ('active_bytes.all.freed', 692252393472), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 692739189760), ('active_bytes.large_pool.current', 1142567936), ('active_bytes.large_pool.freed', 691596621824), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698589184), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655771648), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 693437778944), ('allocated_bytes.all.current', 1185385472), ('allocated_bytes.all.freed', 692252393472), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 692739189760), ('allocated_bytes.large_pool.current', 1142567936), ('allocated_bytes.large_pool.freed', 691596621824), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698589184), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655771648), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 20579), ('allocation.all.current', 322), ('allocation.all.freed', 20257), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3294), ('allocation.large_pool.current', 2), ('allocation.large_pool.freed', 3292), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17285), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 16965), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10675), ('inactive_split.all.current', 82), ('inactive_split.all.freed', 10593), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1397), ('inactive_split.large_pool.current', 2), ('inactive_split.large_pool.freed', 1395), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9278), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9198), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 316721113088), ('inactive_split_bytes.all.current', 863532032), ('inactive_split_bytes.all.freed', 315857581056), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 315987937792), ('inactive_split_bytes.large_pool.current', 856017920), ('inactive_split_bytes.large_pool.freed', 315131919872), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733175296), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725661184), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 52), ('num_ooms', 17), ('reserved_bytes.all.allocated', 431535161344), ('reserved_bytes.all.current', 2048917504), ('reserved_bytes.all.freed', 429486243840), ('reserved_bytes.all.peak', 15904800768), ('reserved_bytes.large_pool.allocated', 431394652160), ('reserved_bytes.large_pool.current', 1998585856), ('reserved_bytes.large_pool.freed', 429396066304), ('reserved_bytes.large_pool.peak', 15852371968), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 805), ('segment.all.current', 26), ('segment.all.freed', 779), ('segment.all.peak', 58), ('segment.large_pool.allocated', 738), ('segment.large_pool.current', 2), ('segment.large_pool.freed', 736), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 14206.5 MB
********************************************************************************

starting 25
Points: 8500000
run forward
OOM occurred: CUDA out of memory. Tried to allocate 960.00 MiB (GPU 0; 15.78 GiB total capacity; 14.11 GiB already allocated; 20.75 MiB free; 14.83 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 314, in forward
    outfeat = self.conv.apply(
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 72, in forward
    return fw_fn(
RuntimeError: CUDA out of memory. Tried to allocate 960.00 MiB (GPU 0; 15.78 GiB total capacity; 14.11 GiB already allocated; 20.75 MiB free; 14.83 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     2909.000 MB GPU memory
OrderedDict([('active.all.allocated', 20658), ('active.all.current', 322), ('active.all.freed', 20336), ('active.all.peak', 753), ('active.large_pool.allocated', 3364), ('active.large_pool.current', 2), ('active.large_pool.freed', 3362), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17294), ('active.small_pool.current', 320), ('active.small_pool.freed', 16974), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 717040686592), ('active_bytes.all.current', 1185385472), ('active_bytes.all.freed', 715855301120), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 716342092800), ('active_bytes.large_pool.current', 1142567936), ('active_bytes.large_pool.freed', 715199524864), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698593792), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655776256), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 717040686592), ('allocated_bytes.all.current', 1185385472), ('allocated_bytes.all.freed', 715855301120), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 716342092800), ('allocated_bytes.large_pool.current', 1142567936), ('allocated_bytes.large_pool.freed', 715199524864), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698593792), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655776256), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 20658), ('allocation.all.current', 322), ('allocation.all.freed', 20336), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3364), ('allocation.large_pool.current', 2), ('allocation.large_pool.freed', 3362), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17294), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 16974), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10718), ('inactive_split.all.current', 82), ('inactive_split.all.freed', 10636), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1431), ('inactive_split.large_pool.current', 2), ('inactive_split.large_pool.freed', 1429), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9287), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9207), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 328151494656), ('inactive_split_bytes.all.current', 863532032), ('inactive_split_bytes.all.freed', 327287962624), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 327418314752), ('inactive_split_bytes.large_pool.current', 856017920), ('inactive_split_bytes.large_pool.freed', 326562296832), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733179904), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725665792), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 55), ('num_ooms', 19), ('reserved_bytes.all.allocated', 445441376256), ('reserved_bytes.all.current', 2048917504), ('reserved_bytes.all.freed', 443392458752), ('reserved_bytes.all.peak', 15919480832), ('reserved_bytes.large_pool.allocated', 445300867072), ('reserved_bytes.large_pool.current', 1998585856), ('reserved_bytes.large_pool.freed', 443302281216), ('reserved_bytes.large_pool.peak', 15869149184), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 818), ('segment.all.current', 26), ('segment.all.freed', 792), ('segment.all.peak', 58), ('segment.large_pool.allocated', 751), ('segment.large_pool.current', 2), ('segment.large_pool.freed', 749), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 14206.5 MB
********************************************************************************

starting 26
Points: 8600000
run forward
OOM occurred: CUDA out of memory. Tried to allocate 1.37 GiB (GPU 0; 15.78 GiB total capacity; 12.32 GiB already allocated; 1.22 GiB free; 13.63 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiNonlinearity.py", line 42, in forward
    output = self.module(input.F)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 102, in forward
    return F.relu(input, inplace=self.inplace)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 1136, in relu
    result = torch.relu(input)
RuntimeError: CUDA out of memory. Tried to allocate 1.37 GiB (GPU 0; 15.78 GiB total capacity; 12.32 GiB already allocated; 1.22 GiB free; 13.63 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     2909.000 MB GPU memory
OrderedDict([('active.all.allocated', 20728), ('active.all.current', 322), ('active.all.freed', 20406), ('active.all.peak', 753), ('active.large_pool.allocated', 3427), ('active.large_pool.current', 2), ('active.large_pool.freed', 3425), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17301), ('active.small_pool.current', 320), ('active.small_pool.freed', 16981), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 738765829632), ('active_bytes.all.current', 1185385472), ('active_bytes.all.freed', 737580444160), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 738067232256), ('active_bytes.large_pool.current', 1142567936), ('active_bytes.large_pool.freed', 736924664320), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698597376), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655779840), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 738765829632), ('allocated_bytes.all.current', 1185385472), ('allocated_bytes.all.freed', 737580444160), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 738067232256), ('allocated_bytes.large_pool.current', 1142567936), ('allocated_bytes.large_pool.freed', 736924664320), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698597376), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655779840), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 20728), ('allocation.all.current', 322), ('allocation.all.freed', 20406), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3427), ('allocation.large_pool.current', 2), ('allocation.large_pool.freed', 3425), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17301), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 16981), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10752), ('inactive_split.all.current', 82), ('inactive_split.all.freed', 10670), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1458), ('inactive_split.large_pool.current', 2), ('inactive_split.large_pool.freed', 1456), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9294), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9214), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 337962648064), ('inactive_split_bytes.all.current', 863532032), ('inactive_split_bytes.all.freed', 337099116032), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 337229464576), ('inactive_split_bytes.large_pool.current', 856017920), ('inactive_split_bytes.large_pool.freed', 336373446656), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733183488), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725669376), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 56), ('num_ooms', 20), ('reserved_bytes.all.allocated', 458064134144), ('reserved_bytes.all.current', 2048917504), ('reserved_bytes.all.freed', 456015216640), ('reserved_bytes.all.peak', 15919480832), ('reserved_bytes.large_pool.allocated', 457923624960), ('reserved_bytes.large_pool.current', 1998585856), ('reserved_bytes.large_pool.freed', 455925039104), ('reserved_bytes.large_pool.peak', 15869149184), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 831), ('segment.all.current', 26), ('segment.all.freed', 805), ('segment.all.peak', 58), ('segment.large_pool.allocated', 764), ('segment.large_pool.current', 2), ('segment.large_pool.freed', 762), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 14206.5 MB
********************************************************************************

starting 27
Points: 8700000
run forward
OOM occurred: CUDA out of memory. Tried to allocate 1.39 GiB (GPU 0; 15.78 GiB total capacity; 12.45 GiB already allocated; 1.05 GiB free; 13.79 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiNonlinearity.py", line 42, in forward
    output = self.module(input.F)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 102, in forward
    return F.relu(input, inplace=self.inplace)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 1136, in relu
    result = torch.relu(input)
RuntimeError: CUDA out of memory. Tried to allocate 1.39 GiB (GPU 0; 15.78 GiB total capacity; 12.45 GiB already allocated; 1.05 GiB free; 13.79 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     2909.000 MB GPU memory
OrderedDict([('active.all.allocated', 20798), ('active.all.current', 322), ('active.all.freed', 20476), ('active.all.peak', 753), ('active.large_pool.allocated', 3490), ('active.large_pool.current', 2), ('active.large_pool.freed', 3488), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17308), ('active.small_pool.current', 320), ('active.small_pool.freed', 16988), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 760739129856), ('active_bytes.all.current', 1185385472), ('active_bytes.all.freed', 759553744384), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 760040528896), ('active_bytes.large_pool.current', 1142567936), ('active_bytes.large_pool.freed', 758897960960), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698600960), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655783424), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 760739129856), ('allocated_bytes.all.current', 1185385472), ('allocated_bytes.all.freed', 759553744384), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 760040528896), ('allocated_bytes.large_pool.current', 1142567936), ('allocated_bytes.large_pool.freed', 758897960960), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698600960), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655783424), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 20798), ('allocation.all.current', 322), ('allocation.all.freed', 20476), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3490), ('allocation.large_pool.current', 2), ('allocation.large_pool.freed', 3488), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17308), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 16988), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10781), ('inactive_split.all.current', 82), ('inactive_split.all.freed', 10699), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1480), ('inactive_split.large_pool.current', 2), ('inactive_split.large_pool.freed', 1478), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9301), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9221), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 347879755264), ('inactive_split_bytes.all.current', 863532032), ('inactive_split_bytes.all.freed', 347016223232), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 347146568192), ('inactive_split_bytes.large_pool.current', 856017920), ('inactive_split_bytes.large_pool.freed', 346290550272), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733187072), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725672960), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 57), ('num_ooms', 21), ('reserved_bytes.all.allocated', 470825304064), ('reserved_bytes.all.current', 2048917504), ('reserved_bytes.all.freed', 468776386560), ('reserved_bytes.all.peak', 15919480832), ('reserved_bytes.large_pool.allocated', 470684794880), ('reserved_bytes.large_pool.current', 1998585856), ('reserved_bytes.large_pool.freed', 468686209024), ('reserved_bytes.large_pool.peak', 15869149184), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 844), ('segment.all.current', 26), ('segment.all.freed', 818), ('segment.all.peak', 58), ('segment.large_pool.allocated', 777), ('segment.large_pool.current', 2), ('segment.large_pool.freed', 775), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 14206.5 MB
********************************************************************************

starting 28
Points: 8800000
run forward
OOM occurred: CUDA out of memory. Tried to allocate 1.40 GiB (GPU 0; 15.78 GiB total capacity; 12.57 GiB already allocated; 944.75 MiB free; 13.92 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiNonlinearity.py", line 42, in forward
    output = self.module(input.F)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 102, in forward
    return F.relu(input, inplace=self.inplace)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 1136, in relu
    result = torch.relu(input)
RuntimeError: CUDA out of memory. Tried to allocate 1.40 GiB (GPU 0; 15.78 GiB total capacity; 12.57 GiB already allocated; 944.75 MiB free; 13.92 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     2909.000 MB GPU memory
OrderedDict([('active.all.allocated', 20868), ('active.all.current', 322), ('active.all.freed', 20546), ('active.all.peak', 753), ('active.large_pool.allocated', 3553), ('active.large_pool.current', 2), ('active.large_pool.freed', 3551), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17315), ('active.small_pool.current', 320), ('active.small_pool.freed', 16995), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 782952858624), ('active_bytes.all.current', 1185385472), ('active_bytes.all.freed', 781767473152), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 782254254080), ('active_bytes.large_pool.current', 1142567936), ('active_bytes.large_pool.freed', 781111686144), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698604544), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655787008), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 782952858624), ('allocated_bytes.all.current', 1185385472), ('allocated_bytes.all.freed', 781767473152), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 782254254080), ('allocated_bytes.large_pool.current', 1142567936), ('allocated_bytes.large_pool.freed', 781111686144), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698604544), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655787008), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 20868), ('allocation.all.current', 322), ('allocation.all.freed', 20546), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3553), ('allocation.large_pool.current', 2), ('allocation.large_pool.freed', 3551), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17315), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 16995), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10812), ('inactive_split.all.current', 82), ('inactive_split.all.freed', 10730), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1504), ('inactive_split.large_pool.current', 2), ('inactive_split.large_pool.freed', 1502), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9308), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9228), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 357913366528), ('inactive_split_bytes.all.current', 863532032), ('inactive_split_bytes.all.freed', 357049834496), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 357180175872), ('inactive_split_bytes.large_pool.current', 856017920), ('inactive_split_bytes.large_pool.freed', 356324157952), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733190656), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725676544), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 58), ('num_ooms', 22), ('reserved_bytes.all.allocated', 483726983168), ('reserved_bytes.all.current', 2048917504), ('reserved_bytes.all.freed', 481678065664), ('reserved_bytes.all.peak', 15919480832), ('reserved_bytes.large_pool.allocated', 483586473984), ('reserved_bytes.large_pool.current', 1998585856), ('reserved_bytes.large_pool.freed', 481587888128), ('reserved_bytes.large_pool.peak', 15869149184), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 857), ('segment.all.current', 26), ('segment.all.freed', 831), ('segment.all.peak', 58), ('segment.large_pool.allocated', 790), ('segment.large_pool.current', 2), ('segment.large_pool.freed', 788), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 14206.5 MB
********************************************************************************

starting 29
Points: 8900000
run forward
OOM occurred: CUDA out of memory. Tried to allocate 1.41 GiB (GPU 0; 15.78 GiB total capacity; 12.70 GiB already allocated; 804.75 MiB free; 14.06 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiNonlinearity.py", line 42, in forward
    output = self.module(input.F)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 102, in forward
    return F.relu(input, inplace=self.inplace)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 1136, in relu
    result = torch.relu(input)
RuntimeError: CUDA out of memory. Tried to allocate 1.41 GiB (GPU 0; 15.78 GiB total capacity; 12.70 GiB already allocated; 804.75 MiB free; 14.06 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     2909.000 MB GPU memory
OrderedDict([('active.all.allocated', 20938), ('active.all.current', 322), ('active.all.freed', 20616), ('active.all.peak', 753), ('active.large_pool.allocated', 3616), ('active.large_pool.current', 2), ('active.large_pool.freed', 3614), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17322), ('active.small_pool.current', 320), ('active.small_pool.freed', 17002), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 805410977792), ('active_bytes.all.current', 1185385472), ('active_bytes.all.freed', 804225592320), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 804712369664), ('active_bytes.large_pool.current', 1142567936), ('active_bytes.large_pool.freed', 803569801728), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698608128), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655790592), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 805410977792), ('allocated_bytes.all.current', 1185385472), ('allocated_bytes.all.freed', 804225592320), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 804712369664), ('allocated_bytes.large_pool.current', 1142567936), ('allocated_bytes.large_pool.freed', 803569801728), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698608128), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655790592), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 20938), ('allocation.all.current', 322), ('allocation.all.freed', 20616), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3616), ('allocation.large_pool.current', 2), ('allocation.large_pool.freed', 3614), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17322), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 17002), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10847), ('inactive_split.all.current', 82), ('inactive_split.all.freed', 10765), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1532), ('inactive_split.large_pool.current', 2), ('inactive_split.large_pool.freed', 1530), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9315), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9235), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 368074455552), ('inactive_split_bytes.all.current', 863532032), ('inactive_split_bytes.all.freed', 367210923520), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 367341261312), ('inactive_split_bytes.large_pool.current', 856017920), ('inactive_split_bytes.large_pool.freed', 366485243392), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733194240), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725680128), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 59), ('num_ooms', 23), ('reserved_bytes.all.allocated', 496775462912), ('reserved_bytes.all.current', 2048917504), ('reserved_bytes.all.freed', 494726545408), ('reserved_bytes.all.peak', 15919480832), ('reserved_bytes.large_pool.allocated', 496634953728), ('reserved_bytes.large_pool.current', 1998585856), ('reserved_bytes.large_pool.freed', 494636367872), ('reserved_bytes.large_pool.peak', 15869149184), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 870), ('segment.all.current', 26), ('segment.all.freed', 844), ('segment.all.peak', 58), ('segment.large_pool.allocated', 803), ('segment.large_pool.current', 2), ('segment.large_pool.freed', 801), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 14206.5 MB
********************************************************************************

starting 30
Points: 9000000
run forward
OOM occurred: CUDA out of memory. Tried to allocate 1.43 GiB (GPU 0; 15.78 GiB total capacity; 12.83 GiB already allocated; 672.75 MiB free; 14.19 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiNonlinearity.py", line 42, in forward
    output = self.module(input.F)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 102, in forward
    return F.relu(input, inplace=self.inplace)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 1136, in relu
    result = torch.relu(input)
RuntimeError: CUDA out of memory. Tried to allocate 1.43 GiB (GPU 0; 15.78 GiB total capacity; 12.83 GiB already allocated; 672.75 MiB free; 14.19 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     2909.000 MB GPU memory
OrderedDict([('active.all.allocated', 21008), ('active.all.current', 322), ('active.all.freed', 20686), ('active.all.peak', 753), ('active.large_pool.allocated', 3679), ('active.large_pool.current', 2), ('active.large_pool.freed', 3677), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17329), ('active.small_pool.current', 320), ('active.small_pool.freed', 17009), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 828114376192), ('active_bytes.all.current', 1185385472), ('active_bytes.all.freed', 826928990720), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 827415764480), ('active_bytes.large_pool.current', 1142567936), ('active_bytes.large_pool.freed', 826273196544), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698611712), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655794176), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 828114376192), ('allocated_bytes.all.current', 1185385472), ('allocated_bytes.all.freed', 826928990720), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 827415764480), ('allocated_bytes.large_pool.current', 1142567936), ('allocated_bytes.large_pool.freed', 826273196544), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698611712), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655794176), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 21008), ('allocation.all.current', 322), ('allocation.all.freed', 20686), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3679), ('allocation.large_pool.current', 2), ('allocation.large_pool.freed', 3677), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17329), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 17009), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10882), ('inactive_split.all.current', 82), ('inactive_split.all.freed', 10800), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1560), ('inactive_split.large_pool.current', 2), ('inactive_split.large_pool.freed', 1558), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9322), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9242), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 378342091776), ('inactive_split_bytes.all.current', 863532032), ('inactive_split_bytes.all.freed', 377478559744), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 377608893952), ('inactive_split_bytes.large_pool.current', 856017920), ('inactive_split_bytes.large_pool.freed', 376752876032), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733197824), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725683712), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 60), ('num_ooms', 24), ('reserved_bytes.all.allocated', 509962354688), ('reserved_bytes.all.current', 2048917504), ('reserved_bytes.all.freed', 507913437184), ('reserved_bytes.all.peak', 15919480832), ('reserved_bytes.large_pool.allocated', 509821845504), ('reserved_bytes.large_pool.current', 1998585856), ('reserved_bytes.large_pool.freed', 507823259648), ('reserved_bytes.large_pool.peak', 15869149184), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 883), ('segment.all.current', 26), ('segment.all.freed', 857), ('segment.all.peak', 58), ('segment.large_pool.allocated', 816), ('segment.large_pool.current', 2), ('segment.large_pool.freed', 814), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 14206.5 MB
********************************************************************************

starting 31
Points: 9100000
run forward
OOM occurred: CUDA out of memory. Tried to allocate 1.44 GiB (GPU 0; 15.78 GiB total capacity; 12.96 GiB already allocated; 508.75 MiB free; 14.35 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiNonlinearity.py", line 42, in forward
    output = self.module(input.F)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 102, in forward
    return F.relu(input, inplace=self.inplace)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 1136, in relu
    result = torch.relu(input)
RuntimeError: CUDA out of memory. Tried to allocate 1.44 GiB (GPU 0; 15.78 GiB total capacity; 12.96 GiB already allocated; 508.75 MiB free; 14.35 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     2909.000 MB GPU memory
OrderedDict([('active.all.allocated', 21078), ('active.all.current', 322), ('active.all.freed', 20756), ('active.all.peak', 753), ('active.large_pool.allocated', 3742), ('active.large_pool.current', 2), ('active.large_pool.freed', 3740), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17336), ('active.small_pool.current', 320), ('active.small_pool.freed', 17016), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 851063911424), ('active_bytes.all.current', 1185385472), ('active_bytes.all.freed', 849878525952), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 850365296128), ('active_bytes.large_pool.current', 1142567936), ('active_bytes.large_pool.freed', 849222728192), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698615296), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655797760), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 851063911424), ('allocated_bytes.all.current', 1185385472), ('allocated_bytes.all.freed', 849878525952), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 850365296128), ('allocated_bytes.large_pool.current', 1142567936), ('allocated_bytes.large_pool.freed', 849222728192), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698615296), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655797760), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 21078), ('allocation.all.current', 322), ('allocation.all.freed', 20756), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3742), ('allocation.large_pool.current', 2), ('allocation.large_pool.freed', 3740), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17336), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 17016), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10912), ('inactive_split.all.current', 82), ('inactive_split.all.freed', 10830), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1583), ('inactive_split.large_pool.current', 2), ('inactive_split.large_pool.freed', 1581), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9329), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9249), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 388678122496), ('inactive_split_bytes.all.current', 863532032), ('inactive_split_bytes.all.freed', 387814590464), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 387944921088), ('inactive_split_bytes.large_pool.current', 856017920), ('inactive_split_bytes.large_pool.freed', 387088903168), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733201408), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725687296), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 61), ('num_ooms', 25), ('reserved_bytes.all.allocated', 523321212928), ('reserved_bytes.all.current', 2048917504), ('reserved_bytes.all.freed', 521272295424), ('reserved_bytes.all.peak', 15919480832), ('reserved_bytes.large_pool.allocated', 523180703744), ('reserved_bytes.large_pool.current', 1998585856), ('reserved_bytes.large_pool.freed', 521182117888), ('reserved_bytes.large_pool.peak', 15869149184), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 894), ('segment.all.current', 26), ('segment.all.freed', 868), ('segment.all.peak', 58), ('segment.large_pool.allocated', 827), ('segment.large_pool.current', 2), ('segment.large_pool.freed', 825), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 14206.5 MB
********************************************************************************

starting 32
Points: 9200000
run forward
OOM occurred: CUDA out of memory. Tried to allocate 1.45 GiB (GPU 0; 15.78 GiB total capacity; 13.08 GiB already allocated; 374.75 MiB free; 14.48 GiB reserved in total by PyTorch) -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiNonlinearity.py", line 42, in forward
    output = self.module(input.F)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 102, in forward
    return F.relu(input, inplace=self.inplace)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 1136, in relu
    result = torch.relu(input)
RuntimeError: CUDA out of memory. Tried to allocate 1.45 GiB (GPU 0; 15.78 GiB total capacity; 13.08 GiB already allocated; 374.75 MiB free; 14.48 GiB reserved in total by PyTorch)

GPU:0
process      18575 uses     2909.000 MB GPU memory
OrderedDict([('active.all.allocated', 21148), ('active.all.current', 322), ('active.all.freed', 20826), ('active.all.peak', 753), ('active.large_pool.allocated', 3805), ('active.large_pool.current', 2), ('active.large_pool.freed', 3803), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17343), ('active.small_pool.current', 320), ('active.small_pool.freed', 17023), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 874260035584), ('active_bytes.all.current', 1185385472), ('active_bytes.all.freed', 873074650112), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 873561416704), ('active_bytes.large_pool.current', 1142567936), ('active_bytes.large_pool.freed', 872418848768), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698618880), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655801344), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 874260035584), ('allocated_bytes.all.current', 1185385472), ('allocated_bytes.all.freed', 873074650112), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 873561416704), ('allocated_bytes.large_pool.current', 1142567936), ('allocated_bytes.large_pool.freed', 872418848768), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698618880), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655801344), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 21148), ('allocation.all.current', 322), ('allocation.all.freed', 20826), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3805), ('allocation.large_pool.current', 2), ('allocation.large_pool.freed', 3803), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17343), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 17023), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10940), ('inactive_split.all.current', 82), ('inactive_split.all.freed', 10858), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1604), ('inactive_split.large_pool.current', 2), ('inactive_split.large_pool.freed', 1602), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9336), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9256), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 399120909312), ('inactive_split_bytes.all.current', 863532032), ('inactive_split_bytes.all.freed', 398257377280), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 398387704320), ('inactive_split_bytes.large_pool.current', 856017920), ('inactive_split_bytes.large_pool.freed', 397531686400), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733204992), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725690880), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 62), ('num_ooms', 26), ('reserved_bytes.all.allocated', 536820580352), ('reserved_bytes.all.current', 2048917504), ('reserved_bytes.all.freed', 534771662848), ('reserved_bytes.all.peak', 15919480832), ('reserved_bytes.large_pool.allocated', 536680071168), ('reserved_bytes.large_pool.current', 1998585856), ('reserved_bytes.large_pool.freed', 534681485312), ('reserved_bytes.large_pool.peak', 15869149184), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 905), ('segment.all.current', 26), ('segment.all.freed', 879), ('segment.all.peak', 58), ('segment.large_pool.allocated', 838), ('segment.large_pool.current', 2), ('segment.large_pool.freed', 836), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 14206.5 MB
********************************************************************************

starting 33
Points: 9300000
run forward
OOM occurred: std::bad_alloc: cudaErrorMemoryAllocation: out of memory -> Traceback (most recent call last):
  File "test.py", line 70, in <module>
    res = testnetwork(xt)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "test.py", line 37, in forward
    temp = convlayer(temp)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 314, in forward
    outfeat = self.conv.apply(
  File "/opt/conda/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiConvolution.py", line 72, in forward
    return fw_fn(
MemoryError: std::bad_alloc: cudaErrorMemoryAllocation: out of memory

GPU:0
process      18575 uses     5341.000 MB GPU memory
OrderedDict([('active.all.allocated', 21212), ('active.all.current', 323), ('active.all.freed', 20889), ('active.all.peak', 753), ('active.large_pool.allocated', 3862), ('active.large_pool.current', 3), ('active.large_pool.freed', 3859), ('active.large_pool.peak', 67), ('active.small_pool.allocated', 17350), ('active.small_pool.current', 320), ('active.small_pool.freed', 17030), ('active.small_pool.peak', 689), ('active_bytes.all.allocated', 895483396608), ('active_bytes.all.current', 3734341632), ('active_bytes.all.freed', 891749054976), ('active_bytes.all.peak', 15763564032), ('active_bytes.large_pool.allocated', 894784774144), ('active_bytes.large_pool.current', 3691524096), ('active_bytes.large_pool.freed', 891093250048), ('active_bytes.large_pool.peak', 15715351552), ('active_bytes.small_pool.allocated', 698622464), ('active_bytes.small_pool.current', 42817536), ('active_bytes.small_pool.freed', 655804928), ('active_bytes.small_pool.peak', 48656384), ('allocated_bytes.all.allocated', 895483396608), ('allocated_bytes.all.current', 3734341632), ('allocated_bytes.all.freed', 891749054976), ('allocated_bytes.all.peak', 15763564032), ('allocated_bytes.large_pool.allocated', 894784774144), ('allocated_bytes.large_pool.current', 3691524096), ('allocated_bytes.large_pool.freed', 891093250048), ('allocated_bytes.large_pool.peak', 15715351552), ('allocated_bytes.small_pool.allocated', 698622464), ('allocated_bytes.small_pool.current', 42817536), ('allocated_bytes.small_pool.freed', 655804928), ('allocated_bytes.small_pool.peak', 48656384), ('allocation.all.allocated', 21212), ('allocation.all.current', 323), ('allocation.all.freed', 20889), ('allocation.all.peak', 753), ('allocation.large_pool.allocated', 3862), ('allocation.large_pool.current', 3), ('allocation.large_pool.freed', 3859), ('allocation.large_pool.peak', 67), ('allocation.small_pool.allocated', 17350), ('allocation.small_pool.current', 320), ('allocation.small_pool.freed', 17030), ('allocation.small_pool.peak', 689), ('inactive_split.all.allocated', 10968), ('inactive_split.all.current', 83), ('inactive_split.all.freed', 10885), ('inactive_split.all.peak', 155), ('inactive_split.large_pool.allocated', 1625), ('inactive_split.large_pool.current', 3), ('inactive_split.large_pool.freed', 1622), ('inactive_split.large_pool.peak', 28), ('inactive_split.small_pool.allocated', 9343), ('inactive_split.small_pool.current', 80), ('inactive_split.small_pool.freed', 9263), ('inactive_split.small_pool.peak', 137), ('inactive_split_bytes.all.allocated', 408058970624), ('inactive_split_bytes.all.current', 864712704), ('inactive_split_bytes.all.freed', 407194257920), ('inactive_split_bytes.all.peak', 7749970432), ('inactive_split_bytes.large_pool.allocated', 407325762048), ('inactive_split_bytes.large_pool.current', 857198592), ('inactive_split_bytes.large_pool.freed', 406468563456), ('inactive_split_bytes.large_pool.peak', 7742461440), ('inactive_split_bytes.small_pool.allocated', 733208576), ('inactive_split_bytes.small_pool.current', 7514112), ('inactive_split_bytes.small_pool.freed', 725694464), ('inactive_split_bytes.small_pool.peak', 7574016), ('num_alloc_retries', 62), ('num_ooms', 26), ('reserved_bytes.all.allocated', 550458359808), ('reserved_bytes.all.current', 4599054336), ('reserved_bytes.all.freed', 545859305472), ('reserved_bytes.all.peak', 15919480832), ('reserved_bytes.large_pool.allocated', 550317850624), ('reserved_bytes.large_pool.current', 4548722688), ('reserved_bytes.large_pool.freed', 545769127936), ('reserved_bytes.large_pool.peak', 15869149184), ('reserved_bytes.small_pool.allocated', 140509184), ('reserved_bytes.small_pool.current', 50331648), ('reserved_bytes.small_pool.freed', 90177536), ('reserved_bytes.small_pool.peak', 54525952), ('segment.all.allocated', 916), ('segment.all.current', 27), ('segment.all.freed', 889), ('segment.all.peak', 58), ('segment.large_pool.allocated', 849), ('segment.large_pool.current', 3), ('segment.large_pool.freed', 846), ('segment.large_pool.peak', 32), ('segment.small_pool.allocated', 67), ('segment.small_pool.current', 24), ('segment.small_pool.freed', 43), ('segment.small_pool.peak', 26)])

Free GPU vram: 11774.5 MB
********************************************************************************

The interesting part in torch.cuda.memory_stats() is the active.large_pool.current stays 0 until no memory leaks, after that anytime an error occurs this pool is incremented. These CUDA arrays are not used and cannot be freed with torch.cuda.empty_cache(). This is how we have less and less available VRAM leading to more OOM errors (so we get exponentially less memory from each error). The active.large_pool.current pool should remain 0 after many errors.

I'm trying to investigate where these tensors leak but it's extremely hard due to the caching allocator. Do you have any idea where it could come from?

Jun 07 '21 14:06 evelkey

Hmm, it is strange that the cached allocator does not clean up when there is MemoryError: std::bad_alloc: cudaErrorMemoryAllocation: out of memory. Potentially this is coming from thrust functions.

Jun 08 '21 22:06 chrischoy

I think this is further convoluted as active.large_pool.current is a torch builtin memory reporter and shows that there are some objects in the large pool. However there are no variables in Python's dir() which means that these are some dangling variables in C++ which are somehow still tracked by the allocator. As there is no way to 'reset' the CUDA driver / memory this leads to an error from which we cannot recover.

Jun 09 '21 08:06 evelkey

Reproduced the issue with the code snippet above using the latest PyTorch (v1.9.0-rc3) using ME master and CUDA11.2:

nvcc: NVIDIA (R) Cuda compiler driver
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0

>>> torch.__version__
'1.9.0a0+gitbc446f6'

The memory leak is the same as with CUDA 11.0.3.

Jun 14 '21 11:06 evelkey

helllo, same error,

xiaokeai1-Z10PE-D8-WS 01/04 14:54:15 ===> Epoch[1](1/102): Loss 2.5535  LR: 1.000e-01   Score 9.130     Data time: 4.7232, Total iter time: 6.3637
xiaokeai1-Z10PE-D8-WS 01/04 14:54:33 ===> Epoch[1](40/102): Loss 1.6340 LR: 9.996e-02   Score 55.188    Data time: 0.0188, Total iter time: 0.4620
xiaokeai1-Z10PE-D8-WS 01/04 14:54:51 ===> Epoch[1](80/102): Loss 1.3488 LR: 9.992e-02   Score 60.322    Data time: 0.0183, Total iter time: 0.4438
xiaokeai1-Z10PE-D8-WS 01/04 14:55:10 ===> Epoch[2](120/102): Loss 1.2015        LR: 9.988e-02   Score 64.078    Data time: 0.0153, Total iter time: 0.4588
xiaokeai1-Z10PE-D8-WS 01/04 14:55:28 ===> Epoch[2](160/102): Loss 1.1561        LR: 9.984e-02   Score 64.175    Data time: 0.0134, Total iter time: 0.4598
xiaokeai1-Z10PE-D8-WS 01/04 14:55:47 ===> Epoch[2](200/102): Loss 1.1369        LR: 9.980e-02   Score 64.650    Data time: 0.0162, Total iter time: 0.4730
xiaokeai1-Z10PE-D8-WS 01/04 14:56:06 ===> Epoch[3](240/102): Loss 1.1743        LR: 9.976e-02   Score 64.137    Data time: 0.0124, Total iter time: 0.4646
xiaokeai1-Z10PE-D8-WS 01/04 14:56:24 ===> Epoch[3](280/102): Loss 1.1038        LR: 9.972e-02   Score 66.222    Data time: 0.0134, Total iter time: 0.4539
xiaokeai1-Z10PE-D8-WS 01/04 14:56:42 ===> Epoch[4](320/102): Loss 1.1172        LR: 9.968e-02   Score 66.377    Data time: 0.0173, Total iter time: 0.4364
xiaokeai1-Z10PE-D8-WS 01/04 14:57:00 ===> Epoch[4](360/102): Loss 1.0834        LR: 9.964e-02   Score 66.571    Data time: 0.0151, Total iter time: 0.4494
xiaokeai1-Z10PE-D8-WS 01/04 14:57:18 ===> Epoch[4](400/102): Loss 1.0275        LR: 9.960e-02   Score 67.610    Data time: 0.0154, Total iter time: 0.4529
xiaokeai1-Z10PE-D8-WS 01/04 14:57:37 ===> Epoch[5](440/102): Loss 1.0090        LR: 9.956e-02   Score 68.603    Data time: 0.0090, Total iter time: 0.4593
xiaokeai1-Z10PE-D8-WS 01/04 14:57:55 ===> Epoch[5](480/102): Loss 1.0110        LR: 9.952e-02   Score 68.017    Data time: 0.0132, Total iter time: 0.4582
Traceback (most recent call last):
  File "/home/xiaokeai1/anaconda3/envs/torch1.9mink/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/xiaokeai1/anaconda3/envs/torch1.9mink/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/media/xiaokeai1/Program_dataset/suyunzheng_ws/code/Minkowski_v0p5/SpatioTemporalSegmentation/main.py", line 160, in <module>
    main()
  File "/media/xiaokeai1/Program_dataset/suyunzheng_ws/code/Minkowski_v0p5/SpatioTemporalSegmentation/main.py", line 153, in main
    train(model, train_data_loader, val_data_loader, config)
  File "/media/xiaokeai1/Program_dataset/suyunzheng_ws/code/Minkowski_v0p5/SpatioTemporalSegmentation/lib/train.py", line 94, in train
    soutput = model(*inputs)
  File "/home/xiaokeai1/anaconda3/envs/torch1.9mink/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/xiaokeai1/Program_dataset/suyunzheng_ws/code/Minkowski_v0p5/SpatioTemporalSegmentation/models/af2s3net.py", line 339, in forward
    x, end_points = self.AFF(inputs)            # J=x
  File "/home/xiaokeai1/anaconda3/envs/torch1.9mink/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/xiaokeai1/Program_dataset/suyunzheng_ws/code/Minkowski_v0p5/SpatioTemporalSegmentation/models/af2s3net.py", line 141, in forward
    x3 = self.activate(self.s3conv1(x))         # N*32
  File "/home/xiaokeai1/anaconda3/envs/torch1.9mink/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/xiaokeai1/Program_dataset/suyunzheng_ws/code/Minkowski_v0p5/SpatioTemporalSegmentation/models/af2s3net.py", line 81, in forward
    x = self.bn(self.conv(x))
  File "/home/xiaokeai1/anaconda3/envs/torch1.9mink/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xiaokeai1/anaconda3/envs/torch1.9mink/lib/python3.8/site-packages/MinkowskiEngine/MinkowskiConvolution.py", line 314, in forward
    outfeat = self.conv.apply(
  File "/home/xiaokeai1/anaconda3/envs/torch1.9mink/lib/python3.8/site-packages/MinkowskiEngine/MinkowskiConvolution.py", line 72, in forward
    return fw_fn(
MemoryError: std::bad_alloc: cudaErrorMemoryAllocation: out of memory

OS:Ubuntu18.04
cuda:11.1
pytorch:1.9
MinkowskiEngine:0.5.4

have you solved this problem?

Jan 04 '22 07:01 suyunzzz

i use docker, and same error was encounterd.

Jan 06 '22 05:01 suyunzzz

@suyunzzz this issue hasn't been solved yet, what I did is a wrapper which checks active.large_pool.current and if the number of the pool grows (signals a memleak), I save all the model weights, kill the training process and restart it from the checkpoint. It's not nice but functional until the root cause is unknown.

Jan 12 '22 01:01 evelkey

@suyunzzz this issue hasn't been solved yet, what I did is a wrapper which checks active.large_pool.current and if the number of the pool grows (signals a memleak), I save all the model weights, kill the training process and restart it from the checkpoint. It's not nice but functional until the root cause is unknown.

This doesn't look elegant, but it seems to work, could you show me your code about how to check active.large_pool.current?, thanks a lot

Jan 20 '22 05:01 suyunzzz

I've met the same error and have struggled with it for a couple of weeks. Since I cannot know which training sample would cause Out of Memory error due to the dynamic allocation, it would kill my training process without any chance to restore. Though there is some workaround, but that breaks my workflow (I use pytorchlightning), and puts my log into a mess.

Really hope that this issue could be resolved.

Jul 16 '22 01:07 qq456cvb

I also have this issue (happens after long time of training), is there any explanation or solution?

Dec 28 '23 12:12 yuvalH9

MinkowskiEngine MinkowskiEngine copied to clipboard

CUDA OOM error leads to GPU memory leak

MinkowskiEngine
MinkowskiEngine copied to clipboard