4K4D
4K4D copied to clipboard
Published
20 hours ago •
zju3dv
zju3dv
CUDA out of memory
Hi. Allow me to thank you for the other day's support.
I was trying out Rendering With Minimal Dataset (Only Encoded Videos) using the script file you provided, but I encountered RuntimeError: CUDA out of memory. However, the memory size displayed under Including non-PyTorch memory is a very large 17179869184.00 GiB, so I'm not sure if my PC's GPU is insufficient or if I've made some mistake in the settings. Is it possible to run 4K4D on a GPU with 8GB of memory?
Environment
- OS: Ubuntu 22.04.3 LTS (Windows11 Pro WSL2)
- Anaconda: conda 23.7.4
- GPU: NVIDIA GeForce RTX 3060 Ti (Memory 8GB)
Command
python scripts/realtime4dv/extract_images.py --data_root data/renbody/0012_01
python scripts/realtime4dv/extract_masks.py --data_root data/renbody/0012_01
evc -t gui -c configs/projects/realtime4dv/rendering/4k4d_0012_01.yaml,configs/specs/video.yaml
Log
(easyvolcap) miu@garnet:~/mmd/4K4D$ evc -t gui -c configs/projects/realtime4dv/rendering/4k4d_0012_01.yaml,configs/specs/video.yaml
2023-12-23 08:46:38.110584 easyvolcap.scripts.main -> preflight: Starting experiment: 4k4d_0012_01, command: main.py:80 gui
2023-… easyv… Loading mask bytes for data/renbody/0012_01/masks_libx265 VAL 100% ━━━━━━ 9,000… 0:00:… < 0:00:… 170.9 p…08:47… -> it/s
load_…
2023-… easyv… Loading imgs bytes for data/renbody/0012_01/images_libx265 VAL 100% ━━━━━━ 9,000… 0:01:… < 0:00… 46.42 p…08:49… -> it/s
load_…
2023-12… easyvol… Cropping msks imgs for data/renbody/0012_01 VAL 100% ━━━━━━━━━ 9,000/9… 0:00:34 < 0:00:00 252.9 p…08:49:3… -> it/s
decode_…
2023-12-23 08:49:41.726240 easyvolcap.runners.visualizers.volumetric_video_visualizer volumetric_video_visualizer.py:76 -> __init__: Visualization output:
data/result/4k4d_0012_01/{RENDER,DEPTH}
2023-12-23 08:49:41.728334 easyvolcap.runners.recorders -> __init__: Saved config file to recorders.py:105 data/record/4k4d_0012_01/4k4d_0012_01_1703288981.yaml
2023-12-23 08:49:45.815186 easyvolcap.utils.console_utils -> inner: Runtime exception: The console_utils.py:337 following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
File "/home/miu/mmd/4K4D/easyvolcap/utils/enerf_utils.py", line 94, in
sample_geometry_feature_image
# -> B, S, P, 4
xyz1 = (xyz1[..., None, :, :] @ src_exts.mT)
xyzs = xyz1[..., :3] @ src_ixts.mT # B, S, P, 3 @ B, S, 3, 3
~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
xy = xyzs[..., :-1] / (xyzs[..., -1:] + 1e-8) # B, S, P, 2
x, y = xy.chunk(2, dim=-1) # B, S, P, 1
RuntimeError: CUDA out of memory. Tried to allocate 180.00 MiB. GPU 0
has a total capacty of 8.00 GiB of which 0 bytes is free. Including
non-PyTorch memory, this process has 17179869184.00 GiB memory in use.
Of the allocated memory 7.80 GiB is allocated by PyTorch, and 89.26 MiB
is reserved by PyTorch but unallocated. If reserved but unallocated
memory is large try setting max_split_size_mb to avoid fragmentation.
See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮│ /home/miu/mmd/4K4D/easyvolcap/utils/console_utils.py:334 in inner ││ ││ ❱ 334 │ │ │ return func(*args, **kwargs) ││ ││ /home/miu/mmd/4K4D/easyvolcap/engine/registry.py:56 in inner ││ ││ ❱ 56 │ │ return call_from_cfg(func, cfg) ││ ││ /home/miu/mmd/4K4D/easyvolcap/engine/registry.py:47 in call_from_cfg ││ ││ ❱ 47 │ return func(**call_args) ││ ││ /home/miu/mmd/4K4D/easyvolcap/scripts/main.py:119 in gui ││ ││ ❱ 119 │ viewer: "VolumetricVideoViewer" = RUNNERS.build(viewer_cfg, runner=runner) # will start the window ││ ││ /home/miu/mmd/4K4D/easyvolcap/engine/registry.py:300 in build ││ ││ ❱ 300 │ │ return self.build_func(*args, **kwargs, registry=self) ││ ││ /home/miu/mmd/4K4D/easyvolcap/engine/registry.py:131 in build_from_cfg ││ ││ ❱ 131 │ return call_from_cfg(obj_cls, args) ││ ││ /home/miu/mmd/4K4D/easyvolcap/engine/registry.py:47 in call_from_cfg ││ ││ ❱ 47 │ return func(**call_args) ││ ││ /home/miu/mmd/4K4D/easyvolcap/runners/volumetric_video_viewer.py:113 in __init__ ││ ││ ❱ 113 │ │ self.epoch = self.runner.load_network() # load weights only (without optimizer states) ││ ││ /home/miu/mmd/4K4D/easyvolcap/runners/volumetric_video_runner.py:179 in load_network ││ ││ ❱ 179 │ │ epoch = load_network(model=self.model, # only loading the network, without recorder? ││ ││ /home/miu/mmd/4K4D/easyvolcap/utils/net_utils.py:2657 in load_network ││ ││ ❱ 2657 │ (model if not isinstance(model, DDP) else model.module).load_state_dict(pretrained_model, strict=strict) ││ ││ /home/miu/anaconda3/envs/easyvolcap/lib/python3.11/site-packages/torch/nn/modules/module.py:2138 in load_state_dict ││ ││ ❱ 2138 │ │ load(self, state_dict) ││ ││ /home/miu/anaconda3/envs/easyvolcap/lib/python3.11/site-packages/torch/nn/modules/module.py:2126 in load ││ ││ ❱ 2126 │ │ │ │ │ load(child, child_state_dict, child_prefix) ││ ││ /home/miu/anaconda3/envs/easyvolcap/lib/python3.11/site-packages/torch/nn/modules/module.py:2131 in load ││ ││ ❱ 2131 │ │ │ │ out = hook(module, incompatible_keys) ││ ││ /home/miu/anaconda3/envs/easyvolcap/lib/python3.11/site-packages/torch/utils/_contextlib.py:115 in decorate_context ││ ││ ❱ 115 │ │ │ return func(*args, **kwargs) ││ ││ /home/miu/mmd/4K4D/easyvolcap/models/samplers/super_charged_r4dv.py:375 in _load_state_dict_post_hook ││ ││ ❱ 375 │ │ │ rgbw, cent = l_average_single_frame(i) ││ ││ /home/miu/anaconda3/envs/easyvolcap/lib/python3.11/site-packages/torch/utils/_contextlib.py:115 in decorate_context ││ ││ ❱ 115 │ │ │ return func(*args, **kwargs) ││ ││ /home/miu/mmd/4K4D/easyvolcap/models/samplers/super_charged_r4dv.py:148 in average_single_frame ││ ││ ❱ 148 │ ibrs_rgbs = sample_geometry_feature_image( │╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
File "/home/miu/mmd/4K4D/easyvolcap/utils/enerf_utils.py", line 94, in sample_geometry_feature_image
# -> B, S, P, 4
xyz1 = (xyz1[..., None, :, :] @ src_exts.mT)
xyzs = xyz1[..., :3] @ src_ixts.mT # B, S, P, 3 @ B, S, 3, 3
~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
xy = xyzs[..., :-1] / (xyzs[..., -1:] + 1e-8) # B, S, P, 2
x, y = xy.chunk(2, dim=-1) # B, S, P, 1
RuntimeError: CUDA out of memory. Tried to allocate 180.00 MiB. GPU 0 has a total capacty of 8.00 GiB of which 0 bytes is free. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use. Of the allocated memory 7.80 GiB is allocated by PyTorch, and 89.26 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-12-23 easyvolcap.model… Caching rgbw and center 0% ━━━━━━━━━━━━━━━━━ 0/150 0:00:03 < -:--:-- ? it/s s…08:49:46.335766 ->
_load_state_dict…
*** The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
File "/home/miu/mmd/4K4D/easyvolcap/utils/enerf_utils.py", line 94, in sample_geometry_feature_image
# -> B, S, P, 4
xyz1 = (xyz1[..., None, :, :] @ src_exts.mT)
xyzs = xyz1[..., :3] @ src_ixts.mT # B, S, P, 3 @ B, S, 3, 3
~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
xy = xyzs[..., :-1] / (xyzs[..., -1:] + 1e-8) # B, S, P, 2
x, y = xy.chunk(2, dim=-1) # B, S, P, 1
RuntimeError: CUDA out of memory. Tried to allocate 180.00 MiB. GPU 0 has a total capacty of 8.00 GiB of which 0 bytes
is free. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use. Of the allocated memory 7.80
GiB is allocated by PyTorch, and 89.26 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory islarge try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and
PYTORCH_CUDA_ALLOC_CONF
> /home/miu/mmd/4K4D/easyvolcap/models/samplers/super_charged_r4dv.py(148)average_single_frame()
146
147 # Compute projected color of every image, using the original size image
--> 148 ibrs_rgbs = sample_geometry_feature_image(
149 xyz,
150 src_feat_inps,
