gaussian-splatting icon indicating copy to clipboard operation
gaussian-splatting copied to clipboard

Error rtx 4080, Expected 4-dimensional input for 4-dimensional weight but got 3-dimensional input of size

Open aphixe opened this issue 10 months ago • 11 comments

C:\Users\aphixe\anaconda3\envs\gaussian_splatting\lib\site-packages\torch\cuda_init_.py:106: UserWarning: NVIDIA GeForce RTX 4080 with CUDA capability sm_89 is not compatible with the current PyTorch installation. The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37. If you want to use the NVIDIA GeForce RTX 4080 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name)) Output folder: ./output/69948d25-2 [14/10 18:51:11] Tensorboard not available: not logging progress [14/10 18:51:11] Reading camera 11/11 [14/10 18:51:11] Converting point3d.bin to .ply, will happen only the first time you open the scene. [14/10 18:51:11] Loading Training Cameras [14/10 18:51:11] Loading Test Cameras [14/10 19:05:40] Number of points at initialisation : 8040 [14/10 19:05:40] Training progress: 0%| | 0/20000 [00:00<?, ?it/s]Traceback (most recent call last): File "train.py", line 216, in training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from) File "train.py", line 89, in training loss = (1.0 - opt.lambda_dssim) * Ll1 + opt.lambda_dssim * (1.0 - ssim(image, gt_image)) File "E:\Documents\gaussian-splatting\utils\loss_utils.py", line 41, in ssim return _ssim(img1, img2, window, window_size, channel, size_average) File "E:\Documents\gaussian-splatting\utils\loss_utils.py", line 44, in _ssim mu1 = F.conv2d(img1, window, padding=window_size // 2, groups=channel) RuntimeError: Expected 4-dimensional input for 4-dimensional weight [3, 1, 11, 11], but got 3-dimensional input of size [3, 610, 1402] instead Training progress: 0%|

aphixe avatar Oct 15 '23 02:10 aphixe

Running into the same problem with NVIDIA GeForce GTX 1650:

nvidia-smi                                                                                                                                                                                                                   :) 
Tue Oct 17 16:00:30 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01             Driver Version: 535.113.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1650        Off | 00000000:01:00.0  On |                  N/A |
| 24%   40C    P8              N/A /  75W |   1009MiB /  4096MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

Moult avatar Oct 17 '23 05:10 Moult

Running into the same bug here with NVIDIA GTX 166...

17373355 avatar Oct 23 '23 07:10 17373355

Can you print the shapes of the tensors (both the rendered image and the ground truth) before feeding them in ssim?

On Mon, Oct 23, 2023, 09:43 17373355 @.***> wrote:

Running into the same bug here with NVIDIA GTX 166...

— Reply to this email directly, view it on GitHub https://github.com/graphdeco-inria/gaussian-splatting/issues/320#issuecomment-1774605734, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACGXXYPCRGJESRK3QTM5G6DYAYN2HAVCNFSM6AAAAAA6AVEQA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZUGYYDKNZTGQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

grgkopanas avatar Oct 23 '23 07:10 grgkopanas

Edit: Just for reference. I was too excited then so I said "solved." The project can run smoothly in my environment, aka a ThinkStation. My GPU only supports very small datasets. I guess this might not solve the cases using RTX 40xx series...

My initial error report: RuntimeError: Expected 4-dimensional input for 4-dimensional weight [3, 1, 11, 11], but got 3-dimensional input of size [3, 1905, 1073] instead

Solution: Adding the following codes to loss_util.py def _ssim(img1, img2, window, window_size, channel, size_average=True): # ADDING by R.Guo # print(img1.shape) img1 = img1.view(1, 3, img1.shape[1], img1.shape[2]) img2 = img2.view(1, 3, img2.shape[1], img2.shape[2]) # print(img1.shape)

Pytorch version: pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

This is my NVIDIA: +-----------------------------------------------------------------------------+ | NVIDIA-SMI 457.85 Driver Version: 457.85 CUDA Version: 11.1 | |-------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 GeForce GTX 166... WDDM | 00000000:01:00.0 On | N/A | | 49% 61C P2 43W / 125W | 915MiB / 6144MiB | 37% Default | | | | N/A | +-------------------------------+----------------------+----------------------+

17373355 avatar Oct 23 '23 08:10 17373355

tried to add code, and its stuck at camera part. I don't know about pytorch version as I am just using recommended file.

aphixe avatar Oct 23 '23 22:10 aphixe

can confirm, added the code suggested by 17373355 but getting this error running pytorch 1.9.0+cu111 and torchvision 1.10.0+cu111 as mentioned above

(gaussian_splatting) C:\Users\matdi\Pictures\blender\gaussian splatting\gaussian-splatting>python train.py -s data\bureau1\bureau1
Optimizing
Output folder: ./output/ca53e87a-e [21/11 10:30:17]
Tensorboard not available: not logging progress [21/11 10:30:17]
Reading camera 448/448 [21/11 10:30:19]
Loading Training Cameras [21/11 10:30:19]
[ INFO ] Encountered quite large input images (>1.6K pixels width), rescaling to 1.6K.
 If this is not desired, please explicitly specify '--resolution/-r' as 1 [21/11 10:30:19]
Traceback (most recent call last):
  File "train.py", line 219, in <module>
    training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from)
  File "train.py", line 35, in training
    scene = Scene(dataset, gaussians)
  File "C:\Users\matdi\Pictures\blender\gaussian splatting\gaussian-splatting\scene\__init__.py", line 73, in __init__
    self.train_cameras[resolution_scale] = cameraList_from_camInfos(scene_info.train_cameras, resolution_scale, args)
  File "C:\Users\matdi\Pictures\blender\gaussian splatting\gaussian-splatting\utils\camera_utils.py", line 58, in cameraList_from_camInfos
    camera_list.append(loadCam(args, id, c, resolution_scale))
  File "C:\Users\matdi\Pictures\blender\gaussian splatting\gaussian-splatting\utils\camera_utils.py", line 52, in loadCam
    image_name=cam_info.image_name, uid=id, data_device=args.data_device)
  File "C:\Users\matdi\Pictures\blender\gaussian splatting\gaussian-splatting\scene\cameras.py", line 57, in __init__
    self.camera_center = self.world_view_transform.inverse()[3, :3]
RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling `cusolverDnCreate(handle)`

this is my GPU

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 546.12                 Driver Version: 546.12       CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090      WDDM  | 00000000:26:00.0  On |                  Off |
|  0%   47C    P8              30W / 450W |   1489MiB / 24564MiB |     16%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

manuelonthemic avatar Nov 21 '23 09:11 manuelonthemic

Edit: Just for reference. I was too excited then so I said "solved." The project can run smoothly in my environment, aka a ThinkStation. My GPU only supports very small datasets. I guess this might not solve the cases using RTX 40xx series...

My initial error report: RuntimeError: Expected 4-dimensional input for 4-dimensional weight [3, 1, 11, 11], but got 3-dimensional input of size [3, 1905, 1073] instead

Solution: Adding the following codes to loss_util.py def _ssim(img1, img2, window, window_size, channel, size_average=True): # ADDING by R.Guo # print(img1.shape) img1 = img1.view(1, 3, img1.shape[1], img1.shape[2]) img2 = img2.view(1, 3, img2.shape[1], img2.shape[2]) # print(img1.shape)

Pytorch version: pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

This is my NVIDIA: +-----------------------------------------------------------------------------+ | NVIDIA-SMI 457.85 Driver Version: 457.85 CUDA Version: 11.1 | |-------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 GeForce GTX 166... WDDM | 00000000:01:00.0 On | N/A | | 49% 61C P2 43W / 125W | 915MiB / 6144MiB | 37% Default | | | | N/A | +-------------------------------+----------------------+----------------------+

it did work. Thanks

zero-joke avatar Nov 22 '23 12:11 zero-joke

Edit: Just for reference. I was too excited then so I said "solved." The project can run smoothly in my environment, aka a ThinkStation. My GPU only supports very small datasets. I guess this might not solve the cases using RTX 40xx series...

My initial error report: RuntimeError: Expected 4-dimensional input for 4-dimensional weight [3, 1, 11, 11], but got 3-dimensional input of size [3, 1905, 1073] instead

Solution: Adding the following codes to loss_util.py def _ssim(img1, img2, window, window_size, channel, size_average=True): # ADDING by R.Guo # print(img1.shape) img1 = img1.view(1, 3, img1.shape[1], img1.shape[2]) img2 = img2.view(1, 3, img2.shape[1], img2.shape[2]) # print(img1.shape)

Pytorch version: pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

This is my NVIDIA: +-----------------------------------------------------------------------------+ | NVIDIA-SMI 457.85 Driver Version: 457.85 CUDA Version: 11.1 | |-------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 GeForce GTX 166... WDDM | 00000000:01:00.0 On | N/A | | 49% 61C P2 43W / 125W | 915MiB / 6144MiB | 37% Default | | | | N/A | +-------------------------------+----------------------+----------------------+

solvded, torch 1.10+cu11.1, so the reason was the absence of a batch dimension.

haofengsiji avatar Feb 01 '24 09:02 haofengsiji

It did work. Thanks!

Edit: Just for reference. I was too excited then so I said "solved."编辑:仅供参考。我太兴奋了所以我说“解决了。“ The project can run smoothly in my environment, aka a ThinkStation. My GPU only supports very small datasets. I guess this might not solve the cases using RTX 40xx series... 该项目可以在我的环境中顺利运行,又名ThinkStation。我的GPU只支持非常小的数据集。我猜这可能无法解决使用RTX 40xx系列的情况.

My initial error report: 我的初始错误报告: RuntimeError: Expected 4-dimensional input for 4-dimensional weight [3, 1, 11, 11], but got 3-dimensional input of size [3, 1905, 1073] instead RuntimeError:应为4维权重[3,1,11,11]输入4维,但得到的却是大小为[3,1905,1073]的3维输入

Solution: Adding the following codes to loss_util.py解决方案:将以下代码添加到loss_util.py def _ssim(img1, img2, window, window_size, channel, size_average=True): def _ssim(img1,img2,window,window_size,channel,size_average=True): # ADDING by R.Guo 作者:R.Guo # print(img1.shape)  # print(img1.shape) img1 = img1.view(1, 3, img1.shape[1], img1.shape[2]) img1 = img1.view(1,3,img1.shape[1],img1.shape[2]) img2 = img2.view(1, 3, img2.shape[1], img2.shape[2]) img2 = img2.view(1,3,img2.shape[1],img2.shape[2]) # print(img1.shape)  # print(img1.shape)

Pytorch version: Pytorch版本: pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

This is my NVIDIA: 这是我的NVIDIA: +-----------------------------------------------------------------------------+ | NVIDIA-SMI 457.85 Driver Version: 457.85 CUDA Version: 11.1 | | NVIDIA SMI 457.85驱动程序版本:457.85 CUDA版本:11.1| |-------------------------------+----------------------+----------------------+ |———————————————————————————————+——————————————————————+——————————————————————+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | GPU名称TCC/WDDM|总线ID显示A|易失性不正确ECC| | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | 风扇温度性能功率:使用/上限|内存使用情况|GPU实用程序计算M. | | | | MIG M. | |===============================+======================+======================| | 0 GeForce GTX 166... WDDM | 00000000:01:00.0 On | N/A | | 0 GeForce GTX 166. WDDM| 00000000:01:00.0开启时|N/A| | 49% 61C P2 43W / 125W | 915MiB / 6144MiB | 37% Default | | 49% 61 C P2 43 W/125 W| 915 MiB/6144 MiB/6144 MiB系列|37%失败率| | | | N/A | | | | N/A| +-------------------------------+----------------------+----------------------+

qianqjia avatar Mar 10 '24 11:03 qianqjia

I add these two lines of code, but raise another error Traceback (most recent call last): File "train_llff.py", line 407, in training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from, args.near) File "train_llff.py", line 174, in training loss.backward() File "/home/starak/anaconda3/envs/3dgs/lib/python3.7/site-packages/torch/_tensor.py", line 255, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/starak/anaconda3/envs/3dgs/lib/python3.7/site-packages/torch/autograd/init.py", line 149, in backward allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag RuntimeError: merge_sort: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

cuda11.1 pytorch1.9.1

Star-XingHe avatar Mar 24 '24 12:03 Star-XingHe

Edit: Just for reference. I was too excited then so I said "solved." The project can run smoothly in my environment, aka a ThinkStation. My GPU only supports very small datasets. I guess this might not solve the cases using RTX 40xx series...

My initial error report: RuntimeError: Expected 4-dimensional input for 4-dimensional weight [3, 1, 11, 11], but got 3-dimensional input of size [3, 1905, 1073] instead

Solution: Adding the following codes to loss_util.py def _ssim(img1, img2, window, window_size, channel, size_average=True): # ADDING by R.Guo # print(img1.shape) img1 = img1.view(1, 3, img1.shape[1], img1.shape[2]) img2 = img2.view(1, 3, img2.shape[1], img2.shape[2]) # print(img1.shape)

Pytorch version: pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

This is my NVIDIA: +-----------------------------------------------------------------------------+ | NVIDIA-SMI 457.85 Driver Version: 457.85 CUDA Version: 11.1 | |-------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 GeForce GTX 166... WDDM | 00000000:01:00.0 On | N/A | | 49% 61C P2 43W / 125W | 915MiB / 6144MiB | 37% Default | | | | N/A | +-------------------------------+----------------------+----------------------+

This worked for me too, thank you.

BenjaminJaume avatar Apr 06 '24 07:04 BenjaminJaume