Latest docker images dont run instant-ngp on 3090 series cards. - CUDA error: no kernel image is available for execution on the device

[kurtjcu]

Describe the bug Current docker images with tags "main", "1.0.1", and "1.0.0" crash when training.

RuntimeError: CUDA error: no kernel image is available for execution on the device

To Reproduce Steps to reproduce the behavior: Do this on a machine with a 3090 (ubuntu server, nvidia driver support up to cuda 12.3)

wandb docker-run \
            --gpus "device=1" \
            -u $(id -u) \
            -v "datadir:/workspace" \
            -v '/home/user/.cache/:/home/user/.cache/' \
            -p 7017:7007 \
            --rm -it --shm-size=12gb \
            dromni/nerfstudio:1.0.1 ns-train instant-ngp \
            --vis wandb \
            --data "/workspace" 

Expected behavior Using the above command with container tag "0.3.4" functions correctly