sionna icon indicating copy to clipboard operation
sionna copied to clipboard

Kernel died on CoverageMap with 1080Ti GPU ad Docker

Open lillidith opened this issue 1 year ago • 6 comments

Hi, I set up the Sionna environment with docker using the associated makefile. The Notebook Sionna_Ray_Tracing_Coverage_Map.ipynb con load the libraries, and the GPU is correctly configured. The scene is loaded successfully, but the kernel died when I computed:

cm = scene.coverage_map(max_depth=2,           # Maximum number of ray scene interactions
                        num_samples=int(2e6), # If you increase: less noise, but more memory required
                        cm_cell_size=(5, 5),   # Resolution of the coverage map
                        cm_center=[0, 0, 0],   # Center of the coverage map
                        cm_size=[400, 400],    # Total size of the coverage map
                        cm_orientation=[0, 0, 0]) # Orientation of the coverage map, e.g., could be also vertical

The same configuration works fine and smooth on Server with RTX A5000 and A100 GPUs.

Server configuration

sudo apt-get update
sudo apt-get install -y cuda-drivers-560
sudo apt-get -y install cuda-toolkit-12-6
sudo usermod -aG docker myuser
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

lillidith avatar Nov 06 '24 09:11 lillidith

Hello @lillidith,

What error message was shown in the terminal when the kernel died? The most likely cause is running out of memory. You could try reducing num_samples and trying again.

merlinND avatar Nov 06 '24 10:11 merlinND

Same issue on a workstation with 1080Ti 11Gb:

scene = load_scene() # Load empty scene

scene.tx_array = PlanarArray(num_rows=1,
                             num_cols=1,
                             vertical_spacing=0.5,  # relative to wavelength
                             horizontal_spacing=0.5,  # relative to wavelength
                             pattern="iso",
                             polarization="V")
scene.rx_array = scene.tx_array

tx0 = Transmitter(name='tx0',
                  position=[15, -10, 20],
                  orientation=[np.pi*5/6, 0, 0],
                  power_dbm=10)
scene.add(tx0)

cm = scene.coverage_map(max_depth=5,           # Maximum number of ray scene interactions
                        num_samples=int(100), # If you increase: less noise, but more memory required
                        cm_cell_size=(5, 5),   # Resolution of the coverage map
                        cm_center=[0, 0, 0],   # Center of the coverage map
                        cm_size=[50, 50],    # Total size of the coverage map
                        cm_orientation=[0, 0, 0])

.... of corse all seem perfect in workstation with A100 40Gb but it's not a memory pb ( num_samples=int(100) )! Compute capability? Or incompatible GPU ? I dont find a minimun requirement on hardware in the Sionna Doc site.

Fedomer avatar Nov 06 '24 11:11 Fedomer

What error message was shown in the terminal when the kernel died?

merlinND avatar Nov 06 '24 11:11 merlinND

By the way, as a workaround you can set the environment variable CUDA_VISIBLE_DEVICE="" before launching the Jupyter server to fall back on the CPU backend. Depending on your hardware, it could be a viable alternative.

merlinND avatar Nov 06 '24 11:11 merlinND

it's seem that with 2080Ti with 11 Gb it works so the the 1080Ti 11Gb card it's to old for sionna.

Fedomer avatar Nov 22 '24 12:11 Fedomer

We may be able to find out the root cause if you share the error message printed in the terminal when the kernel dies.

merlinND avatar Jan 03 '25 11:01 merlinND

Closing due to inactivity

SebastianCa avatar Aug 21 '25 09:08 SebastianCa