NVlabs
UnknownError: {{function_node __wrapped__FloorMod_device_/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:FloorMod]
Every time I try to get a coverage map, I get the following error: UnknownError: {{function_node _wrapped__FloorMod_device/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:FloorMod] I am on Windows Native, using TF 2.10.1, Sionna 0.14.
Things work ok when I try to compute paths. I only observed this problem with coverage_map() function.
And this is the full error tree:
---------------------------------------------------------------------------
UnknownError Traceback (most recent call last)
Cell In[1], line 28
25 tx.look_at([40,80,1.5])
27 # Compute coverage map
---> 28 cm = scene.coverage_map(max_depth=8)
30 # Show coverage map
31 cm.show()
File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\scene.py:908, in Scene.coverage_map(self, rx_orientation, max_depth, cm_center, cm_orientation, cm_size, cm_cell_size, combining_vec, precoding_vec, num_samples, seed)
903 seed = tf.random.uniform(shape=(), maxval=0x7FFFFFFF,
904 dtype=tf.int32)
906 # Compute the coverage map using the solver
907 # [num_sources, num_cells_x, num_cells_y]
--> 908 output = self._solver_cm(max_depth=max_depth,
909 rx_orientation=rx_orientation,
910 cm_center=cm_center,
911 cm_orientation=cm_orientation,
912 cm_size=cm_size,
913 cm_cell_size=cm_cell_size,
914 combining_vec=combining_vec,
915 precoding_vec=precoding_vec,
916 num_samples=num_samples,
917 seed=seed)
919 return output
File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\solver_cm.py:138, in SolverCoverageMap.__call__(self, max_depth, rx_orientation, cm_center, cm_orientation, cm_size, cm_cell_size, combining_vec, precoding_vec, num_samples, seed)
129 # Shoot and bounce
130 # hit : [max_depth+1, num_tx, samples_per_tx]
131 # hit_point : [max_depth+1, num_tx, samples_per_tx, 3]
132 # objects : [max_depth, num_tx, samples_per_tx]
133 # int_point : [max_depth, num_tx, samples_per_tx, 3]
134 rays = self._shoot_and_bounce(cm_center, cm_orientation, cm_size,
135 max_depth,
136 num_samples, seed)
--> 138 cm = self._rays_2_coverage_map(
139 cm_center, cm_orientation, cm_size, cm_cell_size,
140 rx_orientation,
141 combining_vec, precoding_vec,
142 *rays,
143 etas)
144 return cm
File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\solver_cm.py:975, in SolverCoverageMap._rays_2_coverage_map(self, cm_center, cm_orientation, cm_size, cm_cell_size, rx_orientation, combining_vec, precoding_vec, hit, hit_point, objects, int_point, normals, relative_permittivity)
973 theta_t_0, phi_t_0 = theta_t, phi_t
974 # [num_tx, samples_per_tx]
--> 975 ray_contrib = self._ray_contribution(mat_t_hit,
976 k_i_0, theta_t_0, phi_t_0,
977 last_step_norm, theta_r, phi_r,
978 rx_orientation,
979 combining_vec, precoding_vec)
980 # Zeros the contribution of rays that didn't hit the coverage map
981 # [num_tx, samples_per_tx]
982 ray_contrib = tf.where(hit_, ray_contrib,
983 tf.zeros_like(ray_contrib))
File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\solver_cm.py:636, in SolverCoverageMap._ray_contribution(self, mat_t, k_t, theta_t, phi_t, k_r, theta_r, phi_r, rx_orientation, combining_vec, precoding_vec)
633 tx_ant_fields_hat = []
634 for pattern in tx_patterns:
635 # [num_tx, samples_per_tx, 2]
--> 636 tx_ant_f = tf.stack(pattern(theta_prime_t, phi_prime_t), axis=-1)
637 tx_ant_fields_hat.append(tx_ant_f)
639 rx_ant_fields_hat = []
File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\antenna.py:122, in Antenna.pattern_with_slant_angle.<locals>.<lambda>(theta, phi)
120 def pattern_with_slant_angle(self, pattern, slant_angle):
121 """Applies slant angle to antenna pattern"""
--> 122 return lambda theta, phi: pattern(theta, phi, slant_angle, self._dtype)
File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\antenna.py:500, in tr38901_pattern(theta, phi, slant_angle, dtype)
497 phi = tf.cast(phi, rdtype)
499 # Wrap phi to [-PI,PI]
--> 500 phi = tf.math.floormod(phi+PI, 2*PI)-PI
502 if not theta.shape==phi.shape:
503 raise ValueError("theta and phi must have the same shape.")
File ~\anaconda3\envs\RLHF\lib\site-packages\tensorflow\python\ops\gen_math_ops.py:4002, in floor_mod(x, y, name)
4000 return _result
4001 except _core._NotOkStatusException as e:
-> 4002 _ops.raise_from_not_ok_status(e, name)
4003 except _core._FallbackException:
4004 pass
File ~\anaconda3\envs\RLHF\lib\site-packages\tensorflow\python\framework\ops.py:7209, in raise_from_not_ok_status(e, name)
7207 def raise_from_not_ok_status(e, name):
7208 e.message += (" name: " + name if name is not None else "")
-> 7209 raise core._status_to_exception(e) from None
UnknownError: {{function_node __wrapped__FloorMod_device_/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:FloorMod]
Hi,
Are you using XLA? If so, it would explain why you get this error, as Scene.coverage_map() does not work with XLA.
Thanks for the response. I did not explicitly use XLA. Maybe it is activated in Sionna library? How can I disable it?
Sionna does not activate XLA. Could you try with the latest version of Sionna (0.16.0)?
Upgraded Sionna to 0.16 but I got the same error. I tried to put sionna.Config.xla_compat = False before coverage map function but it did not solve the issue.
Sure. I get the error after executing the last line. The execution runs for a while but outputs an error eventually.
import sionna
import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import time
from sionna.rt import load_scene, Transmitter, Receiver, PlanarArray, Camera
from sionna.channel import cir_to_ofdm_channel, subcarrier_frequencies, OFDMChannel, ApplyOFDMChannel, CIRDataset
from sionna.nr import PUSCHConfig, PUSCHTransmitter, PUSCHReceiver
from sionna.utils import compute_ber, ebnodb2no, PlotBER
from sionna.ofdm import KBestDetector, LinearDetector
from sionna.mimo import StreamManagement
gpus = tf.config.list_physical_devices('GPU')
if gpus:
try:
tf.config.experimental.set_memory_growth(gpus[0], True)
except RuntimeError as e:
print(e)
tf.get_logger().setLevel('ERROR')
tf.random.set_seed(1) # Set global random seed for reproducibility
scene = load_scene(sionna.rt.scene.munich)
scene.preview()
scene.tx_array = PlanarArray(num_rows=8,
num_cols=2,
vertical_spacing=0.7,
horizontal_spacing=0.5,
pattern="tr38901",
polarization="VH")
scene.rx_array = PlanarArray(num_rows=1,
num_cols=1,
vertical_spacing=0.5,
horizontal_spacing=0.5,
pattern="dipole",
polarization="cross")
tx = Transmitter(name="tx",
position=[8.5,21,27],
orientation=[0,0,0])
scene.add(tx)
rx = Receiver(name="rx",
position=[45,90,1.5],
orientation=[0,0,0])
scene.add(rx)
tx.look_at(rx)
print(scene.transmitters)
print(scene.receivers)
paths = scene.compute_paths(max_depth=5,
num_samples=1e6)
Hi, I cannot reproduce this error.
Could you describe your full system configuration (OS, GPU, TF, Sionna, Python, etc.)?
There is no coverage computation in this code snippet. You get the error in the compute_path call?
Could you upgrade to TF 2.13?
Yeah, actually it was also giving the same error in compute_path call. However, I somehow solved the issue. When I copied the "nvvm" folder from CUDA/vx.x to my working directory, I stopped getting the error. But, I have still no idea what the problem is. I am sure that I have cuda directory in the Path variables. If you have any idea on that (like what causes the problem) and a better solution than my work-around I will be glad. Otherwise I can close the issue.
