Using sensors with low data rate in synchronous mode

[ksuszka]

CARLA version: 0.9.15

Is there a reliable way in the API to get data from sensors with data generation rate lower than the simulation rate?

If I'm running a simulation in the synchronous mode with 20Hz rate and I have a sensor with 8Hz rate then CARLA will return the sensor data every 2 or 3 frames as expected. However, AFAIK there is no way in the API to tell if for given frame I should wait for the data or not. All examples of using synchronous mode I've seen assume that for all sensors data is returned for every frame.

[PatrickPromitzer]

I don't think you can do that with the API.

You could calculate the simulation time and calculate the time you expect the data.

The only other way I can think of is using 20Hz, returning the sensor data, and removing each frame you don't want.

Simple Example:

client = carla.Client("127.0.0.1", 2000)
world = client.get_world()
fixed_delta_seconds = 0.05
sensor_delta_seconds = 0.125

next_sensor_data_time = sensor_delta_seconds
current_simulation_time = 0.0
while True:
    world.tick()
    if next_sensor_data_time <= current_simulation_time:
        next_sensor_data_time = round(next_sensor_data_time + sensor_delta_seconds, 4)
        print("Data       expected:", current_simulation_time)
    else:
        print("Data not expected:", current_simulation_time)

    current_simulation_time = round(current_simulation_time + fixed_delta_seconds, 4)

For calculation the time of the expected data: To be accurate, it would be better to save the simulation time at which you added the sensor and get the simulation time directly from carla.

For getting each sensor data: To get more accurate sensor data, because in that example you have the sensor data of the simulation time 0.4 and not 0.375, you need to set a different frequency on the sensor.

[ksuszka]

Unfortunately, trying to predict if data should arrive in a given frame is very unreliable. This is exactly what current version of ros-bridge tries to do in the synchronous mode and if fails miserably. I started to gather data for different combinations of sensor tick, but abandon this at is seemed pointless. A few results I wrote down:

carla tick	sensor_tick	frames_till_error
0.05	0.1499999	51184
0.05	0.1499990	>2000000 (test aborted)
0.05	0.1	>2000000 (test aborted)
0.05	0.05	>2000000 (test aborted)
0.05	0.2	~1000

If I understand correctly Carla calculates simulation time additively, so changing someting in the simulation can influence this calculation slightly. Even updating Carla version from 0.9.14 to 0.9.15 changes results (8Hz GNSS hangs simulation every few seconds on my machine in Carla 0.9.15 with ros-bridge in synchronous mode, as ros-bridge calculates that it should wait for data in the wrong frame).

At the moment only reliable approach I know is the one you mentioned - setting sensor tick below or equal to simulation tick and dropping data manually. But it can be expensive to calculate all sensors every frame.

If there is no other way to synchronize data in the synchronous mode then it seems like a serious oversight in the API as it prevents creating reliable API clients.

[PatrickPromitzer]

I noticed that the Simulation time of Carla is not accurate, and maybe someone else knows more about this topic. (I try to answer some issues and help some people out.) I even noticed that I had to add a round() in my example because Python will give me a 0.49999999 value just even with adding 0.125 each time.

I remember something I saw in the documentation. Did you try to get an accurate timing with a world snapshot? https://carla.readthedocs.io/en/latest/python_api/#carla.WorldSnapshot

I don't know if the ros-bridge is using the time on the snapshot or is trying to calculate the time itself.

It could be that the timestamp of world.get_snapshot() is as accurate as you need it, and until there is a fix to that problem, it may be worth a try.

[ksuszka]

Yep, it uses time from the snapshot: https://github.com/carla-simulator/ros-bridge/blob/e9063d97ff5a724f76adbb1b852dc71da1dcfeec/carla_ros_bridge/src/carla_ros_bridge/bridge.py#L271 And here is the class which tries to predict when it should wait for sensor data before triggering the next simulation tick: https://github.com/carla-simulator/ros-bridge/blob/e9063d97ff5a724f76adbb1b852dc71da1dcfeec/carla_ros_bridge/src/carla_ros_bridge/sensor.py#L178, https://github.com/carla-simulator/ros-bridge/blob/e9063d97ff5a724f76adbb1b852dc71da1dcfeec/carla_ros_bridge/src/carla_ros_bridge/sensor.py#L222. In all its calculations It uses only time returned by Carla, but without proper synchronization mechanism it is not enough to accurately predict in which frame it should wait for sensor data.

[ReconGit]

Yes, there is a way to do this, it's easier if your rates match (just example): 20 hz and 10 hz -> wait for sensor data every 2 frames. But you could also work with 20 hz and 8 hz, you'd just have to check only every 3rd frame and get data until queue is empty, otherwise, if you checked every 2nd frame, you'd get stuck because no data would be available. I use something like this (these are just snippets):

# create queues for each camera
async_queues = [queue.Queue() for _ in cam_list]
sync_queues = [queue.Queue() for _ in cam_list]
for i, cam in enumerate(cam_list):
    cam.listen(async_queues[i].put)

# *in simulation loop*
# *if a certain time has been simulated*
# wait for all cameras to produce an image
for i, sync_queue in enumerate(sync_queues):
    sync_queue.put(async_queues[i].get())

[PatrickPromitzer]

This only works in an ideal setting, or if you wait for each entry for a long time.

If you know that your date can take 0.05 - 0.01 seconds to get in the queue, you could wait for 0.2 seconds. If it takes 0.11 seconds for some reason, you have to wait longer. In the end, you have to wait 0.5 seconds, if you could have your data in 0.05 seconds, and if carla takes 2 seconds for some reasons, you have a problem again.

If you don't wait at all, you lose data. If you look for an empty queue just 0.001 seconds after you make a world.tick(), you had too little time for the data to get back to your script.

For example: world.tick() = every 0.001 seconds transferring your cam data = takes 0.005 seconds (just an example)

Sending data at 0.0, and 0.1 for each step/frame 0.000 = Here should the date be received 0.001 = data is not here 0.002 = data is not here 0.003 = data is not here 0.004 = data is not here 0.005 = you get the data from 0.0

[ReconGit]

That is true, that's why I said it's easier with matching rates. With matching rates you don't check until queue is empty, the .get() will be blocking and will wait until data is received, therefore it will be nicely synced. But even if you look at an empty queue and check every 3rd frame, you will receive data eventually, there is no loss, it will be put into your async queue, though could be desynced yeah, which for some scenarios might be unacceptable, there, you just need to use matching rates as far as I know.