How to disable image replication on the borders? How to change the size of the simulated images?

[jinho88]

When I try to use various images to your ESIM, Especially, when the width of image is much bigger than the height of the image, your ESIM cropped the width of image and duplicate image on the top and bottom of the image. I think your ESIM try to make the image specific size. Can I change of that?

I attached the input image and visualization result image.

[supitalp]

Hello, The planar renderer maps your input image to a virtual plane, and then simulates a camera moving in front of this plane. The intrinsic parameters (i.e. sensor resolution and focal lengths) need to be specified in a calibration file (e.g. https://github.com/uzh-rpg/rpg_esim/blob/master/event_camera_simulator/esim_ros/cfg/calib/pinhole_mono_nodistort.yaml).

To disable the image duplication problem you mentioned, you can set the flag --renderer_extend_border to 0.

In your case, the image width is significantly larger than the image height, thus the default sensor size (240 x 180), which is set in the calibration file, is not very well adapted to your input image size. In this case, you can set the virtual sensor size to roughly match the width/height ratio of your input image.

For example, the following calibration file would work well:

Calibration file: shaking_example.yaml

label: "simulated_camera"
id: 433585ebda8d1431223927a14788a127
cameras:
- camera:
    label: dvs0
    id: a0fba5412e961934d842d3a2a78e5cba
    line-delay-nanoseconds: 0
    image_height: 188
    image_width: 621
    type: pinhole
    intrinsics:
      cols: 1
      rows: 4
      data: [310.5, 310.5, 310.5, 94]
    distortion:
      type: none
      parameters:
        cols: 1
        rows: 0
        data: []
  T_B_C:
    cols: 4
    rows: 4
    data: [1, 0, 0,  0,
           0, -1, 0, 0,
           0,  0, -1, 0,
           0, 0, 0, 1]

Note that I set the sensor size to 621 x 188, which is half the resolution of the input image (1242 x 375), so that the aspect ratio is preserved. The focal lengths (310.5) are set to half of the image width, so that the virtual field of view of the camera is 90 degrees. This means you should also set the flag --renderer_hfov_cam_source_deg to 90.0.

As an additional note, your input image has large image size and has lots of speckle noise, so I would also recommend applying some small median blur (--renderer_preprocess_median_blur=6 for example), and some gaussian blur as well (--renderer_preprocess_gaussian_blur=0.25).

Here is a complete example configuration for you to get started with:

Config file: shaking_example.conf

--vmodule=data_provider_online_render=0,camera_simulator=0,planar_renderer=3,renderer_factory=1
--random_seed=0
--data_source=0
--path_to_output_bag=/tmp/out.bag

--contrast_threshold_pos=0.10
--contrast_threshold_neg=0.10
--contrast_threshold_sigma_pos=0.
--contrast_threshold_sigma_neg=0.
--refractory_period_ns=1000000

--exposure_time_ms=50.0
--use_log_image=1
--log_eps=0.01

--calib_filename=/home/user/sim_ws/src/rpg_esim/event_camera_simulator/esim_ros/cfg/calib/shaking_example.yaml
--renderer_hfov_cam_source_deg=90.0

--renderer_type=0
--renderer_texture=/home/user/sim_ws/src/rpg_esim/event_camera_simulator/imp/imp_planar_renderer/textures/driving.png
--renderer_preprocess_gaussian_blur=0.25
--renderer_preprocess_median_blur=3
--renderer_plane_x=0.0
--renderer_plane_y=0.0
--renderer_plane_z=-0.8
--renderer_plane_qw=0.0
--renderer_plane_qx=1.0
--renderer_plane_qy=0.0
--renderer_plane_qz=0.0
--renderer_extend_border=0
--renderer_zmin=0.2

--trajectory_type=0
--trajectory_length_s=10.0
--trajectory_sampling_frequency_hz=100
--trajectory_spline_order=5
--trajectory_num_spline_segments=100
--trajectory_lambda=0.02
--trajectory_multiplier_x=0.2
--trajectory_multiplier_y=0.2
--trajectory_multiplier_z=0.0
--trajectory_multiplier_wx=0.
--trajectory_multiplier_wy=0.
--trajectory_multiplier_wz=0.

--simulation_minimum_framerate=20.0
--simulation_imu_rate=1000.0
--simulation_adaptive_sampling_method=1
--simulation_adaptive_sampling_lambda=0.5

--ros_publisher_frame_rate=50
--ros_publisher_depth_rate=0
--ros_publisher_optic_flow_rate=0
--ros_publisher_pointcloud_rate=10
--ros_publisher_camera_info_rate=1

which should produce something like this: https://youtu.be/pc6qAJp1lto

[jinho88]

Thank you.