RAW Image Pipeline

Image processing utilities used for cameras that provide RAW data, such as the Alphasense Core unit.

Maintainers: Matias Mattamala ([email protected])

Contributors: Matias Mattamala, Timon Homberger, Marco Tranzatto, Samuel Zimmermann, Lorenz Wellhausen, Shehryar Khattak, Gabriel Waibel

License

This source code is released under a MIT License.

raw_image_pipeline_white_balance relies on Shane Yuan's AutoWhiteBalance package licensed under GNU.

raw_image_pipeline_python relies on Pascal Thomet's cvnp, licensed under MIT as well.

Build

Overview

Packages

1. raw_image_pipeline: ROS-independent implementation of the pipeline.
2. raw_image_pipeline_python: Python bindings for raw_image_pipeline.
3. raw_image_pipeline_ros: ROS interface to run the processing pipeline.
4. raw_image_pipeline_white_balance: Additional white balance algorithm built upon Shane Yuan's code, based on Barron's (1, 2).

Pipeline

The package implements diferent modules that are chained together to process the RAW images. Each can be disabled and the image will be processed by the subsequent modules.

• Debayer: auto, bayer_bggr8, bayer_gbrg8, bayer_grbg8, bayer_rggb8
• Flip: Flips the image 180 degrees
• White balance: simple, grey_world, learned (from OpenCV), ccc (from raw_image_pipeline_white_balance package), pca (custom implementation)
• Color correction: Simple color correction based on a mixing BGR 3x3 matrix.
• Gamma correction: default (from OpenCV), custom (custom implementation)
• Vignetting correction: Removes the darkening effect of the lens toward the edges of the image by applying a polynomial mask.
• Color enhancement: Converts the image to HSV and applies a gain to the S (saturation) channel.
• Undistortion: Corrects the image given the camera calibration file.

color_calibration.py [-h] -i INPUT -r REF [-o OUTPUT_PATH] [-p PREFIX] [--loss LOSS] [--compute_bias]

Performs color calibration between 2 images, using ArUco 4X4

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        Input image (to be calibrated), or folder with reference images
  -r REF, --ref REF     Reference image to perform the calibration
  -o OUTPUT_PATH, --output_path OUTPUT_PATH
                        Output path to store the file. Default: current path
  -p PREFIX, --prefix PREFIX
                        Prefix for the calibration file. Default: none
  --loss LOSS           Loss used in the optimization. Options: 'linear', 'soft_l1', 'huber', 'cauchy', 'arctan'
  --compute_bias        If bias should be computed

Requirements and compilation

Dependencies

sudo apt install libyaml-cpp-dev

cd ~/git
git clone [email protected]:catkin/catkin_simple.git
git clone [email protected]:ethz-asl/glog_catkin.git
git clone [email protected]:leggedrobotics/pybind11_catkin.git
cd ~/catkin_ws/src
ln -s ../../git/catkin_simple .
ln -s ../../git/glog_catkin .
ln -s ../../git/pybind11_catkin .

If you need CUDA support, you need to build OpenCV with CUDA. Check the instructions below

Build raw_image_pipeline_ros

To build the ROS package:

catkin build raw_image_pipeline_ros

If you also need the Python bindings, run:

catkin build raw_image_pipeline_python

CUDA support

If you are using a Jetson or another GPU-enabled computer and want to exploit the GPU, you need to compile OpenCV with CUDA support. Clone the opencv_catkin package, which setups OpenCV 4.2 by default.

cd ~/git
git clone [email protected]:ori-drs/opencv_catkin.git
cd ~/catkin_ws/src
ln -s ../../git/opencv_catkin .
cd ~/catkin_ws

:warning: Before compiling, you need to confirm the compute capability of your NVidia GPU, which you can check in this website or the CUDA wikipedia page.

Compilation on Jetson Xavier board (compute capability 7.2)

catkin build opencv_catkin --cmake-args -DCUDA_ARCH_BIN=7.2
source devel/setup.bash

Compilation on Jetson Orin board (compute capability 8.7)

catkin build opencv_catkin --cmake-args -DCUDA_ARCH_BIN=8.7
source devel/setup.bash

Compilation on other platforms (e.g. laptops, desktops)

There are some extra considerations if you plan to compile OpenCV with CUDA in your working laptop/desktop:

1. Compute capability may be different for your GPU: Please check the aforementioned websites to set the flag correctly.
2. The opencv_catkin default flags are the minimum: Graphical support libraries (such as GTK) are disabled, so you cannot use methods such as cv::imshow. If you want to enable it, you can check the flags in the CMakeLists of opencv_catkin
3. Default OpenCV version is 4.2: The package installs by default OpenCV 4.2, which was the version compatible with ROS melodic. This can be changed by modyfing the CMakeLists of opencv_catkin as well.

OpenCV's compilation will take a while - get a coffee in the meantime. When it's done, you can rebuild raw_image_pipeline_ros.

Troubleshooting

• If you get errors due to glog, remove glog_catkin, compile opencv_catkin using the system's glog, and then build raw_image_pipeline_ros (which will compile glog_catkin)
• If OpenCV fails due to CUDA errors, confirm that you compiled using the right compute capability for your GPU.
• If you are using older versions of CUDA (10.x and before), they may require older GCC versions. For example, to use GCC 7 you can use:

catkin build opencv_catkin --cmake-args -DCUDA_ARCH_BIN=<your_compute_capability> -DCMAKE_C_COMPILER=/usr/bin/gcc-7

Run the node

To run, we use the same launch file as before:

roslaunch raw_image_pipeline_ros raw_image_pipeline_node.launch

This launchfile was setup for Alphasense cameras. The parameters can be inspected in the launch file itself.

Alphasense-specific info

Setup

Please refer to Alphasense Setup for the instructions to setup the host PC where the Alphasense will be connected. For further information you can refer the official manual.