Enable HDR on K4A RGB stream

[VisionaryMind]

Is your feature request related to a problem? Please describe.

We have been looking at Azure Kinect as a depth camera in both our SLAM and 3D reconstruction workflows. The ROS driver certainly provides functional access to K4A data streams but unfortunately, those streams are not usable in indoor environments where sunlight is present. The gain adjustment algorithm over-compensates for light and generates completely black images everywhere outside the light source. Manually adjusting gain in the K4A Viewer does workaround this problem, but in settings where it is in a ROS tree, it is completely unusable wherever there are strong sources of light. Even a bright lamp or desk light will throw off the sensor.

Describe the solution you'd like At the very least, implementing HDR capability is a must if this camera is to be remotely functional in SLAM applications. As it stands, light must be carefully controlled in order to get remotely reasonable output and this is a shortcoming that is not seen in literally every other camera in this category, including the RealSense and Occipital depth sensors, which are able to handle wide dynamic range without the slightest problem, albeit with considerably more noise.

Describe alternatives you've considered Any other sensor at this point would be an improvement. Apart from the K4A's obvious handicap with severe latency, making it impossible to sync with in streams above 20Hz (which I see has been well documented elsewhere yet unaddressed to date), the loss of image integrity with even minor fluctuations in light are a deal breaker. To be quite honest, I am very surprised the K4A was released to production without handling this problem. Moreover, I am disappointed that for its price range, it severely underperforms compared with cameras half its cost.

Additional context Here is an image taken in a room with relatively balanced ambient light:

And here is the same scene only 5 seconds later after K4A "optimizes" its parameters:

While it is quite clear to many of us that the K4A was not released with production applications in mind (i.e. it was engineered for developer experimentation and hobbyists), nonetheless, it would be ideal if Microsoft would put more attention on details that are quite basic / fundamental in this industry. The lack of attention to the delay sync problem (which persists to this day) leaves us with little short-term expectations that this camera may be used in time-sensitive applications --- or, for that matter, any reasonably robust applications at all. A ROS driver, itself, is only a token gesture that you are willing to participate in the open-source robotics / automation movement. Not addressing fundamental details such as latency and sensor exposure is illogical, at best, however.

While K4A outperforms other depth cameras in clarity and clean edge detection, it achieves these qualities by sacrificing other basic ones, namely fast throughput and high dynamic range. This renders the K4A unusable in most major ROS applications. Hopefully, you are aware of the problem and are addressing it appropriately.

[ooeygui]

Thank you very much for your feedback and writeup. The Azure Kinect ROS node builds on top of the Azure Kinect SDK. I followed up with the Azure Kinect team and linking this issue to related issues in the SDK repo: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/816 https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/1501

[christian-rauch]

From the description and the example data, this seems to be more like an issue with the automatic gain adjustment, not HDR (i.e. using 12bit per pixel) per se.