PX4-Flow
PX4-Flow copied to clipboard
PX4
Improve flow quality
We noticed quite heavy flickering in the image output from the camera sensor. We tested an array of approaches to mitigate the issue.
Primarily we perform whitening of the camera input. This improves the flow quality and stability immensely for us.
And can you take a picture of the hardware? If the supply is not properly stabilized you might suffer from a bad HW build.
https://www.dropbox.com/sc/8jlcyy7gj9tknvy/AAANLvfISKlj71DTSpeL4wZCa
- Ludvig
On 15 Jun 2017, at 10:52, Lorenz Meier [email protected] wrote:
And can you take a picture of the hardware? If the supply is not properly stabilized you might suffer from a bad HW build.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
~FYI the CI failure seems to be a bug, refresh it and it should work -- it checked out old code.~ Sorry my fault, forgot to commit a second change. Force pushing.
By the way, the difference in signal quality from the flow sensors is enormous for us with this whitening turned on. Might be hardware, of course.
You're right that AGC should probably not be on, empirically though it seems to be better. I'll look into it again tomorrow.
- Ludvig
On 17 Jun 2017, at 20:00, James Goppert [email protected] wrote:
@jgoppert commented on this pull request.
In src/modules/flow/dcmi.c:
}
- else if (dcmi_image_buffer_unused == 2)
- {
for (uint16_t pixel = 0; pixel < image_size; pixel++)(*current_image)[pixel] = (uint8_t)(dcmi_image_buffer_8bit_2[pixel]);- }
- else
- {
for (uint16_t pixel = 0; pixel < image_size; pixel++)(*current_image)[pixel] = (uint8_t)(dcmi_image_buffer_8bit_3[pixel]);
- for (uint16_t pixel = 0; pixel < image_size; pixel++)
(*current_image)[pixel] = source[pixel];+} + +void whitened_image(uint8_t *source, uint8_t *dest, uint16_t image_size) { So I'm wondering how this compares to the auto gain and brightness controls within the firmware. Both pieces of software have the same goal, so you might just be eating more cpu here. You might want to check. But if your method is better than the default firmware, maybe you want to turn auto gain off.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
The noise is about the same with and without AGC, so I removed it as per your recommendation @jgoppert -- one less moving part.
Wait, wait. Did you test in sunlight and at very low indoor conditions? Because without AGC you might perform worse in one of the two. Please make sure to run a solid test.
I primarily tested indoors. Disabling AGC shouldn't affect the signal quality however, since the aim of the AGC mechanism is the same as the whitening operation. I'll try it with sunlight in a second.
Seems to work well in sunlight as well actually. How low light are low light conditions?
Here is an example where the image quality drops (this is the floor of my office right now):
- above dark area: quality = 255
- above bright area: quality = 255
- above border: quality = 0
EDIT: on master, the quality for the same images was respectively 130, 190 and 150
Great feedback, @jlecoeur, thanks. Indeed it does have issues with high-variance areas; in particular multi-modalities such as the one you presented.
Yeah, agc likely controls how many photons it takes to make 1 bit, so it will have a different effect. In software with gain you are stretching the raw data out. That is why I never looked into a post camera image whitening. Do you have any intuitive sense why this is helping more than AGC and desired brightness?
Indeed @lericson , see the histograms:
However computing the histograms and detecting multi-modes might be too expensive, can we afford calling compute_flow two times (before and after whitening), and keep the best one?
I think it won't be worth it, @jlecoeur -- what we could do though is some kind of naive outlier rejection. This would mean half the image becomes either completely black or completely white, which would in this scenario be more appropriate. Something like having two estimates, one for pixels over 127 (i.e. 50%) and one for those under. Let's try that.
@jgoppert The gain mechanism in the camera firmware is just a multiplier AFAICT.
For the point grey cameras it is photons to bits. So worth checking. I think you would still want your whitening though because the agc lags and would explain the flickering.
@jlecoeur Tried binning, not very surprisingly it turns out to give rise to flickering as the mode changes between the bins. Three bins was pretty good though.
Could probably make it flicker less by introducing some penalty for switching mode.
@lericson Yes, I think AGC + whitening makes sense. Basically AGC is your first pass before data collection, you have to guess what the gain should be, then after you collect the image it computes the mean and variance and updates the gain for the next loop. Your whitening method would then operate on the output of that loop and fix transients.
@lericson I fixed the problem with high variance areas on this branch https://github.com/PX4/Flow/compare/master...lis-epfl:master_KTH_image_whitening?expand=1
I added a mode WHITENING_AUTO where optical flow is computed on whitened images, then if the flow quality is lower than a threshold, flow is re-computed using non-whitened images. Now the quality does not fall to 0 like in my previous comment.
The performance cost goes increasing: IMAGE_WHITENING_DISABLED: no performance cost compared to master IMAGE_WHITENING_ALWAYS: fixed cost for whitening of current image IMAGE_WHITENING_AUTO: fixed cost for whitening of current image + variable cost for 2nd flow computation only when quality is poor.
Feel free to cherry-pick the 3 last commits if you like it.
Great stuff, @jlecoeur! Did you try it in practice? Is the added complexity and performance penalty worth the savings compared to just whitening all the time? I've been experimenting with this in a mixed light indoor environment (i.e. semi-occluded sunlight) and I've not found any significant performance hit.
In fact one of our bigger problems right now isn't the flow estimation -- it seems to work more or less flawlessly -- but making the LPE module of the PX4 firmware integrate the flow data into its estimation of the pose in a meaningful way. Right now it seems it trusts the accelerometer too much or something, and the velocity estimate just flies off the handle. Got any experience with this?
@lericson I tried whitening in a building with sharp shadows on the ground. And indeed it was an issue, the drone was rejecting position control only when flying above shadow patterns. I assume this was the same phenomenon I witnessed in my office, but I have not tested the IMAGE_WHITENING_AUTO fix in that same building. I will do it probably on Monday.
Regarding the performance penalty, hand-held test showed that the 2nd flow computation happens only when there s a sharp brightness change in view. So in most cases this is not more expensive than whitening only.
Regarding estimators, I had the same trouble and finally settled with ekf2. As I understood it is not ready for full missions in GPS denied environments yet. But I take off and land in stabilized mode, so I do not care. I increased a bit flow innovation gate to prevent it from rejecting flow when it is not perfect. I also had to lower vibrations to around 1g~2g on x and y axes.
Too bad nothing is happening here! A demonstration of a hexacopter flying indoors using the code in this PR: https://youtu.be/mgxxTmgxnQ0
@jlecoeur @lericson Sorry for being slow on you. I'm very much impressed by your execution speed and contributions.
@ChristophTobler Could you help to get this in once you're back? Thanks!
