darktable-org
Implement capture sharpening inside demosaic module
This has been discussed many times here and on pixls.us so here we go. As we have D&S giving good results i was somewhat reluctant to work on this and do a pr but as CS works so good on RT and after a lot of testing (and using this on my personal images for a while with 3 auto applied presets for my fixed-lens camera - one for high-iso with CS turned off and two presets for wide open and "normal") i am pretty confident it's worth it and would be a significant improvement. (It took quite a while to spot a very bad bug in GPU code leading to system instability and crashes so unfortunately not for 5.2)
Please test in depth and let's decide if this is for master. It's a lot of processing - so not a superfast module.
Capture sharpening has been implemented to work inside the demosaic module so it's raw only. Credits to: Ingo Weyrich ([email protected]), he implemented the original algorithm for rawtherapee, this implementation is based on his work, especially the convolution kernels. CPU and OpenCL code paths are both available. Demosaic module gets more parameters so there is a version bump, one float has been reserved.
The "mini manual" (regularly updated to latest commit)
Capture sharpening (CS) tries to recover details lost due to in-camera blurring, which can be caused by diffraction, the anti-aliasing filter or other sources of gaussian-type blur. Prerequisites are
- good white balance parameters (same requirement as for highlights reconstruction or demosaic)
- no chromatic aberration, you might want to add the "raw chromatic aberration" module
- low noise as noise will be amplified by CS
Controls
- capture sharpen switches CS on if above zero and defines the strength of overall effect. CS works in an iterative process, this defines the number of iterations, mostly a setting of 10 will be enough. As downscaling after demosaicing reduces the visibility of CS we reduce iterations in such cases for performance.
- radius defines the basic convolution gaussian sigma. This should not be set by "creative means" but to the blurring radius of the optical system and sensor, too large values will lead to artifacts like halos. Calculating a correct radius is provided internally. This will be done either if you a) click on the button besides the slider b) activate capture sharpen the first time after resetting to demosaic defaults or developing old edits.
- contrast threshold As sensor noise will be amplified by CS we take some care about this by a per pixel variance analysis and use a logistic function with this threshold to avoid CS in noisy areas. The default is good for low iso images.
- corner boost Increase the radius in image corners. We assume a circle of 1/2 of image size to be "sharp" (only use main radius), locations outside this center circle get an increased convolution radius.
Nice effort, I like it :+1: However for some images I noticed a color shift the more iterations I choose for CS. Can you also reproduce it? Try very colorful images and maybe bump up the saturation a little bit in order to make the difference more visible. In my case I also overlayed the non-CS with the CS image and used the layer mode "difference" to also see the color cast quantitatively.
However for some images I noticed a color shift the more iterations I choose for CS. Can you also reproduce it?
I couldn't yet, could you somehow share such raw files? (And possibly the xmp too? It might be related to white balance coeffs. A pre-requisite would be a correct WB)
As CS is an iterative process any error would be amplified by more iterations.
In my case I also overlayed ...
I don't understand your "reproducer procedure" here.
Just did a quick test, for a consistent UI I would propose to make the sliders non sensitive when sharpen is 0 but not to hide them.
I can do so of course. EDIT latest commit changes this but for my taste it's looking worse. We also have the other optinoal sliders in this module only visible if "relevant"
In my case I also overlayed ...
I don't understand your "reproducer procedure" here.
What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference". For some images I also noticed that the image with CS turned on appear slightly brighter.
I'll try to find images which I can share for you to reproduce.
What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference".
You are aware of the --dump-pipe module option?
Squashed and force-pushed an updated version
- convolution kernels support sigma-steps of 0.01 instead of 0.02
- slightly improved CPU convolution performance
- the per-pixel variance anylysis is done in a circle with a diameter of 5 instead of 3 for improved stability in noisy regions and at sharp luminance transitions.
What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference".
You are aware of the
--dump-pipe moduleoption?
Nope, never used it, what is it good for?
What I meant was exporting an image without CS and one with CS turned on with at least 15 iterations, then compare the images overlayed in separate layers in gimp, using the layer mode "difference".
You are aware of the
--dump-pipe moduleoption?Nope, never used it, what is it good for?
It writes pfm files to tmp directory while processing the specified module in the pipe, it's meant for debugging exactly such issues :-)
darktable --dump-pipe demosaic would be your friend (use HQ processing mode)
Latest version includes a variance lut for performance plus some readability maintenance. (Some NaN issues spotted while testing this are in a bugfix PR for 5.2)
Latest version got some subtle perf gain, more important is improved stability for high-noise underexposed images. Right now i think it's good as is and i would appreciate testing. Please bear in mind, i have squashed and force-pushed the update to keep number of commits low.
I am still not convinced about the UI.
- Is it really better to make the unused sliders insensitive instead of just hiding?
- Any other idea?
@jenshannoschwalm do you have suggested settings or at least a good starting point?
Normally with a strength of 10 would be fine. Evaluate at 100% zoom.
Latest version - as usual load the whole pr for testing -
- Does automatic calculation of radius if capture sharpen has been enabled
- Some improved tooltips
- The maximum radius has been increased to 2.0 as this is still acceptable with the convolution kernels.
make it working again with current master, convolution stuff has not changed, the "avoid-noisy-areas" threshold has been improved.
force-pushed rebased on master and minor code maintenance, nothing algo-wise changed.
- use a different default contrast threshold after a lot of testing - definitely better for medium ISO images while still perfect for low ISO
- Second commit while being here cleans up some demosaicer code.
Ready to go from my side. Latest force-push just has some more aggressive compiler options for CPU performance
Release note suggestion: Implemented capture sharpen (CS) inside the demosaic module so only available for raw images. xtrans and bayer sensors are supported, CPU and OpenCL code paths are provided. When done with dual demosaicing capture sharpen is only applied to the main demosaicer. Capture sharpening (CS) tries to recover details lost due to in-camera blurring, which can be caused by diffraction, the anti-aliasing filter or other sources of gaussian-type blur. The main parameters are "capture sharpen" defining the iterations and "radius". To improve results for noisy images the effect can be controlled via the "contrast threshold" parameter. As many lenses are less sharp in the corners we can do some additional sharpening there via "corner boost"
Latest force-pushed includes an improved luminance correction, thanks @da-phil Otherwise some improved tooltips and more appropriate range of parameters.
To be reviewed: Instead of using a slider defining the iterations use a simple menu. Also hide capture sharpen parameters if not applicable or unused as we do for other demosaicer internals.
This would be my personal preference.
@jenshannoschwalm This is really, really good. I ran some tests against the pixel deblur module I was working on and this is clearly an order of magnitude better than pixeldeblur. Amazing how it does not exaggerate noise. Halos and ringing can be a problem if you push it too far (I did say I was testing ;-) ). I'll have a closer look as time goes by (busy week coming up).
One thing I would choose to change is the "corner radius" implementation. Every lens is going to have different focus/diffraction as a function of distance from the image center.
Halos and ringing can be a problem if you push it too far
Right. No idea yet though how/if we could control this further.
One thing I would choose to change is the "corner radius" implementation. Every lens is going to have different focus/diffraction as a function of distance from the image center.
True. The good point in this implementation is, we can easily adopt for other and improved algorithms as we use a per-pixel map defining the local gaussian sigma, see _cs_precalc_gauss_idx. I would love to see improvements happening after this hopefully get's merged :-)
I could add a "visualizer" to that parameter ...
To be reviewed: Instead of using a slider defining the iterations use a simple menu. Also hide capture sharpen parameters if not applicable or unused as we do for other demosaicer internals.
This would be my personal preference.
I'm fine with hiding the capture sharpen parameters as it is currently done. Having a drop-down menu with presets including "disable" is an elegant way to save one UI element, it feels more natural than the previous slider ;)
However I'm a little bit torn with having only "presets" instead of actually defining the amount of iterations, as in RawTherapee. As I'd consider DT users (including myself) as rather tech savy, I'd prefer to fine-tune iterations, and maybe slightly go above the extreme iteration value in some cases. So I wouldn't mind the extra UI element, and make the drop-down menu only switch CS on / off and have the iterations slider back.
Added a visualizing button to corner boost ...
Thanks, looks good :+1:
A speedup for _modify_blend in src/iop/demosaicing/capture.c -- efficient way to calculate standard deviation (sorry for my git-ignorance, I'm sure there is a better way to get this code to you)
` static void _modify_blend(float *blend, float *Yold, float *luminance, const float threshold, const int width, const int height) { DT_OMP_FOR() for(int irow = 0; irow < height; irow++) { const int row = CLAMP(irow, 2, height-3); for(int icol = 0; icol < width; icol++) { const int col = CLAMP(icol, 2, width-3); const size_t k = (size_t)irow * width + icol; float sum = 0.0f; float sum_sq = 0.0f; int n=0 ;
for(int y = row-1; y < row+2; y++)
{
for(int x = col-2; x < col+3; x++) {
sum += Yold[(size_t)y*width + x];
sum_sq += Yold[(size_t)y*width + x]*Yold[(size_t)y*width + x] ;
n++ ;
}
}
for(int x = col-1; x < col+2; x++)
{
sum += Yold[(size_t)(row-2)*width + x];
sum_sq += Yold[(size_t)(row-2)*width + x]*Yold[(size_t)(row-2)*width + x] ;
n++ ;
sum += Yold[(size_t)(row+2)*width + x];
sum_sq += Yold[(size_t)(row+2)*width + x]*Yold[(size_t)(row+2)*width + x] ;
n++ ;
}
const float sum_of_squares = sum_sq - sum*sum/(float)n ;
const float variance = sum_of_squares / (float)n ;
const float std_deviation = sqrtf(variance) ;
float sv = powf(MAX(0.0f, 5.0f * std_deviation - threshold), CAPTURE_THRESHPOWER);
blend[k] *= CLIP(sv);
luminance[k] = Yold[k];
}
} } `