WIP / RFC - DNG opcode list processing

[paolodepetrillo]

I have been experimenting with adding support for DNG Opcode List processing.

These opcode lists can be used for operations like:

• Color shading correction (needed for most smartphone cameras)
• Vignetting correction
• Distortion and CA correction
• Marking dead pixels or PDAF pixels
• Applying a gamma curve for lossy 8 bit raws

The way the DNG spec defines these is very general - in theory there can be any number of opcodes, in any opcode list, applied to any region of the image. But any specific camera will typically only use one or two, consistently for all images it outputs. It isn't really necessary to support any arbitrary combination of opcodes, just the cases that do appear in images from supported cameras.

This PR will get the opcodes with exiv2 when using rawspeed. It can also get the opcodes using rawler, but only with my branch which makes the opcodes available. Either way it decodes the opcodes into C structs that are easy to access. I added them to dt_image_params_t but maybe would be better to use the new dt_image_metadata_t linked list?

So far I have only added support for the GainMap opcode in OpcodeList2 (pre-demosaic), equivalent to this in darktable.

I'm not sure about the best way to add this processing to a vkdt graph. Here I have created a new module 'dngop2' which would apply all OpcodeList2 processing, and there could be a similar module dngop3 for post-demosaic. It goes immediately after denoise because it needs the black/white point applied to rescale to (0,1), but this isn't exactly correct because it is supposed to operate on the full image prior to the crop that is also applied in denoise. Within the module it would add a node for each opcode in the list, or be a no-op for an image that has no opcodes. This could be added to the generic default graph for raw images or a dng specific graph. But if it just internally applies all the opcodes automatically it doesn't give the end user much control.

Another possibility would be to create a module for each individual opcode. This way, an expert end user who knows which opcodes their camera uses could manually create a graph for that camera which has the opcodes in the right place. An external script could take a dng file and the default graph, and create a modified graph with the opcode modules for that file added. Or a feature could be added to vkdt to automatically add the opcode modules to the graph.

[hanatos]

nice, thanks for the thorough pull request with so many options implemented (rawler + exiv2).

some thoughts/answers:

metadata linked list is probably a good idea, since the memory is read-only, i.e. the pointer will be passed on between modules and thus not cost additional per-module storage.

probably possible to have a (hidden?) parameter in the denoise module that will skip the crop and pass on the full image. in this case the dng op will have to do the cropping. seems important to get this right.

adding modules to the default raw graph will not incur much overhead since you already implemented the bypass option. so maybe that's a good way of doing it. potentially dng would be sufficiently different to generic raw and warrant their own defaults in the future.

another way would be to provide presets (in data/presets/*pst) that assume the default raw graph and add their dng ops in between.

coarse modules could implement a set of parameters that would then internally wire the nodes. probably a matter of simplifying the graph the user sees on screen. i would imagine a combobox with "apply/skip gainmap" being less obtrusive when there are like five of these options one under the other in the parameter list than a list of five modules in the graph.

[butcherg]

I haven't looked at the PR, so take with grain-o-salt...

Worth considering is making sure this capability is factored to allow non-dng images to be corrected with the same ops, specifically WarpRetilinear and FixVignetteRadial. Camera manufacturers are increasingly including coefficients in their raws to feed these, and at least Nikon appears to be using the Adobe model. I'm sure @paolodepetrillo knows more...

Edit: Okay, should read the prose before, so maybe exposing the coefficients as module input parameters would allow directing them from various sources, e.g., exiv2 data structure, data file, maybe even lensfun XML (they'll code an adobe op as 'acm'). I'd like to see a lensfun data interface at some time, would expose vkdt to a significant and maintained body of lens correction data.

[hanatos]

i think this is probably almost good to merge.

putting the dng info in the img_param struct directly is not a big memory impact (only stores pointers).

there's some exiv2 api thing going on, and the windows module api isn't happy. i'm thinking i might move the linux/macintosh way of linking back into core to a similar notion, with explicitly exposed symbols (so normal people will also catch such errors, not only the msys2 github worker).

[hanatos]

a different idea: what if we merge the functionality into the denoise module? it seems we'd want to apply these corrections on [0,1] scaled but uncropped data before denoising, so right in between what the denoise module does. also dng op is kinda technical, so not sure a separate module helps discoverability. also some simple things (gain map) might be applied inline when performing the pixel rescaling/cropping part in the denoise module. this makes it one less roundtrip from/to global memory and thus more efficient to evaluate. (for such simple things probably all the cost is in the memory accesses, the little bit of compute will be lost in the cracks of latency hiding)

[paolodepetrillo]

@butcherg I started working on lens correction in #97 - it should be possible to support the lensfun models eventually. Originally I was following the example of embedded metadata lens correction in darktable where all the different models are turned into a output radius -> input radius LUT, but I will probably switch it to just pass the coefficients into the shader and let it evaluate whatever the model formula is.

[paolodepetrillo]

I think that putting this Bayer GainMap in denoise would work - that's similar to how I had added it to the rawprepare module in darktable.

The opcode lists are defined as:

• OpcodeList1 - apply to raw codes before any other processing
• OpcodeList2 - apply after converting to linear [0,1]
• OpcodeList3 - apply to RGB after demosaic

According to the spec any of these lists can contain any of the opcodes, any number in any order. I don't think it would really be practical to support any arbitrary opcode list in the vkdt pipeline - a much more limited implementation should be able to support almost all cameras that output DNG natively, and typical Adobe DNG Converter output. Probably assuming there is only one instance of each, and assuming that order doesn't really matter, would be ok in almost all cases.

I have been looking around for samples and this is what I've seen actually used so far:

• FixBadPixelsList in OpcodeList1 - from a DJI Mavic Pro (FC220). It looks like they listed every individual PDAF pixel as a 'bad pixel'. This wouldn't be suitable for GPU processing, probably would have to apply it by CPU in i-raw right after loading the image.
• GainMap in OpcodeList2. This is probably the most important one to implement because colors can be way off without it. Many smartphone cameras, drones, GoPro, etc use it, usually to correct both color shading and vignetting at the same time. There are always 4 instances in the list, one for each Bayer filter.
• GainMap in OpcodeList3. So far have only seen this from a DJI drone - it's used only for the color shading and there is a separate FixVignetteRadial for vignetting.
• FixVignetteRadial in OpcodeList3. I guess this would also work just as well in OpcodeList2 although I've never seen it.
• WarpRectilinear in OpcodeList3 for distortion and / or CA.
• MapPolynomial in OpcodeList2 - I assume this is for a lossy low bit depth raw

So most of the important corrections could be handled by a combination of:

• FixBadPixelsList applied by CPU code in i-raw
• FixBadPixelsConstant could easily be added to hotpx
• Bayer GainMap from OpcodeList2 applied in denoise
• RGB GainMap from OpcodeList3 could maybe be converted to a Bayer GainMap having the same effect and applied with the same implementation in denoise. Or would need a new module after demosaic to apply it.
• WarpRectilinear, WarpRectilinear2, WarpFisheye would be in a lens correction module like #97
• FixVignetteRadial - even though it's usually in OpcodeList3 for demosaic, probably could just apply it in denoise at the same time as GainMap. Or create a new type of vignette module that uses a different polynomial, or add it to the existing vignette module?

[hanatos]

fyi, merged and cleaned conflicts in branch "gainmap". will look into merging the opcode2 module into the denoise module (noop.comp and doub.comp).