[OTHER] saving raw files - missing second color matrix in exif-data?

[cpixip]

I used the qt-app in the "next" branch to take some raw files at the resolution 2028x1520. Scanning the resulting .dng-file with the classical exiftool, I discover a "Camera Matrix 1", but no "Camera Matrix 2". Is this the intended behavior? The source code of pidng operates with two color matrices, and I guess that is generally now the accepted standard: one for a low color illuminant, and one for a higher color illuminant. Intermediate ccms are interpolated, depending of the chosen color temperature. The new 1. 6 spec even lists a third color matrix as an option.

Note added:

Here's what exiftools spits out of a full resolution capture:

$ ./executables/exiftool full_res.dng
ExifTool Version Number         : 12.44
File Name                       : full_res.dng
Directory                       : .
File Size                       : 18 MB
File Modification Date/Time     : 2022:08:09 17:53:36+02:00
File Access Date/Time           : 2022:08:10 10:23:03+02:00
File Creation Date/Time         : 2022:08:09 17:56:21+02:00
File Permissions                : -rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Full-resolution image
Image Width                     : 4056
Image Height                    : 3040
Bits Per Sample                 : 12
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Make                            : RaspberryPi
Camera Model Name               : PiDNG / PiCamera2
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 1
Software                        : PiDNG
Tile Width                      : 4056
Tile Length                     : 3040
Tile Offsets                    : 756
Tile Byte Counts                : 18495360
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 2 1 1 0
Exposure Time                   : 1/15
ISO                             : 800
DNG Version                     : 1.4.0.0
DNG Backward Version            : 1.0.0.0
Black Level Repeat Dim          : 2 2
Black Level                     : 256 256 256 256
White Level                     : 4095
Color Matrix 1                  : 0.4295 -0.1043 -0.0155 -0.249 0.9409 0.2716 0.0523 0.1083 0.5023
Camera Calibration 1            : 1 0 0 0 1 0 0 0 1
Camera Calibration 2            : 1 0 0 0 1 0 0 0 1
As Shot Neutral                 : 0.2869440459 1 0.7051191651
Baseline Exposure               : 1
Calibration Illuminant 1        : D65
Raw Data Unique ID              : 343736393335353037303030
Profile Name                    : PiDNG / PiCamera2 Profile
Profile Embed Policy            : No Restrictions
CFA Pattern                     : [Blue,Green][Green,Red]
Image Size                      : 4056x3040
Megapixels                      : 12.3
Shutter Speed                   : 1/15

As one can see, there's only a single Color Matrix available. Is that the intended behaviour?

The class definition class RaspberryPiHqCamera(BaseCameraModel) in pidng lists actually four matrices corresponding to two illuminants:

        profile_name = "Repro 2_5D no LUT - D65 is really 5960K"
        profile_embed = 3

        ccm1 = [[6759, 10000], [-2379, 10000], [751, 10000],
                [-4432, 10000], [13871, 10000], [5465, 10000],
                [-401, 10000], [1664, 10000], [7845, 10000]]

        ccm2 = [[5603, 10000], [-1351, 10000], [-600, 10000],
                [-2872, 10000], [11180, 10000], [2132, 10000],
                [600, 10000], [453, 10000], [5821, 10000]]

        fm1 = [[7889, 10000], [1273, 10000], [482, 10000],
                [2401, 10000], [9705, 10000], [-2106, 10000],
                [-26, 10000], [-4406, 10000], [12683, 10000]]

        fm2 = [[6591, 10000], [3034, 10000], [18, 10000],
                [1991, 10000], [10585, 10000], [-2575, 10000],
                [-493, 10000], [-919, 10000], [9663, 10000]]
...
        ci1 = CalibrationIlluminant.Standard_Light_A
        ci2 = CalibrationIlluminant.D65

as it would be expected.

[davidplowman]

The problem is that the libcamera metadata gives us only one colour matrix - the one being used. So it's not really clear where we'd get the other one from. The user could obviously take captures in different conditions to get them, or even compute them from the tuning file (not difficult), and add them "by hand". But as things stand, I don't have a way to make all that happen automagically. Sorry!

[cpixip]

Ok, so that makes kind of sense. So the single matrix stored in the .dng-file is the matrix libcamera used in creating the jpg-visualisation of the camera's view.

I think that the usual approach is the following (according to page 85 of the DNG 1.6 spec):

If exactly two calibrations are included, then it is recommended that one of the calibrations be for a low color temperature illuminant (e.g., Standard-A) and the second calibration illuminant be for a higher color temperature illuminant (e.g., D55 or D65). This combination has been found to work well for a wide range of real-world digital camera images.

Usually the general raw-converters use this data (the two/four matrices plus the illuminant information) to come up with an own ccm for rendering. I assumed that is the purpose of the two ccms which can be found in the pidng-source and that this would be written out in the .dng as well.

In fact, DNG 1.3 spec now even allows for a third matrix, citing again "If three calibrations are included, then it is recommended that the first two calibrations are as describedin the preceding paragraph, and the third calibration be used to characterize the "as shot" illuminant."

For the case of two ccms, the DNG 1.6 spec actually describes the following procedure of interpolation:

"DNG 1.2.0.0 and later requires a specific interpolation algorithm: linear interpolation using inverse correlated color temperature. To find the interpolation weighting factor between the two tag sets, find the correlated color temperature for the user-selected white balance and the two calibration illuminants. If the white balance temperature is between two calibration illuminant temperatures, then invert all the temperatures and use linear interpolation. Otherwise, use the closest calibration tag set."

Anyway, at the moment I have the impression that the ccm contained in the .dng-file is not really the optimal one. When I open the image for example in RawTherapee 5.8, I get something like the image displayed on the right side

The small colored patches which can be noticed inside each of the color checker's patches indicate how the color of each patch should actually look like. Clearly, most of the images colors are off a little bit. Note especially the blue patch. Generally, the saturation seems to be too strong with the embedded matrix.

For comparison, on the left side is the best image rendering (ccm-matrix) I could come up with. Note that the blue patch on the left is rendered much closer to the reference than on the right. In fact, the average color error over all patches is reduced by about a factor of four in the left image, compared to the right side.

Anyway, thanks for the clarification. In summary, it seems that the single ccm recorded in the .dng comes from libcamera's choice of ccm at the time the picture was taken, which in turn depends on the matrices given in the tuning file. As far as I know, the actual matrix libcamera is using is computed on the fly from the AWB's estimation of current color temperature and the closest two matrices in the tuning file. So one way to improve the situation would be to have a closer look into the tuning file.

[cpixip]

Libcamera/picamera2 writes the estimate of the current color temperature and the compromise color matrix used to create the embedded preview-jpg into the DNG-files. Generally better results can be achieved by by-passing this data. One way to do this is to use DCP input profiles in the raw converter. For the HQ camera/IMX477 sensor, these can files can be downloaded from here.

Once this input profile is selected in the raw converter, the embedded color matrix is no longer used. A second necessary step is to adjust the white balance manually, for example by sampling a neutral colored patch with the white balance picker. This second step discards the embedded white-balance parameter. The resulting image

is much closer to ground truth than the image created with the data embedded in the DNG: